Blue/Green Deployment vs Canary Deployment

Blue/Green Deployment #

Blue/Green is very straightforward. It involves setting up an infrastructure environment, GREEN, that is perfectly identical to the currently serving BLUE environment, and then, at deployment time, flipping a switch on the load balancer to instantly shift 100% of the traffic.

It can be considered the standard for Zero-Downtime. If Connection Draining (where existing requests are completed and only new requests are routed to green) is properly configured to prevent session interruptions during the switch, perfect zero-downtime deployment is possible.

Warm-up is also possible: before directing real user traffic, you can send internal test traffic to the Green environment to pre-fill caches, optimize JVM JIT compilers, or use it for final QA.

The downside is cost. If you normally operate with 100 servers, you need to spin up an additional 100 servers for deployment. In a cloud environment, you can save costs by immediately terminating the blue environment after deployment, but in on-premise or similar environments, hardware costs double.

Canary Deployment #

It's easy to understand as a 'testing the waters' method, originating from how miners historically sent canaries into coal mines to detect toxic gases.

You first deploy only one new version server or about 5% of the new version, route only a portion of user traffic to it, monitor for spikes in error rates or latency, and then gradually increase it to 100%.

Minimizing the Blast Radius is possible; even if there's a critical bug in the code, only 5% of users experience the error instead of all users, which drastically reduces business risk or saves error budget.

There's a risk of version mixing, where old and new versions run simultaneously in the current system. If there are database schema changes or API response spec differences, implement the client to handle both v1 and v2 without exceptions, and then, once 100% deployment is complete, operate by removing the old version.

Session persistence is essential. If user A is routed to the old version one time and the new version the next every time they refresh, it can lead to significant confusion in data consistency or UX. Therefore, a cookie or header-based routing rule is needed to ensure that once a user is assigned to canary, they consistently go to the new version.

Example #

Let's look at Blue/Green rollback with k8s service selector.

In a k8s environment, you can shift 100% of the traffic by simply changing the label target of the Service that connects traffic, without needing to touch the ingress or load balancer.

# 1. Service configuration currently pointing to the old version (Blue)
apiVersion: v1
kind: Service
metadata:
  name: my-backend-service
spec:
  selector:
    app: my-backend
    version: v1.0 # <--- Currently, traffic goes to Blue (v1.0) pods
  ports:
    - port: 80
      targetPort: 8080

kubectl patch service my-backend-service -p '{"spec":{"selector":{"version":"v2.0"}}}' While there are other methods, as an intuitive example, you can change the version like this to direct traffic to green v2.

For Canary deployment, we'll also look at traffic distribution using nginx split_clients.

This configuration uses Nginx's built-in split_clients feature to hash the user's IP or session cookie and send only 5% of the traffic to the canary server.

http {
    # Split traffic based on user IP ($remote_addr)
    split_clients "${remote_addr}" $upstream_variant {
        5%      backend_canary; # 5% of users go to the new version
        * backend_main;   # The remaining 95% go to the old version
    }

    # Old version (Main) server group
    upstream backend_main {
        server 10.0.1.10:8080;
        server 10.0.1.11:8080;
    }

    # New version (Canary) server group
    upstream backend_canary {
        server 10.0.1.99:8080; # One newly deployed server
    }

    server {
        listen 80;
        location / {
            # Route traffic according to the split result ($upstream_variant)
            proxy_pass http://$upstream_variant;
        }
    }
}

The core of this configuration is hashing the remote_addr (user IP). Thus, a specific user A will always go to either the canary or the main version whenever they connect, which somewhat resolves session persistence issues.

In practice, if you have ample budget or not too many servers, Blue/Green is preferred. However, recently, to avoid wasting resources in microservices and k8s environments, automated canary deployments using tools like Istio and Argo CD are becoming quite common.