How to Establish a Rollback Strategy

In a deployment pipeline, the most crucial aspect is not how to deploy well, but how quickly you can revert without bleeding when things go wrong.

App DB Separation #

This is a major premise for rollback strategies: when establishing one, you must first realize that while application code can be rolled back in a second, database rollbacks are hell.

Therefore, strategies should always be established by strictly separating the stateless app layer from the stateful DB layer.

Application Layer Rollback Strategy #

Let's explore strategies for immediately redirecting traffic back to a stable version when code goes wrong.

Blue/Green Starting with Blue Green Deployment, this method involves launching a new version (green) server with an identical environment to the old version (blue) at 100%, and then simply switching the target of the load balancer (Nginx, AWS ALB, etc.) to green.

• The rollback method is to simply point back to blue if an error occurs in the new green version.
• The advantage is an overwhelmingly fast rollback speed, less than 1 second.
• The disadvantage is that server infrastructure resources are required at twice the normal amount.

Canary involves deploying only a small percentage (e.g., 5% of the total) of new version servers and routing only a small portion of traffic to them to test the waters.

It gradually increases traffic while monitoring for error rates or CPU spikes, eventually reaching 100%.

• The rollback method is that if an error is detected on a canary server, the traffic routing rule to that server is deleted, and 100% of traffic is immediately redirected to the old version.
• The advantage is that even if a failure occurs, only 5% of users are affected, not all users.

This is not a deployment method but rather involves using feature toggles. It means configuring the code with a switch, managed either in a config file within the code or in a DB, like if (isNewFeatureEnabled) { newLogic() } else { oldLogic }, without touching the infrastructure.

• The rollback method is that without needing to redeploy, changing the switch value to false in an external configuration server (Redis, AWS AppConfig, etc.) immediately reverts to the old logic.
• The advantage is the flexibility to enable it only for specific users (e.g., internal IPs).

Database Layer Rollback Strategy (Forward Recovery) #

This is the trickiest part: if you change a column name and then roll back the app, the old version of the app won't find the changed DB column, leading to a complete outage.

Therefore, when performing DB migrations (Flyway, Liquibase), the key is not to consider rollback (drop, alter), but to unconditionally maintain backward compatibility so that any version of the app can operate.

This is known as the Expand & Contract pattern.

1. DB Change Expand (Additions Only): Only add new columns without modifying or deleting existing ones. The old version of the app uses the existing columns, so it operates normally.
2. App Deployment (Dual Write): Deploy the new version of the app. This app writes data simultaneously to both existing and new columns (double write) and reads from the new columns. This allows for rolling back to the old version of the app at any time.
3. Data Migration (Synchronization Batch): Run a background batch job to copy historical data from the existing column to the new column.
4. DB Cleanup (Drop Deployment): After a few days, once the system has stabilized, finally delete the unused old version column.

Command and Configuration Example #

1-second Blue/Green Rollback Switching using Nginx

This is the most basic yet reliable infrastructure-level rollback method. It involves changing only the upstream referenced in nginx.conf and reloading Nginx.

# nginx.conf
http {
    # Blue (old version) server group
    upstream backend_blue {
        server 10.0.0.1:8080;
        server 10.0.0.2:8080;
    }

    # Green (new version) server group
    upstream backend_green {
        server 10.0.0.3:8080;
        server 10.0.0.4:8080;
    }

    server {
        listen 80;
        
        location / {
            # 🚨 In case of failure, just swap the comments below and Nginx Reload!
            # proxy_pass http://backend_green; # New version failed! Comment out
            proxy_pass http://backend_blue;  # Immediately roll back to old version!
        }
    }
}

After making the change with nginx -t && nginx -s reload, restore traffic without downtime.

Let's also look at DB migration that maintains backward compatibility, as discussed above, using a Flyway SQL example.

When you want to split the 'name' column in the 'user' table into 'first_name' and 'last_name', you must never delete the existing 'name' column immediately.

-- V2__add_new_name_columns.sql (Script to apply during deployment)

-- 1. Only 'add' new columns. (Never DROP the existing 'name' column)
ALTER TABLE users ADD COLUMN first_name VARCHAR(50);
ALTER TABLE users ADD COLUMN last_name VARCHAR(50);

-- In this state, even if the App deployment fails and rolls back to the old version, 
-- the old version of the App still looks at the existing 'name' column, so no outage occurs.

There are other ways to change DB schemas, such as online DDL, which avoids table locks, but these are beyond today's topic, so they will be omitted.