Stability Checks to Consider for Deployment Automation

In deployment automation, stability checks should be systematically enforced as gatekeeping across three stages: pre-deployment CI, during-deployment CD, and post-release. The core idea is to replace manual verification processes with scripts and monitoring tools.

Pre-Deployment Verification Continuous Integration #

These are quality and security checks performed before code is built into an artifact (e.g., Docker image, JAR).

Static Code Analysis: This involves integrating tools like SonarQube to identify code smells, potential bugs, and security vulnerabilities, and to fail the build if test coverage criteria are not met.

There's also Container Image Vulnerability Scanning, which checks if the built Docker image contains known OS vulnerabilities (CVEs) or outdated libraries. Deployment is blocked if High/Critical severity vulnerabilities are found.

In-Deployment Status Verification Continuous Deployment #

This check verifies whether a new version of the application, when deployed to actual servers or container orchestration environments, is ready to receive traffic.

Readiness Probe: This checks if the application is ready to handle actual requests, not just if the server process has started, but also if DB connection pools are initialized or caches are loaded. Only after passing this check will the load balancer route traffic to that server.

Liveness Probe: This periodically checks if the application is running normally and has not entered a deadlock state.

Post-Deployment Verification / Automated Rollback #

Immediately after traffic starts flowing to the new version, this stage determines the success of the deployment based on actual metrics and automatically rolls back if an anomaly occurs.

Smoke tests involve sending automated HTTP requests to core APIs immediately after deployment, such as /health or /api/v1/status, to check for a 200 OK response and verify response times.

Metric-based Rollback collects error rate or latency metrics for 5-10 minutes after deployment. If a configured threshold is exceeded, traffic is immediately reverted to the previous version without human intervention.

Practical Examples (Code, Commands, Configuration Files) #

Let's look at examples of how to implement stability checks for each stage in code.

CI Stage: Container Image Security Scan using Trivy

This command scans for security vulnerabilities in the image to be deployed, added to CI scripts in tools like GitHub Actions or Jenkins.

# Trivy를 사용하여 'CRITICAL' 등급의 취약점이 하나라도 발견되면 exit code 1을 반환하여 파이프라인을 중단시킴
trivy image --severity CRITICAL --exit-code 1 my-registry/my-backend-app:v2.0

my-registry/my-backend-app:v2.0 (alpine 3.15.0)
===============================================
Total: 1 (CRITICAL: 1)

+----------------+------------------+----------+-------------------+---------------+---------------------------------------+
|    Library     |  Vulnerability   | Severity | Installed Version | Fixed Version |                 Title                 |
+----------------+------------------+----------+-------------------+---------------+---------------------------------------+
| openssl        | CVE-2022-0778    | CRITICAL | 1.1.1l-r0         | 1.1.1n-r0     | openssl: Infinite loop in BN_mod_sqrt |
+----------------+------------------+----------+-------------------+---------------+---------------------------------------+

Interpreting the above result, it indicates that a critical infinite loop vulnerability was found in the OpenSSL library, causing the CI pipeline to immediately stop.

Next, let's look at CD Stage: Kubernetes Pod Status Verification with Liveness and Readiness Probes

This is the most basic and essential Kubernetes stability check configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      containers:
      - name: payment-api
        image: my-registry/payment-api:v2.0
        ports:
        - containerPort: 8080
        
        # Readiness Probe: 트래픽을 받을 준비가 되었는지 검사
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness # Spring Boot Actuator 경로 예시
            port: 8080
          initialDelaySeconds: 10 # 컨테이너 시작 후 10초 뒤부터 검사 시작
          periodSeconds: 5        # 5초 주기로 검사
          successThreshold: 1     # 1번 성공하면 Ready 상태로 전환 (트래픽 인입 시작)
          failureThreshold: 3     # 3번 연속 실패 시 Not Ready 처리 (트래픽 차단)

        # Liveness Probe: 앱이 죽어있는지(Deadlock 등) 검사
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 10
          failureThreshold: 3     # 3번 연속 실패 시 해당 파드를 강제 재시작(Restart)

Post-Deployment: Implementing Metric-based Automated Rollback using ArgoCD Rollouts

This configuration automatically rolls back if the error rate exceeds a threshold, by checking Prometheus metrics during a canary deployment.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  metrics:
  - name: error-rate
    # 5분 간격으로 에러율을 측정하여, 5% 이하(<= 0.05)일 때만 배포를 통과시킴
    successCondition: result[0] <= 0.05
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        # 최근 5분간의 5xx 에러율을 계산하는 PromQL 쿼리
        query: >+
          sum(rate(http_requests_total{status=~"5.*", service="payment-api"}[5m])) 
          / 
          sum(rate(http_requests_total{service="payment-api"}[5m]))

After ArgoCD deploys a new version, it evaluates this query. If the result is 0.06 (6%), it does not satisfy the successCondition, immediately stopping the deployment process and rolling back 100% of traffic to the old version.