estudy 한국어/English

Potential Risks When Using VPA

973 단어·2 분·원문(.md)

VPA, or Vertical Pod Autoscaler, is a feature that automatically adjusts the CPU, memory, request, and limit values of a pod.

In other words, it analyzes the actual resource usage of an application (metrics like CPU utilization that HPA monitors + memory usage patterns) to

conclude that "this pod needs more memory than it currently has" or "CPU can be reduced," and then automatically updates the pod's specifications.

Core Operations

  • Recommender: Collects pod resource usage and calculates appropriate request/limit values.
  • Updater: Restarts pods as needed to apply new resource specifications.
  • Admission Controller: Applies recommended values to newly created Pods.

Why is it needed #

It's difficult for humans to accurately set request/limit values. We often see resources being allocated heuristically,

but due to traffic fluctuations, batch jobs, GC patterns, memory peaks, etc., it's virtually impossible to set precise, fixed resource values.

VPA sets values based on actual measurements.

Preventing Over-provisioning

While generously allocating CPU and memory ensures stability, it wastes node resources and increases costs.

VPA maintains minimal, well-justified Request/Limit values based on actual usage.

Preventing Under-provisioning

Conversely, if memory peaks are not considered, OOMKilled can occur. VPA analyzes memory patterns to

increase the appropriate Request.

HPA Complement

HPA adjusts the number of pods, but Request/Limit values remain unchanged. So, if you scale out to 10 pods, but the request is

unreasonably high, scale-out may not work properly due to insufficient nodes.

VPA addresses precisely this inefficiency.

Potential Risks When Using VPA #

This is most crucial in production environments, where most issues arise from Pod restarts.

  1. Forced Pod Restart -> Temporary Traffic Outage (downtime)

The updater requires pod recreation to apply new resource values.

If zero-downtime configurations (rolling updates, PodDisruptionBudget to maintain a minimum number of pods) are not properly set up, the service can experience momentary interruptions.

StatefulSets, in particular, are very risky.

  1. Memory Over-Recommendation -> Node Shortage / Cost Surge

The Recommender tends to recommend generous memory Requests to ensure safe operation.

In this case, the following problems can occur:

  • A single pod's memory request increases significantly, making it unschedulable on a node.
  • Overall cluster resource pressure -> increased costs
  • Unnecessary Scale out
  1. Risk of Conflict with HPA

When HPA and VPA are used together, the following issues arise:

  • HPA scales based on CPU utilization.
  • VPA changes the CPU request/limit itself.
  • If the CPU Request changes, CPU utilization (usage, request) is recalculated, which can cause HPA to malfunction or excessively increase the number of pods.

Generally, VPA is often configured to adjust only the request, leaving the Limit unchanged.

Alternatively, if you want to use HPA and VPA simultaneously, set VPA to update only the Request.

  1. Risk of Incorrect Resource Values Being Recommended Due to Instantaneous Load Peaks

If a sudden memory spike occurs at a specific time, the Recommender might interpret it as a normal peak and excessively increase the Request.

This is especially true for temporary patterns like GC pauses, specific batch jobs, abnormal failures, or memory leaks.

It is also affected by such transient patterns.

  1. Increased Unpredictability for Critical Workloads

VPA recommendations are statistics-based, so their prediction accuracy is not 100%.

For resource-sensitive workloads, it might be better for humans to set fixed values directly.

  1. Conflict with Node Scaling Logic

In an EKS cluster autoscaler environment, if requests increase due to VPA, the following happens simultaneously:

  1. Pods cannot be scheduled on existing nodes.
  2. The cluster autoscaler attempts to scale out nodes.
  3. Before nodes are added, VPA calculates again.
  4. This repeats.

Autoscaling can become unstable, and costs may increase.

Conclusion #

VPA is a powerful feature that significantly enhances resource efficiency and stability, but caution is needed due to its restart-based nature.

Care must be taken when coexisting with HPA due to potential conflicts. Additionally, there's a risk of the recommender over-provisioning.

Paradoxically, while it can prevent human over-provisioning, VPA itself might over-provision due to spike traffic or high CPU utilization batch jobs.

Let's just determine the number of pods through performance testing...

Example #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: app
          image: example/sample:latest
          resources:
            requests:
              cpu: "200m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

Preventing downtime with a rolling update deployment 1

This is a CPU-based HPA that scales up to 10 replicas.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sample-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

This is a custom HPA that scales based on RPS values. It is not affected by CPU fluctuations.

It's based on QPS HTTP latency.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sample-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 2
  maxReplicas: 15
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "50"

Let's also look at a VPA example.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: sample-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       sample-app
  updatePolicy:
    updateMode: "Auto"  # 자동으로 Request 조정 + 필요한 경우 Pod 재시작
  resourcePolicy:
    containerPolicies:
      - containerName: app
        mode: "Auto"
        controlledValues: "RequestsOnly"   # 핵심: Limit은 변경하지 않도록 강제
        minAllowed:
          cpu: "100m"
          memory: "128Mi"
        maxAllowed:
          cpu: "1"
          memory: "2Gi"

If configured as above, let's ensure at least one pod remains alive by enabling a PodDisruptionBudget policy.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: sample-app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: sample-app

Using LimitRange to protect the cluster and restrict VPA from recommending excessively low or high requests at the namespace level.

apiVersion: v1
kind: LimitRange
metadata:
  name: namespace-limit-range
spec:
  limits:
    - max:
        cpu: "2"
        memory: "4Gi"
      min:
        cpu: "50m"
        memory: "64Mi"
      type: Container
SRE/question/q_7.md