CNI (Container Network Interface)

CNI is a standard specification that container runtimes use to attach network interfaces to pods and assign IP addresses.

Kubernetes does not implement the network itself.

It operates with the structure: pod created -> CNI plugin call -> pod network settings.

In other words, it defines the following two things:

1. Standard for creating/deleting container network interfaces
2. IPAM standard for IP address management

Kubernetes must satisfy the following four network rules:

1. All pods must be able to communicate directly with each other via IP.
2. All nodes must be able to communicate with all pods.
3. Direct communication without NAT (Pod IP is not a virtual IP).
4. Pods must be routed correctly when communicating externally.

To achieve this, each pod needs a unique IP address, and pods must be able to communicate even if they are on different nodes.

Implementing all these tasks directly in Kubernetes would be too complex, so Kubernetes only provides CNI and delegates network implementation to plugins.

CNI Overall Architecture #

The configuration largely consists of three parts:

1. container runtime (e.g., containerd, CRI-O): Calls CNI when creating a pod.
2. CNI Plugin (Calico, Flannel, Cilium, Weave Net): Performs actual network interface creation, routing, overlay, ECMP, etc.
3. IPAM Plugin: Responsible for IP address allocation policies.

ECMP (Equal-Cost Multi Path) is a routing method in which traffic is distributed across multiple paths when there are several paths with equal cost (=same metric) in a network.

When kubelet requests the runtime (container) to create a pod, containerd executes an ADD request (JSON payload) to CNI. CNI then creates a veth pair, connecting one end to the pod's namespace and the other to the host and overlay endpoint. Afterward, IPAM assigns an IP address to the pod and configures routing tables, iptables, BPF, etc.

When a pod is deleted, a DEL request is sent, and it is removed.

CNI Plugin #

• Flannel
  • Simplest, VXLAN-based overlay.
  • No security features, suitable for small clusters.
• Calico
  • Most widely used in enterprises.
  • BGP-based routing with powerful NetworkPolicy.
  • Can operate without an overlay.
• Cilium
  • High-performance, eBPF-based.
  • Capable of L3/L4/L7 policies.
  • Supports service mesh without sidecars.
  • The next-generation standard.
• Weave Net
  • Easy installation in small environments.
  • Lower performance than Calico, Cilium.

CNI Directory Structure #

The default locations in Kubernetes are as follows:

# 설정 파일 위치(예: 10-calico.conflist)
/etc/cni/net.d/

# cni 플러그인 바이너리 위치(calico, flannel 등)
/opt/cni/bin/

# kubelet config
--cni-conf-dir=/etc/cni/net.d
--cni-bin-dir=/opt/cni/bin

Pod-Node Communication Process #

I will explain based on Calico.

1. A pod connects to the host namespace via a veth pair.
2. Routing for that Pod IP is registered on the host.
3. When communicating with a pod on another node, routing information is exchanged via BGP, and then an L3-level connection is directly established.

This means communication occurs at the actual L3 network level without an overlay.

If CNI is misconfigured, a pod might be running but not ready, pings might fail, CoreDNS might enter a CrashLoop, a Node might be NotReady, cni0, vxlan, calico interfaces might fail, or veth interfaces might be in a dangling state when checked with ip -d link.

Common causes among these include:

• IPAM pool exhaustion.
• CNI configuration conflicts.
• Host routing table corruption.
• AWS/VPC and CNI routing conflicts.
• MTU Mismatch (when using Overlay (VXLAN), the actual MTU can become less than 1500, potentially causing IP fragmentation).