Pod to Pod Cross-Node Packet Flow

In k8s, Pod to Pod cross-node communication ultimately follows the virtual network created by the CNI + kernel routing + iptables/ipvs rules set by the CNI.

Let's look at the basic flow that applies commonly to Flannel/VXLAN, Calico, and Cilium.

The overall packet flow summary is:

pod A -> veth -> Node1 kernel -> CNI routing -> Node2 NodeIP -> Node2 kernel -> veth -> Pod B

The packet originates from Pod eth0 (one side of the veth pair), with the destination being the IP of the peer pod (e.g., 10.244.2.15).

The transmission from the pod's veth to the node's veth is a veth pair, where the pod side is eth0 and the node side is vethXXXX.

Packets originating from the Pod are directly forwarded to the vethXXX in the node's namespace. This happens automatically at the Linux kernel level.

The Node kernel checks the routing table and uses the CNI plugin information on the node.

# ex flannel
10.244.1.0/24 via 192.168.1.12 dev flannel.1
10.244.2.0/24 via 192.168.1.13 dev flannel.1

# ex Calico, BGP
10.244.2.0/24 via 192.168.1.13 dev enp1s0

The Node kernel checks the route to determine if the destination pod IP belongs to the local node or another node.

If the destination is another node, CNI Overlay or Routing operations occur. This is where the technologies differ:

Flannel VXLAN: Node1 encapsulates the packet in VXLAN (UDP 8472) and sends it to Node2.

Pod → veth → Node1 → flannel0 → VXLAN encapsulate → UDP packet sent to Node2.IP

Calico (BGP/no-overlay): In the Node's routing table, 10.244.2.0/24 -> Node2.IP is directly configured like this.

It performs L3 routing directly without VXLAN.

Pod → veth → Node1 → enp1s0 → sent to Node2.IP as destination

Cilium (eBPF): It rarely uses iptables and applies routing rules directly via eBPF XDP/TC Hooks.

This varies depending on the configuration, such as VXLAN, Geneve, or direct routing.

Upon arrival at Node2, the kernel processes the packet. Node2 knows its own pod CIDR, receives the packet, and locates the destination pod.

Forwarded to Node2's veth -> arrives at PodB.

Destination 10.244.2.15 → vethABCD → Pod B

Simplified diagram of the entire process:

[Pod A]
   ↓ (veth)
[Node1 kernel]
   ↓ (CNI route)
[Node1 → Node2 transmission]   ← VXLAN or BGP or Geneve
   ↓
[Node2 kernel]
   ↓ (veth)
[Pod B]

iptables/ipvs/eBPF Involvement Points #

Direct pod-to-pod communication rarely involves iptables/ipvs unless it goes through a service.

However, if accessed via ClusterIP:

iptables (kube-proxy) performs DNAT from service -> endpoints -> pod. It uses connection tracking (conntrack).

In IPVS mode, it acts like an L4 load balancer, minimizing conntrack, and supports DR, TUN, and NAT modes.

eBPF mode (Cilium): It doesn't use iptables and uses its own BPF conntrack.

It performs Service LB directly at the kernel level.

Why Node-to-Node Communication is Possible #

Each Node has its own pod CIDR, and the CNI automatically synchronizes the extensive routing tables between nodes.

When the kernel sees the destination pod IP, it determines which node it belongs to and performs cross-node transmission via overlay or L3 routing.

In summary, pod-to-pod communication involves:

1. The CNI prepares veth and CIDR.
2. The node's kernel routing knows the cross-node path.
3. Transmission via overlay or direct routing.

It consists of these three elements.