How Docker Overlay Networks Work: VXLAN Under the Hood

Introduction

Overlay networks allow containers running on entirely different hosts to communicate directly over a flat, secure, layer 2 network, regardless of the underlying physical network topology. Docker builds this capability on top of libnetwork and its native overlay driver, making it simple to configure while remaining secure by default.

Building a Swarm

Overlay networks rely on the swarm's key-value store and security features, so a Swarm cluster is required. A two-node swarm can be created by initializing the first node and joining the second with the generated token:

docker swarm init
docker swarm join --token SWMTKN-1-... 172.31.1.5:2377

Communication between nodes requires several ports to be open: 2377/tcp for management traffic, 7946/tcp and 7946/udp for control plane gossip, and 4789/udp for VXLAN data plane traffic.

Creating an Overlay Network

An encrypted overlay network spanning the entire swarm can be created with a single command:

docker network create -d overlay -o encrypted uber-net

The -o encrypted flag ensures both control and data plane traffic are encrypted using AES in GCM mode, with encryption keys rotated automatically every 12 hours. Without it, only control plane traffic is encrypted by default, since encrypting the data plane can reduce performance by roughly 10%.

Interestingly, a newly created overlay network is only visible on the manager node until a worker node actually runs a container that needs it — Docker extends overlay networks lazily to reduce unnecessary network gossip across the cluster.

Attaching Containers and Testing Connectivity

By default, only containers belonging to swarm services can join overlay networks, unless the network is created with the --attachable flag. Deploying a service with replicas spread across nodes demonstrates this in action:

docker service create --name test \
  --network uber-net \
  --replicas 2 \
  ubuntu sleep infinity

Once deployed, containers on different physical nodes can ping each other directly using their overlay network IPs, and tracing the route confirms a single hop — proof that the containers communicate directly over the overlay, completely unaware of the underlying network topology they're actually traversing.

How Overlay Networks Work: VXLAN

Under the hood, Docker builds overlay networks using VXLAN (Virtual Extensible LAN) tunnels, which create a virtual layer 2 network on top of existing layer 3 infrastructure. Because VXLAN is an encapsulation technology, it's completely transparent to the routers and switches of the underlay network — they simply see regular IP/UDP traffic.

Each end of the tunnel terminates in a VXLAN Tunnel Endpoint (VTEP), responsible for encapsulating and de-encapsulating traffic as it enters and exits the tunnel. On each host, Docker creates a network sandbox with a virtual switch, connects a VTEP to it, and binds the VTEP's host-facing side to UDP port 4789. The two VTEPs then form the VXLAN tunnel that carries all overlay traffic between hosts.

Tracing a Packet's Journey

When one container pings another across the overlay, its local bridge learns how to reach the destination via the VTEP through Docker's gossip protocol, which propagates container information across the swarm. The VTEP encapsulates the packet with a VXLAN header identifying the correct VXLAN Network ID (VNID), wraps it in a UDP packet addressed to the remote VTEP, and sends it across the underlay network. On arrival, the receiving host's VTEP de-encapsulates the packet and forwards it to the correct destination container — all invisible to the underlying physical network.

Docker's overlay driver also supports layer 3 routing between multiple subnets on a single overlay network, handled automatically without additional configuration.

Conclusion

What looks like a single simple command — docker network create -d overlay — hides a genuinely sophisticated system of virtual switches, tunnel endpoints, and encapsulation working together to let containers communicate seamlessly across any underlying network. Understanding this machinery is valuable both for troubleshooting production Swarm deployments and for informed conversations with networking teams.