Upstream discussion to try to figure out which tool to use to set up networks for Atomic/Docker.

Thoughts, comments, suggestions?

-------- Forwarded Message -------- Subject: [atomic-devel] recommending Flannel (w/ vxlan backend) for atomic -- thoughts? Date: Tue, 18 Nov 2014 14:36:03 -0500 From: John W. Linville linville@redhat.com To: atomic-devel@projectatomic.io

Greetings,

Internally we've been doing a little looking at projects for setting-up overlay networks between minions in a Kubernetes cluster. One of the most interesting options has been Flannel (formerly Rudder). Flannel requires minimal configuration to slice a large subnet into a series of smaller subnets, one per minion running flanneld.

Flannel uses a configuration stored as a JSON file in etcd. The JSON configuration looks a bit like this:

{ "Network": "192.168.88.0/24", "SubnetLen": 28, "Backend": { "Type": "vxlan" } }

The above configuration would allow up to 16 minions to each allocate a /28 subnet for use by their local docker daemon. (Larger or smaller subnets are, of course, a simple matter of configuration.) The local configuration information is written by flanneld to a file under /var/run, and the info is used to pass the --bip option to docker so that it configures its docker0 bridge appropriately.

Beyond that, the vxlan backend for flanneld on each minion creates a vxlan tunnel endpoint and configures it to use the DOVE extensions for routing. The route to the larger (e.g. /24) subnet points at the vxlan interface, so traffic to other minions is directed through it. Such traffic triggers L2MISS and L3MISS messages that are handled by flanneld, directing traffic to the appropriate minions.

The result is a vxlan-based overlay network that enables connectivity between all the minions (and their pods) with a minimal amount of configuration required. This seems like a powerful and usable means to enable this communication.

Given the description above (and whatever other sources you might have at your disposal), does anyone have any objections to using this as a default Kubernetes networking solution in Atomic? Or any questions about the use of Flannel in general?

Thanks,

John