Container Networking Guide: Part I

This is the first part of the series Container Networking. I will cover Virtual Ethernet devices in this blog post. Generally, any machine has loopback and ethernet network interfaces. You can check the available interfaces using ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff enp0s3 is an interface of type ethernet. You can communicate with machines outside the VM using this interface. Let’s create a Virtual Ethernet device. veth man page says The veth devices are Virtual Ethernet devices. They can act as tunnels between network namespaces to create a bridge to a physical network device in another namespace, but can also be used as standalone network devices. We will see namespaces and bridges later, but let us see how we can create veth interface(s) and their usage. Creating veth Pair sudo ip link add vethX type veth peer name vethY ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff By default, they are down. You need to make them up. sudo ip link set vethX up sudo ip link set vethY up ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff The veth pair is very special. Packets transmitted on one device in the pair are immediately received on the other device. Traffic on Veth Pair To send the packet using veth, we will use the Scapy tool. It is a Python-based interactive packet manipulation program and library. The code to send a single packet looks like cat > onepkt.py #! /usr/bin/env python3 import sys from scapy.all import * if __name__ == '__main__': usage_string = """Usage: onepkt.py where is unique identifier for this message""" # total arguments if (len(sys.argv) != 5): sys.exit("Incorrect usage - num args.\n"+usage_string) src_mac = sys.argv[1] dst_mac = sys.argv[2] iface = sys.argv[3] msg = sys.argv[4] src_ip = "1.1.1.1" dst_ip = "2.2.2.2" sport = 1111 dport = 2222 pkt = Ether(dst=dst_mac, src=src_mac) / IP (src=src_ip, dst=dst_ip) / TCP(sport=sport, dport=dport) / msg pkt.show() sendp(pkt, iface=iface) EOF Code Credits: https://github.com/eric-keller/npp-linux-01-intro/blob/main/demo3/onepkt.py In one terminal window, start the tshark on vethY sudo tshark -T fields -e eth -i vethY Running as user "root" and group "root". This could be dangerous. Capturing on 'vethY' ** (tshark:9349) 10:26:21.105768 [Main MESSAGE] -- Capture started. ** (tshark:9349) 10:26:21.105897 [Main MESSAGE] -- File: "/tmp/wireshark_vethY4R8LY2.pcapng" Now, run the following Python code to send the single packet to vethX device. sudo python3 ./onepkt.py 22:11:11:11:11:11 22:22:22:22:22:22 vethX 123 ###[ Ethernet ]### dst = 22:22:22:22:22:22 src = 22:11:11:11:11:11 type = IPv4 ###[ IP ]### version = 4 ihl = None tos = 0x0 len = None id = 1 flags = frag = 0 ttl = 64 proto = tcp chksum = None src = 1.1.1.1 dst = 2.2.2.2 \options \ ###[ TCP ]### sport = 1111 dport = 2222 seq = 0 ack = 0 dataofs = None reserved = 0 flags = S window = 8192 chksum = None urgptr = 0 options = [] ###[ Raw ]### load = '123' . Sent 1 packets. On the tshark window, you will see the packet has been received. sudo tshark -T fields -e eth -i vethY Running as user "root" and group "root". This could be dangerous. Capturing on 'vethY' ** (tshark:9349) 10:26:21.105768 [Main MESSAGE] -- Capture started. ** (tshark:9349) 10:26:21.105897 [Main MESSAGE] -- File: "/tmp/wireshark_vethY4R8LY2.pcapng" Ethernet II, Src: 22:11:11:11:11:11 (22:11:11:11:11:11), Dst: 22:22:22:22:22:22 (22:22:22:22:22:22) The veth pairs play a critical role in establishing Container Networking. We will see it in the next part of the blog. More on Veth If you have many veth pairs, then given a veth device how can you identify the peer? ethtool utility comes to the rescue. ethtool tells us the peer’s index. ethtool -S vethX | grep peer peer_ifindex: 3 peer_ifindex: 3 indicates an index of peer devices. The index is shown in ip a command. ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff Although device name vethX@vethY tells the peer name, but when devices are created by container software (like docker, podman, etc.), the device names are not that straightforward. You can get detailed, prettier information about a veth device using ip -d -j -p link show vethX [ { "ifindex": 4, "link": "vethY", "ifname": "vethX", "flags": [ "BROADCAST","MULTICAST","UP","LOWER_UP" ], "mtu": 1500, "qdisc": "noqueue", "operstate": "UP", "linkmode": "DEFAULT", "group": "default", "txqlen": 1000, "link_type": "ether", "address": "5e:4b:13:90:7d:63", "broadcast": "ff:ff:ff:ff:ff:ff", "promiscuity": 0, "min_mtu": 68, "max_mtu": 65535, "linkinfo": { "info_kind": "veth" }, "inet6_addr_gen_mode": "eui64", "num_tx_queues": 2, "num_rx_queues": 2, "gso_max_size": 65536, "gso_max_segs": 65535 } ] With this, we come to the end of this blog post. Next in the series is Network Namespaces and Bridges This is the first part of the series Container Networking . I will cover Virtual Ethernet devices in this blog post. Container Networking Generally, any machine has loopback and ethernet network interfaces. You can check the available interfaces using ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff enp0s3 is an interface of type ethernet. You can communicate with machines outside the VM using this interface. enp0s3 Let’s create a Virtual Ethernet device. veth man page says veth man page The veth devices are Virtual Ethernet devices. They can act as tunnels between network namespaces to create a bridge to a physical network device in another namespace, but can also be used as standalone network devices. The veth devices are Virtual Ethernet devices. They can act as tunnels between network namespaces to create a bridge to a physical network device in another namespace, but can also be used as standalone network devices. We will see namespaces and bridges later, but let us see how we can create veth interface(s) and their usage. veth Creating veth Pair sudo ip link add vethX type veth peer name vethY sudo ip link add vethX type veth peer name vethY ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff By default, they are down. You need to make them up. sudo ip link set vethX up sudo ip link set vethY up sudo ip link set vethX up sudo ip link set vethY up ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff The veth pair is very special. Packets transmitted on one device in the pair are immediately received on the other device. veth Traffic on Veth Pair To send the packet using veth , we will use the Scapy tool. It is a Python-based interactive packet manipulation program and library. veth Scapy The code to send a single packet looks like cat > onepkt.py #! /usr/bin/env python3 import sys from scapy.all import * if __name__ == '__main__': usage_string = """Usage: onepkt.py where is unique identifier for this message""" # total arguments if (len(sys.argv) != 5): sys.exit("Incorrect usage - num args.\n"+usage_string) src_mac = sys.argv[1] dst_mac = sys.argv[2] iface = sys.argv[3] msg = sys.argv[4] src_ip = "1.1.1.1" dst_ip = "2.2.2.2" sport = 1111 dport = 2222 pkt = Ether(dst=dst_mac, src=src_mac) / IP (src=src_ip, dst=dst_ip) / TCP(sport=sport, dport=dport) / msg pkt.show() sendp(pkt, iface=iface) EOF cat > onepkt.py #! /usr/bin/env python3 import sys from scapy.all import * if __name__ == '__main__': usage_string = """Usage: onepkt.py where is unique identifier for this message""" # total arguments if (len(sys.argv) != 5): sys.exit("Incorrect usage - num args.\n"+usage_string) src_mac = sys.argv[1] dst_mac = sys.argv[2] iface = sys.argv[3] msg = sys.argv[4] src_ip = "1.1.1.1" dst_ip = "2.2.2.2" sport = 1111 dport = 2222 pkt = Ether(dst=dst_mac, src=src_mac) / IP (src=src_ip, dst=dst_ip) / TCP(sport=sport, dport=dport) / msg pkt.show() sendp(pkt, iface=iface) EOF Code Credits: https://github.com/eric-keller/npp-linux-01-intro/blob/main/demo3/onepkt.py In one terminal window, start the tshark on vethY tshark vethY sudo tshark -T fields -e eth -i vethY Running as user "root" and group "root". This could be dangerous. Capturing on 'vethY' ** (tshark:9349) 10:26:21.105768 [Main MESSAGE] -- Capture started. ** (tshark:9349) 10:26:21.105897 [Main MESSAGE] -- File: "/tmp/wireshark_vethY4R8LY2.pcapng" sudo tshark -T fields -e eth -i vethY Running as user "root" and group "root". This could be dangerous. Capturing on 'vethY' ** (tshark:9349) 10:26:21.105768 [Main MESSAGE] -- Capture started. ** (tshark:9349) 10:26:21.105897 [Main MESSAGE] -- File: "/tmp/wireshark_vethY4R8LY2.pcapng" Now, run the following Python code to send the single packet to vethX device. vethX sudo python3 ./onepkt.py 22:11:11:11:11:11 22:22:22:22:22:22 vethX 123 ###[ Ethernet ]### dst = 22:22:22:22:22:22 src = 22:11:11:11:11:11 type = IPv4 ###[ IP ]### version = 4 ihl = None tos = 0x0 len = None id = 1 flags = frag = 0 ttl = 64 proto = tcp chksum = None src = 1.1.1.1 dst = 2.2.2.2 \options \ ###[ TCP ]### sport = 1111 dport = 2222 seq = 0 ack = 0 dataofs = None reserved = 0 flags = S window = 8192 chksum = None urgptr = 0 options = [] ###[ Raw ]### load = '123' . Sent 1 packets. sudo python3 ./onepkt.py 22:11:11:11:11:11 22:22:22:22:22:22 vethX 123 ###[ Ethernet ]### dst = 22:22:22:22:22:22 src = 22:11:11:11:11:11 type = IPv4 ###[ IP ]### version = 4 ihl = None tos = 0x0 len = None id = 1 flags = frag = 0 ttl = 64 proto = tcp chksum = None src = 1.1.1.1 dst = 2.2.2.2 \options \ ###[ TCP ]### sport = 1111 dport = 2222 seq = 0 ack = 0 dataofs = None reserved = 0 flags = S window = 8192 chksum = None urgptr = 0 options = [] ###[ Raw ]### load = '123' . Sent 1 packets. On the tshark window, you will see the packet has been received. tshark sudo tshark -T fields -e eth -i vethY Running as user "root" and group "root". This could be dangerous. Capturing on 'vethY' ** (tshark:9349) 10:26:21.105768 [Main MESSAGE] -- Capture started. ** (tshark:9349) 10:26:21.105897 [Main MESSAGE] -- File: "/tmp/wireshark_vethY4R8LY2.pcapng" Ethernet II, Src: 22:11:11:11:11:11 (22:11:11:11:11:11), Dst: 22:22:22:22:22:22 (22:22:22:22:22:22) sudo tshark -T fields -e eth -i vethY Running as user "root" and group "root". This could be dangerous. Capturing on 'vethY' ** (tshark:9349) 10:26:21.105768 [Main MESSAGE] -- Capture started. ** (tshark:9349) 10:26:21.105897 [Main MESSAGE] -- File: "/tmp/wireshark_vethY4R8LY2.pcapng" Ethernet II, Src: 22:11:11:11:11:11 (22:11:11:11:11:11), Dst: 22:22:22:22:22:22 (22:22:22:22:22:22) The veth pairs play a critical role in establishing Container Networking. We will see it in the next part of the blog. veth next part More on Veth If you have many veth pairs, then given a veth device how can you identify the peer? ethtool utility comes to the rescue. ethtool tells us the peer’s index. veth veth ethtool ethtool ethtool -S vethX | grep peer peer_ifindex: 3 ethtool -S vethX | grep peer peer_ifindex: 3 peer_ifindex: 3 indicates an index of peer devices. The index is shown in ip a command. peer_ifindex: 3 ip a ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff 3: vethY@vethX: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5a:bc:4d:7e:76:b1 brd ff:ff:ff:ff:ff:ff 4: vethX@vethY: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 5e:4b:13:90:7d:63 brd ff:ff:ff:ff:ff:ff Although device name vethX@vethY tells the peer name, but when devices are created by container software (like docker, podman, etc.), the device names are not that straightforward. vethX@vethY You can get detailed, prettier information about a veth device using veth ip -d -j -p link show vethX [ { "ifindex": 4, "link": "vethY", "ifname": "vethX", "flags": [ "BROADCAST","MULTICAST","UP","LOWER_UP" ], "mtu": 1500, "qdisc": "noqueue", "operstate": "UP", "linkmode": "DEFAULT", "group": "default", "txqlen": 1000, "link_type": "ether", "address": "5e:4b:13:90:7d:63", "broadcast": "ff:ff:ff:ff:ff:ff", "promiscuity": 0, "min_mtu": 68, "max_mtu": 65535, "linkinfo": { "info_kind": "veth" }, "inet6_addr_gen_mode": "eui64", "num_tx_queues": 2, "num_rx_queues": 2, "gso_max_size": 65536, "gso_max_segs": 65535 } ] ip -d -j -p link show vethX [ { "ifindex": 4, "link": "vethY", "ifname": "vethX", "flags": [ "BROADCAST","MULTICAST","UP","LOWER_UP" ], "mtu": 1500, "qdisc": "noqueue", "operstate": "UP", "linkmode": "DEFAULT", "group": "default", "txqlen": 1000, "link_type": "ether", "address": "5e:4b:13:90:7d:63", "broadcast": "ff:ff:ff:ff:ff:ff", "promiscuity": 0, "min_mtu": 68, "max_mtu": 65535, "linkinfo": { "info_kind": "veth" }, "inet6_addr_gen_mode": "eui64", "num_tx_queues": 2, "num_rx_queues": 2, "gso_max_size": 65536, "gso_max_segs": 65535 } ] With this, we come to the end of this blog post. Next in the series is Network Namespaces and Bridges Network Namespaces and Bridges