k8s实践10:docker网桥和pod连接网络结构

1.1
使用到的组件

network namespace
bridge
veth pair

注意事项:所有命令都是在集群master节点操作测试.

1.2

network namespace

执行命令ip netns会显示创建的network namespace
如果显示空白,请做link,命令见下:

[root@k8s-master2 ~]# ln -s /var/run/docker/netns/ /var/run/netns

[root@k8s-master1 ~]# ip netns
default
[root@k8s-master1 ~]#

[root@k8s-master2 ~]# ip netns
netns
290d0c6ad2ab (id: 3)
49bfcc08baa8 (id: 2)
aeb6eaef384f (id: 1)
8d103e06202b (id: 0)
default
[root@k8s-master2 ~]#

[root@k8s-master3 ~]# ip netns
789ea14c2216 (id: 4)
a4dbfec964e5 (id: 3)
4bdb1409b6e3 (id: 2)
daea28f1e8b2 (id: 1)
0566deb27814 (id: 0)
default
[root@k8s-master3 ~]#

可以看到每个节点显示的内容不同,默认只有一个default.

 1.3
为什么每个节点显示的不同？

以master3做示例,见下

[root@k8s-master3 ~]# ip a |grep veth
8: veth3a37cd9@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
10: veth37e9e7e@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
12: veth4ed5bcb@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
14: vethc4d2905@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
16: vethd67ea28@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
[root@k8s-master3 ~]#

[root@k8s-master3 ~]# ip netns
789ea14c2216 (id: 4)
a4dbfec964e5 (id: 3)
4bdb1409b6e3 (id: 2)
daea28f1e8b2 (id: 1)
0566deb27814 (id: 0)
default
[root@k8s-master3 ~]#

注意,一个netns对应了一个虚拟的网卡.怎么对应的呢？
示例,见下:

[root@k8s-master3 ~]# ip netns exec  0566deb27814 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:ac:1e:5b:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.30.91.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe1e:5b02/64 scope link
       valid_lft forever preferred_lft forever

请看这条:

eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP

对应的是:

8: veth3a37cd9@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP

这是一个veth pair对.
可用命令

[root@k8s-master3 ~]# ethtool -i veth3a37cd9
driver: veth
version: 1.0
firmware-version:
expansion-rom-version:
bus-info:
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[root@k8s-master3 ~]# ethtool -S veth3a37cd9
NIC statistics:
     peer_ifindex: 7
[root@k8s-master3 ~]#

回到开始的问题,为什么每个节点显示的不同呢？
network namespace是怎么来的呢？

先敲命令做显示对比.

pod

[root@k8s-master3 ~]# kubectl get pod -n kube-system -o wide
NAME                                          READY     STATUS    RESTARTS   AGE       IP               NODE
coredns-779ffd89bd-5znxf                      1/1       Running   2          18h       172.30.91.3      k8s-master3
kubernetes-dashboard-65c76f6c97-4kg2r         1/1       Running   3          18h       172.30.91.5      k8s-master3
tiller-deploy-58d57fd669-w8wsz                1/1       Running   3          18h       172.30.91.2      k8s-master3
traefik-ingress-controller-77b549b5d9-4xd58   1/1       Running   4          4d        192.168.32.130   k8s-master3
traefik-ingress-controller-77b549b5d9-dj65f   1/1       Running   10         34d       192.168.32.129   k8s-master2
traefik-ingress-controller-77b549b5d9-hpt4z   1/1       Running   11         34d       192.168.32.128   k8s-master1
[root@k8s-master3 ~]#

pod的ip

[root@k8s-master3 ~]# kubectl exec -it tiller-deploy-58d57fd669-w8wsz /bin/sh -n kube-system
~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:ac:1e:5b:02 brd ff:ff:ff:ff:ff:ff
    inet 172.30.91.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe1e:5b02/64 scope link
       valid_lft forever preferred_lft forever

eth0@if8,看看这个network namespace的ip情况

[root@k8s-master3 ~]# ip netns exec  0566deb27814 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:ac:1e:5b:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.30.91.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe1e:5b02/64 scope link
       valid_lft forever preferred_lft forever

一个pod对应了一个network namespace,pod使用的网络连接口就是network namespace的网络口.

1.4

bridge

执行命令,见下:

[root@k8s-master3 ~]# brctl show
bridge name        bridge id                STP enabled        interfaces
docker0                8000.0242c5980278        no                veth37e9e7e
                                                        veth3a37cd9
                                                        veth4ed5bcb
                                                        vethc4d2905
                                                        vethd67ea28

[root@k8s-master3 ~]# ip a |grep veth
8: veth3a37cd9@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
10: veth37e9e7e@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
12: veth4ed5bcb@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
14: vethc4d2905@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
16: vethd67ea28@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP

结论:
这些虚拟网卡每个对应的eth pair一端(另一端在namaspace里其实也就是在pod里,参考上面的分析),同时这些虚拟网卡桥接到了网桥docker上.

1.5
网桥docker和pod的逻辑拓扑关系图

拓扑见下: