openstack(pike) dvr 中南北数据流向分析

1  在DVR环境中各组件的连接情况如图


2 带floatingip,从外部访问虚机，ping floatingip 192.168.100.190
此时数据流如图所示


2.1 获取MAC
外网中的机器首先要通过 ARP 获取虚机浮动 IP 对应的 MAC 地址。浮动 IP 并没有配置在 fip 的端口上，因此 fip 无法直接响应 ARP 请求，那怎么办呢？Neutron 在 fip NS 的 fg 端口上配置了 arp proxy，这样，fip 既可以响应它自己的 interface 上的 IP 地址的 ARP 请求，也可以响应能通过它路由到的 IP 地址的 ARP 请求,
fip netns上配置的proxy_arp
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e sysctl net.ipv4.conf.fg-42d94dcb-8b.proxy_arp
net.ipv4.conf.fg-42d94dcb-8b.proxy_arp = 1
fip netns 收到 ARP 请求后，将其 fg interface 的 MAC 地址返回,外网中的机器获取到虚机浮动 IP 的 MAC 地址后，发出 ICMP 网络包
2.2 fip netns 路由到下qrouter netns
网络包经过 br-ex，被 fip 的 fg 端口收到，查路由表，命中第一条路由，从其 fpr interface 发出，到达 169.254.106.114.
ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route
169.254.106.114/31 dev fpr-b2ce2e13-1 proto kernel scope link src 169.254.106.115 
192.168.100.0/24 dev fg-42d94dcb-8b proto kernel scope link src 192.168.100.184 
192.168.100.190 via 169.254.106.114 dev fpr-b2ce2e13-1 
fpr interface 和rfp interface 是一堆veth pair , fpr interface 发出的包被 veth 另一端的 qrouter 的 rfp-b2ce2e13-1 interface 收到。
可以看到qrouter netns 的 rfp-b2ce2e13-1口地址是169.254.106.114/31


[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: rfp-b2ce2e13-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b2:e7:86:35:3d:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.114/31 scope global rfp-b2ce2e13-1
       valid_lft forever preferred_lft forever
    inet6 fe80::b0e7:86ff:fe35:3dac/64 scope link 
       valid_lft forever preferred_lft forever
134: qr-d2b8d53f-64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:02:97:ab brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global qr-d2b8d53f-64
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe02:97ab/64 scope link 
       valid_lft forever preferred_lft forever
2.3 DNAT
在 qrouter 上，首先做 DNAT
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
DNAT后，dest ip 改为172.17.0.7
2.4 再次路由
查 qrouter 的 main 路由表，命中第二条，从qr-d2b8d53f-64 发出
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route
169.254.106.114/31 dev rfp-b2ce2e13-1 proto kernel scope link src 169.254.106.114 
172.17.0.0/16 dev qr-d2b8d53f-64 proto kernel scope link src 172.17.0.1 
数据包经qr-d2b8d53f-64,发到br-int 到达虚机


3 带floatingip从虚机访问外网，从172.17.0.7 ping 8.8.8.8
 3.1路由 
进入qroute netns,查看路由规则走route 16表
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
57483:from 172.17.0.7 lookup 16 
2886795265:from 172.17.0.1/16 lookup 2886795265 


[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 16
default via 169.254.106.115 dev rfp-b2ce2e13-1 
3.2 SNAT
路由后会通过netfilter的POSTROUTING链中进行SNAT，将源IP改为192.168.100.190
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-b2ce2e13-1 ! -o rfp-b2ce2e13-1 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
-A neutron-l3-agent-float-snat -s 172.17.0.7/32 -j SNAT --to-source 192.168.100.190
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat


从rfp-b2ce2e13-1口将包发到169.254.106.115，rfp-b2ce2e13-1和fip netns的fpr-b2ce2e13-1，是一对veth pair，fpr-b2ce2e13-1收到数据包
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: fpr-b2ce2e13-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:f7:5f:31:67:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.115/31 scope global fpr-b2ce2e13-1
       valid_lft forever preferred_lft forever
    inet6 fe80::a8f7:5fff:fe31:67db/64 scope link 
       valid_lft forever preferred_lft forever




135: fg-42d94dcb-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:c0:dc:94 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.184/24 brd 192.168.100.255 scope global fg-42d94dcb-8b
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec0:dc94/64 scope link 
       valid_lft forever preferred_lft forever
3.3 数据包在fip netns 中再次路由
包通过fpr-b2ce2e13-1发到169.254.106.115，后再在fip netns中查路由规则，路由
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
2852022899:from all iif fpr-b2ce2e13-1 lookup 2852022899 
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route list table 2852022899
default via 192.168.100.1 dev fg-42d94dcb-8b 
从fg-42d94dcb-8b 口将包发到192.168.100.1，再有192.168.100.1继续路由转发


4 不带浮动IP，虚机访问外网
 4.1 计算节点qrouter 路由
查看路由规则
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
2886795265:from 172.17.0.1/16 lookup 2886795265 
查看路由表2886795265
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 2886795265
default via 172.17.0.10 dev qr-d2b8d53f-64
只有一条策略，经过qr口，发到下一个路由172.17.0.10
172.17.0.10　路由是在network节点的snat netns sg口
[root@network ~]# ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
93: sg-935eff5f-b8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:bf:6d:b5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.10/16 brd 172.17.255.255 scope global sg-935eff5f-b8
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:febf:6db5/64 scope link 
       valid_lft forever preferred_lft forever
94: qg-2b2169ee-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:9d:5c:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.187/24 brd 192.168.100.255 scope global qg-2b2169ee-a5
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe9d:5cbd/64 scope link 
       valid_lft forever preferred_lft forever
4.2 SNAT
network 节点的snat netns 做SNAT
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-A neutron-l3-agent-snat -o qg-2b2169ee-a5 -j SNAT --to-source 192.168.100.187
4.3 snat netns路由
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
没有额外策略，走main表
 ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table main
default via 192.168.100.1 dev qg-2b2169ee-a5 
172.17.0.0/16 dev sg-935eff5f-b8 proto kernel scope link src 172.17.0.10 
192.168.100.0/24 dev qg-2b2169ee-a5 proto kernel scope link src 192.168.100.187 
根据目的地址，命中第一条，包从qg-2b2169ee-a5口出去，再从192.168.100.1网关上继续路由