Neutron L3 Agent packet flow

介绍一下neutron-l3-agent使用时，虚机的数据流。

Matt_Mao

3339人浏览 · 2014-03-25 15:37:22

Matt_Mao · 2014-03-25 15:37:22 发布

部署环境如下：

目前我们将neutron-l3-agent放在controller节点上，之后会将其移至专属的network节点，专属的network节点的网卡能力更强些。

以下是网络逻辑拓扑：

从虚机到计算节点的eth1的packet flow与provider network是一致的，可以参照：

http://blog.csdn.net/matt_mao/article/details/17231045

不同的地方是虚机获取的网关地址是qr-XXX的ip地址。这样虚机的数据流就会进入controller节点。

例如：

# ip netns exec qrouter-942b3c47-0c90-44c9-b01d-8c6c7f4368a1 ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
 
qg-123e8e11-94 Link encap:Ethernet  HWaddr FA:16:3E:5F:96:A1
          inet addr:10.224.159.170  Bcast:10.224.159.191  Mask:255.255.255.224
          inet6 addr: fe80::f816:3eff:fe5f:96a1/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:899 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:56096 (54.7 KiB)  TX bytes:941 (941.0 b)
 
qr-f1977c17-37 Link encap:Ethernet  HWaddr FA:16:3E:F6:C9:36
          inet addr:192.168.76.1  Bcast:192.168.76.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fef6:c936/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:25 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2375 (2.3 KiB)  TX bytes:1208 (1.1 KiB)

虚机的默认网关地址是qr-f1977c17-37的ip地址，数据流就会进入controller的这个netns，然后经过内核的netfilter进行SNAT后，如果你没有设置floating ip，则源地址变为qg-123e8e11-94的地址。

[root@ci91szcmp001 ~]# ip netns exec qrouter-942b3c47-0c90-44c9-b01d-8c6c7f4368a1 iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 960 packets, 46441 bytes)
 pkts bytes target     prot opt in     out     source               destination
  960 46441 neutron-l3-agent-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    3   221 neutron-l3-agent-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    3   221 neutron-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 neutron-l3-agent-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 
Chain neutron-l3-agent-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
 
Chain neutron-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  !qg-123e8e11-94 !qg-123e8e11-94  0.0.0.0/0            0.0.0.0/0           ! ctstate DNAT
 
Chain neutron-l3-agent-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            169.254.169.254     tcp dpt:80 redir ports 9697
 
Chain neutron-l3-agent-float-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination
 
Chain neutron-l3-agent-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination
    3   221 neutron-l3-agent-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    3   221 SNAT       all  --  *      *       192.168.76.0/24      0.0.0.0/0           to:10.224.159.170
 
Chain neutron-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source               destination
    3   221 neutron-l3-agent-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0

如果设置了floating ip，则在接口qg-123e8e11-94上可以找到这个floating ip：

# ip netns exec qrouter-942b3c47-0c90-44c9-b01d-8c6c7f4368a1 ip route
10.224.159.160/27 dev qg-123e8e11-94  proto kernel  scope link  src 10.224.159.170
192.168.76.0/24 dev qr-f1977c17-37  proto kernel  scope link  src 192.168.76.1
default via 10.224.159.161 dev qg-123e8e11-94

并且会添加以下SNAT和DNAT规则：

[root@ci91szcmp001 ~]# ip netns exec qrouter-942b3c47-0c90-44c9-b01d-8c6c7f4368a1 iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 1101 packets, 53272 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1101 53272 neutron-l3-agent-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    6   428 neutron-l3-agent-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    6   428 neutron-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 neutron-l3-agent-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 
Chain neutron-l3-agent-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       all  --  *      *       0.0.0.0/0            10.224.159.171      to:192.168.76.2
 
Chain neutron-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  !qg-123e8e11-94 !qg-123e8e11-94  0.0.0.0/0            0.0.0.0/0           ! ctstate DNAT
 
Chain neutron-l3-agent-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            169.254.169.254     tcp dpt:80 redir ports 9697
    0     0 DNAT       all  --  *      *       0.0.0.0/0            10.224.159.171      to:192.168.76.2
 
Chain neutron-l3-agent-float-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  *      *       192.168.76.2         0.0.0.0/0           to:10.224.159.171
 
Chain neutron-l3-agent-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination
    6   428 neutron-l3-agent-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 SNAT       all  --  *      *       192.168.76.0/24      0.0.0.0/0           to:10.224.159.170
 
Chain neutron-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source               destination
    6   428 neutron-l3-agent-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0

接口qg-123e8e11-94会被连接到br-int上：

[root@ci91szcmp001 ~]# ovs-vsctl show
b1c45d14-3a2b-4d80-9d14-60c50770d7e1
    Bridge "br-eth1"
        Port "br-eth1"
            Interface "br-eth1"
                type: internal
        Port "phy-br-eth1"
            Interface "phy-br-eth1"
        Port "eth1"
            Interface "eth1"
    Bridge br-int
        Port "qr-f1977c17-37"
            tag: 10
            Interface "qr-f1977c17-37"
                type: internal
        Port "int-br-eth1"
            Interface "int-br-eth1"
        Port "qg-123e8e11-94"
            tag: 5
            Interface "qg-123e8e11-94"
                type: internal
        Port br-int
            Interface br-int
                type: internal
...

接口qg-123e8e11-94来自于一个external网络，此处这个external网络也是一个provider 网络。可参照：

http://blog.csdn.net/matt_mao/article/details/19088127