Cisco VPC peer-gateway 对直连BFD邻居建立的影响

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者简介：冯亚伟，去哪儿网 NETOPS，2014年7月加入去哪儿网，拥有丰富的网络运维经验，现负责公司IDC和骨干传输网络的运维工作。","attrs":{}}]}],"attrs":{}},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、场景介绍","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示：两台思科 nexus交 换机通过 VPC 为服务器提供双上联接入；服务器通过bond0（主备模式）双上联到两台交换机上面。连接 SW1 的网卡为主用网卡，连接 SW2 的网卡为备用网卡；服务器作为 k8s 节点需要与两台接入交换机建立 iBGP ，为了缩短 BGP 的收敛时间用单跳 BFD 进行链路故障检测。两台接入交换机使用 interface vlan IP 与 Server 建立 BGP 和 BFD 邻居。Server 与 SW2 的流量需要经过 VPC peer-link。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/74/74a711c37c353b1025c725ca33598307.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、问题介绍：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Server与SW2的BGP正常建立，BFD邻居不能正常建立。","attrs":{}}]},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、问题处理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"设备IP及MAC地址：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b7131ab474d48ce4292a2df85d29ecda.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示：通过抓包，发现 SW2 收到 server 的 BFD 数据包后，给 server 回复了一个 ICMP port unreachable 消息，说明 SW2 没有处理 server 的 BFD 数据包。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b7108a0a42c63794cb314b1402327222.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图：再次仔细观察发现， SW2 收到的来自 server 的 BFD 数据包的 TTL 值是254（正常应该是255），数据包的源 MAC 地址是 SW1 的 MAC 地址。说明数据包在经过 SW1 时， SW1 对它进行了三层转发，因此 TTL 值才会被减1，源 MAC 地址被替换。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fb/fb535e1aaa5d100d24a442f2a9638f4c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过查阅 Cisco NX-OS BFD 的文档发现出于安全考虑，思科交换机在处理直连 BFD 的数据包之前会先检查数据包的 TTL 值，如果不是255，则不会对数据包进行处理。SW2看到来自 server 的 BFD 数据包的 TTL 值为254，所以没有处理 BFD 数据包，然后给 server 回复了一个ICMP端口不可达消息。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c9/c9cb8b96287ce6529ae3f3a3e98f0b67.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么，server，SW1，SW2 都属于同一网段，为什么 SW1 要对数据包进行三层转发呢？要弄清楚这个问题，我们首先需要了解一下 VPC peer-gateway 的功能：VPC peer-gateway 使交换机在收到数据包的目的MAC地址是它的 vpc peer 的 MAC 地址时，充当数据包的网关对其进行三层转发，以减少了跨 VPC peer-link 的二层流量。这是一种保护 VPC Peer-link 的机制。启用 vpc peer-gateway 后，交换机复制本地 interface vlan MAC 地址给它的 vpc peer，复制过去的 MAC 地址在它的 vpc peer 的MAC地址表中使用 G flag（Gateway flag）进行标记。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们在实验环境中对vpc peer-gateway 进行了观察：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"测试拓扑同场景介绍中的拓扑。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"测试设备：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0b/0be352bea18438b9e1f841053ed19566.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"测试1：SW1 启用 vpc peer-gateway, SW2 不启用 vpc peer-gateway，查看两台交换机的 MAC 地址表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示：在 SW1 上启用 vpc peer-gateway 后，其 interface vlan 400 的 MAC地址 00ea.bd5f.dfa7 在 SW2 的 MAC 地址表中有 G 标记。发送到 SW1的interface vlan 400 的 MAC 地址的数据包在经过 SW2 时，SW2 会对数据包进行三层转发，数据包的源 MAC 地址会被替换为 SW2 的 MAC 地址 6c8b.d3ca.ff67，且 TTL 值会被减1。在SW2上未启用vpc peer-gateway，其interface vlan 400 的MAC地址 6c8b.d3ca.ff67 在 SW1 的 MAC 地址表中没有 G 标记。发送到 SW2 的 interface vlan 400 的 MAC 地址的数据包在经过 SW1 时，会通过二层转发经过 vpc peer-link ，源 MAC 地址不会改变，TTL 值也不会减1。而在 SW1 和 SW2上 HSRP VIP 的虚拟 MAC 地址 0000.0c07.ac01 都有 G 标记。所以，发送给 HSRP VIP 的数据包在 SW1 和 SW2 上都会进行本地三层转发。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b834ce6a32b2681d4c34a7e76fe19da.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b834ce6a32b2681d4c34a7e76fe19da.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"测试2：SW1 关闭vpc peer-gateway , SW2 启用 vpc peer-gateway ，查看两台交换机的 MAC 地址表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示：SW1 关闭 vpc peer-gateway 以后，其 interface vlan 400 的 MAC 地址在 SW2 的MAC地址表中的G标记消失；SW2 启用 vpc peer-gateway 以后，其 interface vlan 400 的 MAC 地址在 SW1 的 MAC 地址表中出现 G 标记；而在 SW1 和 SW2 上 interface vlan 400 的 HSRP VIP 的虚拟MAC地址依然都有 G 进行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5a/5ae041c90d8411622e05c3be056d5f72.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9db96c2f7998d69aec5125934d8f7dd9.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"测试3：SW1 和 SW2 都关闭 vpc peer-gateway ，查看两台交换机的MAC地址表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示：在 SW1 和 SW2 上关闭了vpc peer-gateway 以后，其 interface vlan 400 的 MAC 地址在它的 vpc peer 的 MAC 地址表中的 G 标记消失；而 interface vlan 400 的 HSRP VIP 的虚拟 MAC 地址在两台 vpc peer 的 MAC 地址表中仍然有 G 标记，所以在关闭了 vpc peer-gateway 以后，以 HSRP VIP 为网关的要去往 vpc 域外或者其它 vlan 的数据包，在两台 vpc 对等体上仍然会进行本地三层转发，而不会出现HSRP standby 设备将去往 HSRP VIP MAC 地址的数据包通过 vpc peer-link 二层转发到 HSRP active 设备的现象。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d5/d56600b713974116f75e6439caab2c22.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8da43f259ddd77b85f0c6d2cf06dcc4.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"测试结论：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"vpc peer-gateway 的启用会让本地 interface vlan 的 MAC 地址在它的 VPC peer 的MAC 地址表中有 G 标记。交换机对发送到有 G 标记的 MAC 地址的数据包进行三层转发。不管 vpc peer-gateway 是否启用，HSRP VIP 的虚拟 MAC 地址在两台 vpc 对等体的MAC地址表中都有 G 标记，两台 vpc 对等体在不启用 vpc peer-gateway 的情况下，也会本地三层转发目的 MAC 地址是 HSRP VIP 虚拟 MAC 地址的数据包。","attrs":{}}]},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、处理结果：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现在我们知道是 vpc peer-gateway 功能使 SW1 对 server 发送给 SW2 的 BFD 数据包进行了三层转发。在关闭了 vpc peer-gateway 功能以后，SW1 二层转发 server 发送给 SW2 的 BFD 数据包。数据包的 TTL 值不再被减1，server 与 SW2 的 BFD 邻居正常建立。","attrs":{}}]},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、注意事项与参考文献","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在测试时发现 vpc peer-link 不使用 LACP 时，也会影响跨 vpc peer-link 的 BFD 邻居正常建立，建议作为 vpc peer-link 的 port-channel 使用 LACP 协议。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BFD RFC：","attrs":{}},{"type":"link","attrs":{"href":"https://datatracker.ietf.org/doc/html/rfc5880","title":null,"type":null},"content":[{"type":"text","text":"datatracker.ietf.org/doc/html/rf…","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cisco NX-OS bfd：","attrs":{}},{"type":"link","attrs":{"href":"https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3600/sw/92x/unicast/configuration/guide/b-cisco-nexus-3600-nx-os-unicast-routing-configuration-guide-92x/b-cisco-nexus-3600-nx-os-unicast-routing-configuration-guide-92x%5C_chapter%5C_010000.pdf","title":null,"type":null},"content":[{"type":"text","text":"www.cisco.com/c/en/us/td/…","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cisco VPC Peer-gateway：","attrs":{}},{"type":"link","attrs":{"href":"https://community.cisco.com/t5/switching/vpc-peer-gateway/td-p/2726210","title":null,"type":null},"content":[{"type":"text","text":"community.cisco.com/t5/switchin…","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BFD over LACP portchannel：","attrs":{}},{"type":"link","attrs":{"href":"https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/200871-BFD-for-BGP-in-AdminDown-State-on-Nexus7.html","title":null,"type":null},"content":[{"type":"text","text":"www.cisco.com/c/en/us/sup…","attrs":{}}]}]}]}