Cisco VPC peer-gateway 對直連BFD鄰居建立的影響

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者簡介：馮亞偉，去哪兒網 NETOPS，2014年7月加入去哪兒網，擁有豐富的網絡運維經驗，現負責公司IDC和骨幹傳輸網絡的運維工作。","attrs":{}}]}],"attrs":{}},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、場景介紹","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示：兩臺思科 nexus交 換機通過 VPC 爲服務器提供雙上聯接入；服務器通過bond0（主備模式）雙上聯到兩臺交換機上面。連接 SW1 的網卡爲主用網卡，連接 SW2 的網卡爲備用網卡；服務器作爲 k8s 節點需要與兩臺接入交換機建立 iBGP ，爲了縮短 BGP 的收斂時間用單跳 BFD 進行鏈路故障檢測。兩臺接入交換機使用 interface vlan IP 與 Server 建立 BGP 和 BFD 鄰居。Server 與 SW2 的流量需要經過 VPC peer-link。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/74/74a711c37c353b1025c725ca33598307.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、問題介紹：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Server與SW2的BGP正常建立，BFD鄰居不能正常建立。","attrs":{}}]},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、問題處理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設備IP及MAC地址：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b7131ab474d48ce4292a2df85d29ecda.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示：通過抓包，發現 SW2 收到 server 的 BFD 數據包後，給 server 回覆了一個 ICMP port unreachable 消息，說明 SW2 沒有處理 server 的 BFD 數據包。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b7108a0a42c63794cb314b1402327222.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖：再次仔細觀察發現， SW2 收到的來自 server 的 BFD 數據包的 TTL 值是254（正常應該是255），數據包的源 MAC 地址是 SW1 的 MAC 地址。說明數據包在經過 SW1 時， SW1 對它進行了三層轉發，因此 TTL 值纔會被減1，源 MAC 地址被替換。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fb/fb535e1aaa5d100d24a442f2a9638f4c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過查閱 Cisco NX-OS BFD 的文檔發現出於安全考慮，思科交換機在處理直連 BFD 的數據包之前會先檢查數據包的 TTL 值，如果不是255，則不會對數據包進行處理。SW2看到來自 server 的 BFD 數據包的 TTL 值爲254，所以沒有處理 BFD 數據包，然後給 server 回覆了一個ICMP端口不可達消息。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c9/c9cb8b96287ce6529ae3f3a3e98f0b67.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼，server，SW1，SW2 都屬於同一網段，爲什麼 SW1 要對數據包進行三層轉發呢？要弄清楚這個問題，我們首先需要了解一下 VPC peer-gateway 的功能：VPC peer-gateway 使交換機在收到數據包的目的MAC地址是它的 vpc peer 的 MAC 地址時，充當數據包的網關對其進行三層轉發，以減少了跨 VPC peer-link 的二層流量。這是一種保護 VPC Peer-link 的機制。啓用 vpc peer-gateway 後，交換機複製本地 interface vlan MAC 地址給它的 vpc peer，複製過去的 MAC 地址在它的 vpc peer 的MAC地址表中使用 G flag（Gateway flag）進行標記。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在實驗環境中對vpc peer-gateway 進行了觀察：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試拓撲同場景介紹中的拓撲。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試設備：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0b/0be352bea18438b9e1f841053ed19566.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試1：SW1 啓用 vpc peer-gateway, SW2 不啓用 vpc peer-gateway，查看兩臺交換機的 MAC 地址表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示：在 SW1 上啓用 vpc peer-gateway 後，其 interface vlan 400 的 MAC地址 00ea.bd5f.dfa7 在 SW2 的 MAC 地址表中有 G 標記。發送到 SW1的interface vlan 400 的 MAC 地址的數據包在經過 SW2 時，SW2 會對數據包進行三層轉發，數據包的源 MAC 地址會被替換爲 SW2 的 MAC 地址 6c8b.d3ca.ff67，且 TTL 值會被減1。在SW2上未啓用vpc peer-gateway，其interface vlan 400 的MAC地址 6c8b.d3ca.ff67 在 SW1 的 MAC 地址表中沒有 G 標記。發送到 SW2 的 interface vlan 400 的 MAC 地址的數據包在經過 SW1 時，會通過二層轉發經過 vpc peer-link ，源 MAC 地址不會改變，TTL 值也不會減1。而在 SW1 和 SW2上 HSRP VIP 的虛擬 MAC 地址 0000.0c07.ac01 都有 G 標記。所以，發送給 HSRP VIP 的數據包在 SW1 和 SW2 上都會進行本地三層轉發。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b834ce6a32b2681d4c34a7e76fe19da.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b834ce6a32b2681d4c34a7e76fe19da.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試2：SW1 關閉vpc peer-gateway , SW2 啓用 vpc peer-gateway ，查看兩臺交換機的 MAC 地址表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示：SW1 關閉 vpc peer-gateway 以後，其 interface vlan 400 的 MAC 地址在 SW2 的MAC地址表中的G標記消失；SW2 啓用 vpc peer-gateway 以後，其 interface vlan 400 的 MAC 地址在 SW1 的 MAC 地址表中出現 G 標記；而在 SW1 和 SW2 上 interface vlan 400 的 HSRP VIP 的虛擬MAC地址依然都有 G 進行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5a/5ae041c90d8411622e05c3be056d5f72.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9db96c2f7998d69aec5125934d8f7dd9.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試3：SW1 和 SW2 都關閉 vpc peer-gateway ，查看兩臺交換機的MAC地址表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示：在 SW1 和 SW2 上關閉了vpc peer-gateway 以後，其 interface vlan 400 的 MAC 地址在它的 vpc peer 的 MAC 地址表中的 G 標記消失；而 interface vlan 400 的 HSRP VIP 的虛擬 MAC 地址在兩臺 vpc peer 的 MAC 地址表中仍然有 G 標記，所以在關閉了 vpc peer-gateway 以後，以 HSRP VIP 爲網關的要去往 vpc 域外或者其它 vlan 的數據包，在兩臺 vpc 對等體上仍然會進行本地三層轉發，而不會出現HSRP standby 設備將去往 HSRP VIP MAC 地址的數據包通過 vpc peer-link 二層轉發到 HSRP active 設備的現象。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d5/d56600b713974116f75e6439caab2c22.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8da43f259ddd77b85f0c6d2cf06dcc4.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試結論：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"vpc peer-gateway 的啓用會讓本地 interface vlan 的 MAC 地址在它的 VPC peer 的MAC 地址表中有 G 標記。交換機對發送到有 G 標記的 MAC 地址的數據包進行三層轉發。不管 vpc peer-gateway 是否啓用，HSRP VIP 的虛擬 MAC 地址在兩臺 vpc 對等體的MAC地址表中都有 G 標記，兩臺 vpc 對等體在不啓用 vpc peer-gateway 的情況下，也會本地三層轉發目的 MAC 地址是 HSRP VIP 虛擬 MAC 地址的數據包。","attrs":{}}]},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、處理結果：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在我們知道是 vpc peer-gateway 功能使 SW1 對 server 發送給 SW2 的 BFD 數據包進行了三層轉發。在關閉了 vpc peer-gateway 功能以後，SW1 二層轉發 server 發送給 SW2 的 BFD 數據包。數據包的 TTL 值不再被減1，server 與 SW2 的 BFD 鄰居正常建立。","attrs":{}}]},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、注意事項與參考文獻","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在測試時發現 vpc peer-link 不使用 LACP 時，也會影響跨 vpc peer-link 的 BFD 鄰居正常建立，建議作爲 vpc peer-link 的 port-channel 使用 LACP 協議。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BFD RFC：","attrs":{}},{"type":"link","attrs":{"href":"https://datatracker.ietf.org/doc/html/rfc5880","title":null,"type":null},"content":[{"type":"text","text":"datatracker.ietf.org/doc/html/rf…","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cisco NX-OS bfd：","attrs":{}},{"type":"link","attrs":{"href":"https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3600/sw/92x/unicast/configuration/guide/b-cisco-nexus-3600-nx-os-unicast-routing-configuration-guide-92x/b-cisco-nexus-3600-nx-os-unicast-routing-configuration-guide-92x%5C_chapter%5C_010000.pdf","title":null,"type":null},"content":[{"type":"text","text":"www.cisco.com/c/en/us/td/…","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cisco VPC Peer-gateway：","attrs":{}},{"type":"link","attrs":{"href":"https://community.cisco.com/t5/switching/vpc-peer-gateway/td-p/2726210","title":null,"type":null},"content":[{"type":"text","text":"community.cisco.com/t5/switchin…","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BFD over LACP portchannel：","attrs":{}},{"type":"link","attrs":{"href":"https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/200871-BFD-for-BGP-in-AdminDown-State-on-Nexus7.html","title":null,"type":null},"content":[{"type":"text","text":"www.cisco.com/c/en/us/sup…","attrs":{}}]}]}]}