計算機的時鐘（五）：混合邏輯時鐘

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本系列文章主要介紹計算機系統中時鐘的處理。主要內容包含NTP，Lamport邏輯時鐘，向量時鐘，TrueTime等。本文是第五篇，介紹","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"混合邏輯時鐘(Hybrid Logical Clocks)","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（封面圖片來自 ","attrs":{}},{"type":"link","attrs":{"href":"https://unsplash.com/photos/bZbJ1GwvM6U","title":""},"content":[{"type":"text","text":"mana5280","attrs":{}}],"marks":[{"type":"strong"}]},{"type":"text","text":"）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/82f4ecb7a8776e5538180e05e","title":""},"content":[{"type":"text","text":"計算機的時鐘（一）：NTP協議","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/0e2fe2ec64bb4c34605c3edcc","title":""},"content":[{"type":"text","text":"計算機的時鐘（二）：Lamport邏輯時鐘","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/70276f26902a31452013d4aaf","title":""},"content":[{"type":"text","text":"計算機的時鐘（三）：向量時鐘","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://yang.observer/2020/11/02/true-time/","title":null},"content":[{"type":"text","text":"計算機的時鐘（四）：TrueTime","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在本系列前面的文章中我們介紹了計算機處理時鐘的兩種方式，一種是物理時鐘，包括NTP協議和TrueTime都屬於物理時鐘，另一種是邏輯時鐘，包括Lamport邏輯時鐘和向量時鐘。這兩種時鐘有各自的優缺點。物理時鐘的優點在於直觀，就是真實世界的時間，使用方便，缺點在於無法做到絕對精確，成本相對高一些。邏輯時鐘的優點在於可以做到精確的因果關係，缺點在於節點之間需要通信，而且使用上不如物理時鐘直觀。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文介紹的混合邏輯時鐘(HLC)將物理時鐘和邏輯時鐘結合起來，我們來看看它是怎麼一回事。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"HLC介紹","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HLC由Sandeep Kulkarni, Murat Demirbas, Deepak Madeppa, Bharadwaj Avva, and Marcelo Leone在2014年的論文","attrs":{}},{"type":"link","attrs":{"href":"https://cse.buffalo.edu/~demirbas/publications/hlc.pdf","title":null},"content":[{"type":"text","text":"《Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases》","attrs":{}}]},{"type":"text","text":"中提出。目的是爲了填補理論（邏輯時鐘）和實際（物理時鐘）之間的差距，支持因果關係，同時又有物理時鐘的直觀特點。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"給分佈式系統中每個事件分配一個HLC，比如e事件的HLC記作 l.e，HLC保證能夠滿足以下四個性質：","attrs":{}}]},{"type":"blockquote","content":[{"type":"numberedlist","attrs":{"start":"","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"如果 e 事件發生在 f 事件之前（e happened before f），那麼 l.e 一定小於 l.f，也就是滿足因果關係。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"l.e 可以存儲在一個整數中，不會隨着分佈式系統中節點數的增加而增加。（這點和向量時鐘不一樣，向量時鐘會隨着節點數的增加而增加）","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"l.e 的值不會無限增長。（這點和Lamport邏輯時鐘不一樣，Lamport邏輯時鐘會無限增長）","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"l.e 的值和 e 事件發生的物理時鐘值接近，| l.e - pt.e |的值會小於一定的範圍。","attrs":{}}]}],"attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"第一版算法","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回顧之前介紹過的Lamport邏輯時鐘算法：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"記事件 j 的邏輯時鐘爲 l.j","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"初始化：l.j = 0\n發送消息事件或者本地事件：l.j = l.j + 1\n接收m消息事件：l.j = max(l.j, l.m) + 1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果想在算法中引入物理時鐘，最簡單的做法是每次修改邏輯時鐘的時候，比較邏輯時鐘和物理時鐘的大小，取最大的值。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"記事件 j 發生時的物理時鐘值爲 pt.j，算法變成了：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"初始化：l.j = 0\n發送消息事件或者本地事件：l.j = max(l.j + 1, pt.j)\n接收m消息事件：l.j = max(l.j + 1, l.m + 1, pt.j)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從算法中可以很容易地看出滿足上面說的HLC性質1和2。然而算法不滿足性質3和性質4。舉個例子：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1e/1ea18aabd36ef8c849c4843112b4decc.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統中有4個節點，每個矩形中左邊的數字是本節點的物理時鐘 pt，右邊是本節點的邏輯時鐘 l。消息從節點0依次到節點1，節點2，節點3，再到節點1。到了節點1的時候，從圖中可以看到物理時鐘和邏輯時鐘的差距 |l - pt| 是13。如果整個系統中事件發生的頻率比較高，可以想象到，物理時鐘和邏輯時鐘的差距會越來越大。這違反了性質3和性質4。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼會發生這個問題？原因是雖然在Lamport邏輯時鐘的基礎上引入了物理時鐘，但是我們卻不知道這個值究竟是物理時鐘增長導致的還是邏輯時鐘增長導致的。這樣即使物理時鐘的增長追趕上了邏輯時鐘的增長，我們也沒辦法重置邏輯時鐘部分的值。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一版算法涼涼。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"第二版算法","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決這個問題，很自然地想到把物理時鐘和邏輯時鐘分開來表示。我們把HLC分成兩部分 l.j 和 c.j。l.j 表示事件 j 發生時所感知到的最大物理時鐘值，c.j 是事件 j 的邏輯時鐘部分，當幾個事件在同一個物理時鐘值內發生時，c 用於記錄事件之間的因果關係。pt 依然表示本地NTP協議的物理時鐘值。新的算法如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"初始化：l.j = 0; c.j = 0\n發送消息事件或者本地事件：\n if pt.j <= l.j then c.j = c.j + 1\n else c.j = 0; l.j = pt.j\n接收m消息事件：\n if l.j == m.j == pt.j then c.j = max(c.j, c.m) + 1\n else if pt.j <= l.j and l.m <= l.j then c.j = c.j + 1\n else if pt.j <= l.m and l.j <= l.m then c.j = c.m + 1; l.j = l.m\n else c.j = 0; l.j = pt.j","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新算法執行的過程中，本地時鐘的值通過NTP協議更新，HLC的值並不會修改本地時鐘的值。由於分離了物理時鐘和邏輯時鐘，新的事件發生時，如果物理時鐘部分的值沒增長，就只增加邏輯時鐘部分的值。如果本地的物理時鐘趕上了HLC的物理時鐘部分的值 l.j，就可以重置邏輯時鐘部分的值 c.j，並把 l.j 更新爲新的本地物理時鐘。這樣就可以解決第一版算法中HLC無限增長的問題，也滿足了性質3和性質4的要求。具體數學證明過程參考","attrs":{}},{"type":"link","attrs":{"href":"https://cse.buffalo.edu/~demirbas/publications/hlc.pdf","title":null},"content":[{"type":"text","text":"論文","attrs":{}}]},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於任何一個事件 j，pt.j <= l.j，也即HLC的物理時鐘部分的值一定大於等於本地NTP的時鐘值。假設整個分佈式系統中，NTP協議的時鐘誤差值爲 ε。新算法中，對於任何一個事件 j，| l.j - pt.j | <= ε，也就是HLC物理部分的值和本地物理時鐘值的差距不會超過 ε。根據前文","attrs":{}},{"type":"link","attrs":{"href":"http://yang.observer/2020/07/11/time-ntp/","title":null},"content":[{"type":"text","text":"計算機的時鐘（一）：NTP協議","attrs":{}}]},{"type":"text","text":"的介紹，這個誤差值在局域網內大概1毫秒內，廣域網可能達到100毫秒或更大。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用新算法來分析第一版算法中的例子：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1e/1ea18aabd36ef8c849c4843112b4decc.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖中，如果消息只在節點123之間流動，HLC的物理時鐘部分 l.j 會一直保持爲10，c.j 部分會不斷增長，直到節點123中任何一個的NTP時鐘值超過10，這時會將 c.j 重置爲0。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工程實現上，HLC可以設置成64比特的整數，高位48比特是以毫秒爲單位的物理時間。低位16比特是一個單調增長的整數，最大65535。整體結構有點像","attrs":{}},{"type":"link","attrs":{"href":"http://yang.observer/2020/08/30/snowflake/","title":null},"content":[{"type":"text","text":"雪花算法","attrs":{}}]},{"type":"text","text":"生成的唯一ID。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"HLC的異常處理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果某個節點的NTP物理時鐘值出現了異常，比如變成一個極大的值，其他節點收到這個故障節點的消息，會導致其他節點的HLC值出現問題。爲了阻止故障的擴散，可以設置HLC物理部分偏差的上限，如果收到的消息物理部分值的偏差超過上限，忽略這條消息，同時發送告警等待人工處理。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"HLC的性能數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://cse.buffalo.edu/~demirbas/publications/hlc.pdf","title":null},"content":[{"type":"text","text":"論文","attrs":{}}]},{"type":"text","text":"中給出了HLC的性能測試數據。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e9/e9a4a3b98d9639bbb9e125faa9c39ac2.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖中，4個節點組成的系統，NTP的平均誤差值 offset 分別設爲5毫秒和1.5毫秒兩種情況。在這兩種情況下，HLC的邏輯時鐘部分的值 c < 4，保持了一個很低的值。 在offset爲5毫秒的情況下，| l - pt | 的最大差距是21.7毫秒，90%的差距值小於7.8毫秒，平均差距是0.2毫秒。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c4446a327574dacce901ab0b316c328.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果把節點數變成16個，NTP的平均誤差值 offset 分別設爲16毫秒和6毫秒兩種情況。c < 8，依然保持了一個很低的值。從圖中可以看出 offset越小，c 的值整體會更小。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"HLC應用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HLC可以用於分佈式數據庫一致性快照讀的處理中。很多系統中都使用了HLC，比如HBase和CockRoachDB。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1d/1de4a5241b02e0d998ab5085d22b6aa9.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如，我們要獲取 t = 10 這個時間點的數據快照，也即HLC爲(l = 10, c = 0)，拿這個HLC值去每個節點查找，可以得出上圖中黑色的粗線，這條線對應的數據就是系統在 t = 10 的數據快照。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CockRoachDB在分佈式事務中使用了HLC。根據HLC的性質4，| l.e - pt.e |的值會小於一定的範圍 MaxOffset，CockRoachDB默認這個值爲500毫秒，pt.e + MaxOffset一定是系統中最大的物理時間。啓動事務時會獲取本地的HLC值 hlc.e，並且確定一個區間 [ hlc.e, hlc.e + MaxOffset ]，然後發往其他節點執行快照讀，如果節點上某個數據的HLC值爲 hlc.g，分三種情況考慮：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":"","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"如果 hlc.e + MaxOffset < hlc.g，即處於區間的右邊，那麼e事件肯定發生在g事件之前，不能讀取這個數據。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"如果 hlc.e < hlc.g <= hlc.e + MaxOffs，即處於區間之中，由於物理時鐘的不確定性，不能分辨出e事件和g事件的先後關係。這個時候需要重啓事務，獲取一個更大的hlc，相當於等待這個不確定的時間過去，推遲事務的執行。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"如果 hlc.g < hlc.e，那麼g事件肯定發生在e之前，這時可以讀取這個數據。（對於這點我有點疑問，由於不確定時間的存在，物理時間可能快，也可能慢，這個區間應該是[ hlc.e - MaxOffset, hlc.e + MaxOffset ]，爲什麼這裏hlc.g < hlc.e就認爲g在e前面？）","attrs":{}}]}],"attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個系列一直在物理時鐘和邏輯時鐘之間打轉。先介紹了最直觀的NTP協議，由於NTP協議的不可靠性，引入了邏輯時鐘，包括Lamport邏輯時鐘和向量時鐘，但是邏輯時鐘只能確保因果關係，並且需要消息的傳遞，由此又引入了更精確的物理時鐘TrueTime，最後是把物理時鐘和邏輯時鐘結合起來的混合邏輯時鐘。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"參考","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://cse.buffalo.edu/~demirbas/publications/hlc.pdf","title":null},"content":[{"type":"text","text":"Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://www.cockroachchina.cn/?p=976","title":null},"content":[{"type":"text","text":"CockroachDB分佈式事務解密","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}