Redis做消息隊列全攻略

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Redis消息隊列"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在程序員這個圈子打拼了太多年，見過太多的程序員使用redis，其中一部分喜歡把redis做緩存（cache）使用，其中最典型的當屬存儲用戶session，除此之外，把redis作爲消息隊列使用也不在少數，可見redis在互聯網中應用是多麼的廣泛。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"redis作爲消息隊列使用，redis支持的數據結構是可以支撐這類業務，主要是利用了list這種數據結構的特性。Redis的列表相當於編程語言裏面的 LinkedList，是一個雙向的列表結構，這意味着列表新增和刪除元素是非常快的，時間複雜度爲O(1)，但是查找一個元素的時候需要遍歷列表，時間複雜度爲O(n)。由於列表的元素操作和消息隊列操作類似，所以redis可以適用於消息隊列的場景，當然，在適用於的棧的場景下也可以勝任。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要提醒一下，生產環境中如果對消息的可靠性有十分高的要求（比如訂單支付的消費消息），請使用專業的消息隊列（例如：rmq，amq等），對消息的丟失有一定容忍度的程序完全可以使用redis，例如我們的日誌收集程序"}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/854c906fccca4207b49a7107ceb533d4.jpeg","alt":"image","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"列表這種數據結構的命令爲"}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"移出並獲取列表的第一個元素， 如果列表沒有元素會阻塞列表直到等待超時或發現可彈出元素爲止。\nBLPOP key1 [key2 ] timeout\n\n移出並獲取列表的最後一個元素， 如果列表沒有元素會阻塞列表直到等待超時或發現可彈出元素爲止。\nBRPOP key1 [key2 ] timeout \n\n\n從列表中彈出一個值，將彈出的元素插入到另外一個列表中並返回它； 如果列表沒有元素會阻塞列表直到等待超時或發現可彈出元素爲止。\nBRPOPLPUSH source destination timeout \n\n通過索引獲取列表中的元素\nLINDEX key index \n\n在列表的元素前或者後插入元素\nLINSERT key BEFORE|AFTER pivot value \n\n獲取列表長度\nLLEN key \n\n移出並獲取列表的第一個元素\nLPOP key \n\n將一個或多個值插入到列表頭部\nLPUSH key value1 [value2] \n\n將一個值插入到已存在的列表頭部\nLPUSHX key value \n\n獲取列表指定範圍內的元素\nLRANGE key start stop \n\n移除列表元素\nLREM key count value \n\n通過索引設置列表元素的值\nLSET key index value \n\n對一個列表進行修剪(trim)，就是說，讓列表只保留指定區間內的元素，不在指定區間之內的元素都將被刪除。\nLTRIM key start stop \n\n移除列表的最後一個元素，返回值爲移除的元素。\nRPOP key \n\n移除列表的最後一個元素，並將該元素添加到另一個列表並返回\nRPOPLPUSH source destination \n\n在列表中添加一個或多個值\nRPUSH key value1 [value2] \n\n爲已存在的列表添加值\nRPUSHX key value \n\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"link","attrs":{"href":"#缺陷","title":null}},{"type":"text","text":"缺陷"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列的本質還是消費者和生產者的問題，只要是這樣的場景，就會涉及到兩端不平衡的情況，具體可表現爲："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"生產者生產速度大於消費者消費速度，面臨消息不斷堆積的問題，隨着消息數據的堆積，隊列是開啓限流措施，還是丟棄某些消息，更或者是把消息數據進行持久化。對於基於redis實現的消息隊列，一般爲可忍受部分消息丟失的業務，所以很多人選擇丟棄消息的方案。另一種方案是基於redis單線程機制，可以增加消費者數量，這也是僅僅針對消息只被消費一次的場景。當然也可以選擇持久化方案，但是會對redis的性能產生影響。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"消費者消費速度大於生產者生產速度，有的同學會說，這樣挺好啊，是，在某種意義上是比反過來的那個場景要好一些，畢竟可以避免產生消息的堆積問題。但是消費者沒有消息消費，會導致消費者進程一直在那裏浪費cpu資源，而且還會把redis的QPS拉高。類似於這種死循環的場景，一般而且最常用的解決方案是讓線程sleep 一小段時間，既降低了消費端cpu也降低了redis的QPS。 但是sleep會有一個問題，會導致處理消息的延遲，例如sleep了一秒，那消息的延遲處理就有可能會延遲一秒，雖然在大部分場景下這都不是什麼問題，但是作爲程序員怎麼能不追求極致和完美呢？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於消息延遲的問題，最暴力簡單的方式就是增加消費客戶端，這樣可用多消費端交錯的方式來縮小延遲的間隔，當然redis的設計者也考慮了這個問題，所有有了Blpop 命令"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis Blpop 命令移出並獲取列表的第一個元素， 如果列表沒有元素會阻塞列表直到等待超時或發現可彈出元素爲止。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"redis 127.0.0.1:6379> BLPOP LIST1 LIST2 .. LISTN TIMEOUT"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且還可以設置超時自動返回，豈不是完美。但是還要順便一句，redis的連接在空閒一段時間後，服務端可能會主動斷開，Blpop命令會拋出異常，所以還要做好了重試或者其他策略爲好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"如果作爲專業的消息隊列，一個消息被多個不同的業務消費（一個消息被消費多次）是必須要支持的，但是redis是基於自己的list數據結構來實現的僞隊列，所以這種業務場景下就不要考慮redis了，或者自己封裝一個類似分發器的中間件也可以。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"基於redis的消息隊列沒有Ack的保證，換句話說，一個消息是否被正常處理redis是不知道的，這在很大程度上限制了它的適用場景。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"link","attrs":{"href":"#寫在最後","title":null}},{"type":"text","text":"寫在最後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我還是建議不要用redis做專業的MQ使用，畢竟MQ這種場景不是redis的設計初衷，但是太多人把redis做MQ使用，於是redis的作者基於redis的核心代碼實現了一個消息隊列：disque，也許未來會作爲redis的核心組件,地址爲 "},{"type":"link","attrs":{"href":"https://github.com/antirez/disque","title":null},"content":[{"type":"text","text":"https://github.com/antirez/disque"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了disque，Redis Stream也是一個把redis作爲MQ的比較好的解決方案，有興趣的同學可以研究一下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"千萬不要把任何一個業務場景想象的太簡單"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"更多精彩文章"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?action=getalbum&album_id=1342955119549267969&__biz=MzIwNTc3OTAxOA==#wechat_redirect","title":null},"content":[{"type":"text","text":"分佈式大併發系列"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?action=getalbum&album_id=1342959003139227648&__biz=MzIwNTc3OTAxOA==#wechat_redirect","title":null},"content":[{"type":"text","text":"架構設計系列"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?action=getalbum&album_id=1342962375443529728&__biz=MzIwNTc3OTAxOA==#wechat_redirect","title":null},"content":[{"type":"text","text":"趣學算法和數據結構系列"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?action=getalbum&album_id=1342964237798391808&__biz=MzIwNTc3OTAxOA==#wechat_redirect","title":null},"content":[{"type":"text","text":"設計模式系列"}]}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f8af5984765a267892bf1a1272272625.png","alt":"image","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}