Redis3.0集羣crc16算法php實現方法(php取得redis3.0集羣中redis數據所在的redis分區插槽,並根據分區插槽取得分區所在redis服務器地址)

數據分區

       Redis集羣將數據分區後存儲在多個節點上,即不同的分區存儲在不同的節點上,每個節點可以存儲多個分區。每個分區在Redis中也被稱爲“hash slot”,Redis集羣中總共規劃了16384個分區。
例如:當集羣中有3個節點時,節點A將包含0-5460分區,節點B將包含5461-10922分區,節點C將包含10923-16383分區。
每個key將會存儲到一個唯一的分區中,每個分區其實就是一組key的集合,兩者對應關係爲:key的CRC16校驗碼%16384=hash slot(分區標記).可見Redis並沒有像Memecache一樣使用一致性哈希。社區說採用此規則的key分佈是相當的均勻,在我們的測試中也印證了這一點。
在Redis集羣中添加或者移除一個節點時相當容易的事情。例如:添加新節點D時,需要做的只是從A、B、C節點中移動一些分區給D。類似的,移除A時,只需將原屬A的分區移動給B和C,等A變空時移除即可。
節點間的分區移動不需要停止服務,所以添加節點、移除節點或者改變節點的分區數量不需要停止集羣服務。
客戶端訪問集羣時,理論上可以訪問集羣中的任一節點。此時,被訪問的數據可能不存在於被訪問的節點中,但是被訪問節點能自動獲知目標節點,並重定向客戶端的訪問,即將目標節點地址返回給客戶端,客戶端再次發起訪問請求。當然,好的客戶端工具應該實現數據分區和節點對應關係的緩存,並在對應關係發生改變時能自動更新。目前,包括Jedis在內的幾個客戶端工具已經實現了此功能。

CRC概念

        CRC基本原理不懂的,請移步維基百科:循環冗餘檢驗碼

        通常根據CRC校驗碼的位數(也等於生成多項式【G(x)】最高的冪次)的不同來區分不同的CRC算法,如CRC-1、CRC-8、CRC-16等。冪次相同的情況下,不同的標準也有不同的CRC算法。比如G(x)最高次冪爲16的時候有:CRC-16-CCITT、CRC-16-IBM等。Redis使用的是CRC-16-CCITT標準,即G(x)爲:x16 + x12 + x5 + 1 。

        G(x)的通常表徵方式是將多項式轉換成二進制: 1 0001 0000 0010 0001。用十六進制表示爲:0x11021。該數存儲空間是17位(2個字節+1個位,C語言實際存儲是3個字節),實際上,在模二除的時候,被除數的最高位 1 和除數最高位 1 總是對齊的,其異或結果,總爲0,故可省略,則G(x) = 0x1021(2個字節),節省了一個字節的空間。 摘取天上星 原創,轉載請標明作者出處


源碼

redis的src目錄下的 crc16.c文件:
static const uint16_t crc16tab[256]= {
    0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7,
    0x8108,0x9129,0xa14a,0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef,
    0x1231,0x0210,0x3273,0x2252,0x52b5,0x4294,0x72f7,0x62d6,
    0x9339,0x8318,0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de,
    0x2462,0x3443,0x0420,0x1401,0x64e6,0x74c7,0x44a4,0x5485,
    0xa56a,0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d,
    0x3653,0x2672,0x1611,0x0630,0x76d7,0x66f6,0x5695,0x46b4,
    0xb75b,0xa77a,0x9719,0x8738,0xf7df,0xe7fe,0xd79d,0xc7bc,
    0x48c4,0x58e5,0x6886,0x78a7,0x0840,0x1861,0x2802,0x3823,
    0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b,
    0x5af5,0x4ad4,0x7ab7,0x6a96,0x1a71,0x0a50,0x3a33,0x2a12,
    0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a,
    0x6ca6,0x7c87,0x4ce4,0x5cc5,0x2c22,0x3c03,0x0c60,0x1c41,
    0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49,
    0x7e97,0x6eb6,0x5ed5,0x4ef4,0x3e13,0x2e32,0x1e51,0x0e70,
    0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78,
    0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f,
    0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025,0x7046,0x6067,
    0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e,
    0x02b1,0x1290,0x22f3,0x32d2,0x4235,0x5214,0x6277,0x7256,
    0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d,
    0x34e2,0x24c3,0x14a0,0x0481,0x7466,0x6447,0x5424,0x4405,
    0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c,
    0x26d3,0x36f2,0x0691,0x16b0,0x6657,0x7676,0x4615,0x5634,
    0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab,
    0x5844,0x4865,0x7806,0x6827,0x18c0,0x08e1,0x3882,0x28a3,
    0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a,
    0x4a75,0x5a54,0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92,
    0xfd2e,0xed0f,0xdd6c,0xcd4d,0xbdaa,0xad8b,0x9de8,0x8dc9,
    0x7c26,0x6c07,0x5c64,0x4c45,0x3ca2,0x2c83,0x1ce0,0x0cc1,
    0xef1f,0xff3e,0xcf5d,0xdf7c,0xaf9b,0xbfba,0x8fd9,0x9ff8,
    0x6e17,0x7e36,0x4e55,0x5e74,0x2e93,0x3eb2,0x0ed1,0x1ef0
};

uint16_t crc16(const char *buf, int len) {
    int counter;
    uint16_t crc = 0;
    for (counter = 0; counter < len; counter++)
            crc = (crc<<8) ^ crc16tab[((crc>>8) ^ *buf++)&0x00FF];
    return crc;
}

前文提到了CRC校驗碼不同的機構有不同的標準,這裏Redis遵循的標準是CRC-16-CCITT標準,這也是被XMODEM協議使用的CRC標準,所以也常用XMODEM CRC代指。

該段代碼的算法原理並不是作者首創的,這是比較經典的“基於字節查表法的CRC校驗碼生成算法”(本文爲"摘取天上星"週末實操所得,完全可用!)

php實現redis crc16驗證標準的方法.

function redisCRC16 (&$ptr){
    $crc_table=array(
	    0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7,
	    0x8108,0x9129,0xa14a,0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef,
	    0x1231,0x0210,0x3273,0x2252,0x52b5,0x4294,0x72f7,0x62d6,
	    0x9339,0x8318,0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de,
	    0x2462,0x3443,0x0420,0x1401,0x64e6,0x74c7,0x44a4,0x5485,
	    0xa56a,0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d,
	    0x3653,0x2672,0x1611,0x0630,0x76d7,0x66f6,0x5695,0x46b4,
	    0xb75b,0xa77a,0x9719,0x8738,0xf7df,0xe7fe,0xd79d,0xc7bc,
	    0x48c4,0x58e5,0x6886,0x78a7,0x0840,0x1861,0x2802,0x3823,
	    0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b,
	    0x5af5,0x4ad4,0x7ab7,0x6a96,0x1a71,0x0a50,0x3a33,0x2a12,
	    0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a,
	    0x6ca6,0x7c87,0x4ce4,0x5cc5,0x2c22,0x3c03,0x0c60,0x1c41,
	    0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49,
	    0x7e97,0x6eb6,0x5ed5,0x4ef4,0x3e13,0x2e32,0x1e51,0x0e70,
	    0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78,
	    0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f,
	    0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025,0x7046,0x6067,
	    0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e,
	    0x02b1,0x1290,0x22f3,0x32d2,0x4235,0x5214,0x6277,0x7256,
	    0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d,
	    0x34e2,0x24c3,0x14a0,0x0481,0x7466,0x6447,0x5424,0x4405,
	    0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c,
	    0x26d3,0x36f2,0x0691,0x16b0,0x6657,0x7676,0x4615,0x5634,
	    0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab,
	    0x5844,0x4865,0x7806,0x6827,0x18c0,0x08e1,0x3882,0x28a3,
	    0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a,
	    0x4a75,0x5a54,0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92,
	    0xfd2e,0xed0f,0xdd6c,0xcd4d,0xbdaa,0xad8b,0x9de8,0x8dc9,
	    0x7c26,0x6c07,0x5c64,0x4c45,0x3ca2,0x2c83,0x1ce0,0x0cc1,
	    0xef1f,0xff3e,0xcf5d,0xdf7c,0xaf9b,0xbfba,0x8fd9,0x9ff8,
	    0x6e17,0x7e36,0x4e55,0x5e74,0x2e93,0x3eb2,0x0ed1,0x1ef0
    );
    $crc = 0x0000;
    for ($i = 0; $i < strlen($ptr); $i++)
        $crc =  $crc_table[(($crc>>8) ^ ord($ptr[$i]))] ^ (($crc<<8) & 0x00FFFF);
    return $crc;
}
$test = chr(0xC6).chr(0xCE).chr(0xA2).chr(0x03); // CRC16-CCITT = 0xE2B4
$key1='key1'; 
$key2='key2';
$key3='key3';
echo $key1_db=redisCRC16($key1)%16384; //得出在redis(集羣)中鍵值爲'key1'的數據存儲插槽爲9189
echo "<br/>";
echo $key2_db=redisCRC16($key2)%16384; //得出在redis(集羣)中鍵值爲'key2'的數據存儲插槽爲4998
echo "<br/>";
echo $key3_db=redisCRC16($key3)%16384; //得出在redis(集羣)中鍵值爲'key3'的數據存儲插槽爲935

根據crc16算法取得的redis數據分區插槽取得 分區所在服務器地址:

首先我們先看一下本地集羣環境中的redis分區插槽區間範圍(cd 切換到redis安裝包目錄下的src目錄[注意:redis解壓包包非安裝後的程序所在目錄地址]):

[zatsx@localhost src]$  ./redis-trib.rb check 127.0.0.1:6384
Connecting to node 127.0.0.1:6384: OK
Connecting to node 127.0.0.1:6381: OK
Connecting to node 127.0.0.1:6383: OK
Connecting to node 127.0.0.1:6379: OK
Connecting to node 127.0.0.1:6380: OK
Connecting to node 127.0.0.1:6382: OK
>>> Performing Cluster Check (using node 127.0.0.1:6384)
S: 91329dacb2ac77d9295ed46ecaaec6f2c415f7f6 127.0.0.1:6384
   slots: (0 slots) slave
   replicates 0be8c0b96e23a5843ece9d86cb6d287c92529a8c
M: 0be8c0b96e23a5843ece9d86cb6d287c92529a8c 127.0.0.1:6381
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 53926c1f8b757c6db2d53e12ee94b8c1a761e663 127.0.0.1:6383
   slots: (0 slots) slave
   replicates beff88cb6dcf6897a6c6de36350032984a4bdf33
M: 2fb7a8edab2038d7fabe305dd0099de8bdf1f1e6 127.0.0.1:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: beff88cb6dcf6897a6c6de36350032984a4bdf33 127.0.0.1:6380
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 4738074195072ae29c3f3160382e97c3b56a6392 127.0.0.1:6382
   slots: (0 slots) slave
   replicates 2fb7a8edab2038d7fabe305dd0099de8bdf1f1e6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
以上內容可以看到M位主,S爲從(官方要求的redis集羣環境必須爲6臺以上偶數形式的服務器數量,否則無法創建集羣環境)
從上面輸出可以看到分區插槽所在服務器位置:
 
slots:10923-16383 (5461 slots) 127.0.0.1:6381 對應從機 地址爲127.0.0.1:6384
slots:0-5460 (5461 slots) 127.0.0.1:6379 master 對應從機 地址爲127.0.0.1:6381
slots:5461-10922 (5462 slots) 127.0.0.1:6380 master對應從機 地址爲:127.0.0.1:6383


一共三臺主分區,可以通過 取得的插槽位置 定位到 不同的redis分區上去取對應的數據,一但主分區數量有變動,就需要根據實際分區數量重新定位插槽分區範圍
存取redis地址根據ceil(redisCRC16(redis $key值)%16384 / intval(16384/集羣master數)) 得到 對應 的 redis集羣服務器順序 地址,並根據對應區間地址指向到對應的 redis區間master地址即可,注意:如果對應的master區間掛掉了,也不必擔心,只需要將地址變更爲master主區間對應的 slave從區間服務器地址即可一樣取得redis數據,而這正是redis3.0後續版本的精妙之處,不需要第三方插件即可在服務端制動實現集羣主備模式,不會影響用戶正常讀取,當然如果是master和對應的slave一起掛掉了那就沒辦法了,通常這種機率是很小很小的 小到可以忽略即可...
關於redis添加刪除集羣節點的方法請參閱相關資料,摘取天上星 原創,轉載請標明作者出處,本文暫不詳敘!

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章