記錄redis "Connection timed out"處理
最近程序報錯:Failed to connect to redis: Connection timed out
發現THP的有關信息,決定做測試及修改
1.page 與 Huge Pages
page:
一般而言,內存管理的最小塊級單位叫做page,一個page是4096bytes,1M的內存會有256個page,1GB的話就會有256,000個page。
CPU通過內置的內存管理單元維護着page表記錄。
Huge pages:
Huge Pages就是大小爲2M到1GB的內存page,主要用於管理數千兆的內存,比如1GB的page對於1TB的內存來說是相對比較合適的。
HugePage廣泛啓用開始於Kernal 2.6,一些版本下2.4內核也可以是用。在操作系統Linux環境中,內存是以頁Page的方式進行分配,默認大小爲4K。如果需要比較大的內存空間,則需要進行頻繁的頁分配和管理尋址動作。
HugePage是傳統4K Page的替代方案。顧名思義,是用HugePage可以讓我們有更大的內存分頁大小。無論是HugePage還是傳統的正常Page,這個過程都涉及到OS內存尋址過程
當一個進程訪問內存的時候,並不是直接進行內存位置訪問,是需要通過Page Table進行轉移變換。在使用HugePage的情況下,PageTable具有了額外的屬性,就是判斷該頁記錄是HugePage還是Regular Page。
2.THP
THP(Transparent Huge Pages)是一個使管理Huge Pages自動化的抽象層。
Linux本身的頁大小是固定的4KB,在2.6.38內核新增了THP,透明地支持huge page(2MB)的使用,並且默認開啓。
THP優點:
(1) 減少page fault。一次page fault可以加載更大的內存塊。
(2) 更小的頁表。相同的內存大小,需要更少的頁。
(3) 由於頁表更小,虛擬地址到物理地址的翻譯也更快。
THP缺點:
(1)降低分配內存效率。需要大塊、連續內存塊,內核線程會比較激進的進行compaction,解決內存碎片,加劇鎖爭用。
(2)降低IO吞吐。由於swapable huge page,在swap時需要切分成原有的4K的頁。Oracle的測試數據顯示會降低30%的IO吞吐。
3.測試性能
查看系統版本:
uname -r
Linux ckl-master1 2.6.32-504.el6.x86_64
查看當前的Huge pages設置:
grep Huge /proc/meminfo
AnonHugePages: 5971968 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
查看是否開啓了Huge pages:
cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
測試THP開啓redis性能:
/usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000
====== SET ======
10000000 requests completed in 41.43 seconds
50 parallel clients
3 bytes payload
keep alive: 1
98.62% <= 1 milliseconds
98.62% <= 2 milliseconds
98.63% <= 4 milliseconds
98.63% <= 5 milliseconds
98.63% <= 7 milliseconds
98.63% <= 12 milliseconds
98.64% <= 13 milliseconds
98.64% <= 14 milliseconds
98.65% <= 15 milliseconds
98.66% <= 16 milliseconds
98.68% <= 17 milliseconds
98.69% <= 18 milliseconds
98.74% <= 19 milliseconds
98.83% <= 20 milliseconds
98.96% <= 21 milliseconds
99.13% <= 22 milliseconds
99.32% <= 23 milliseconds
99.49% <= 24 milliseconds
99.60% <= 25 milliseconds
99.67% <= 26 milliseconds
99.74% <= 27 milliseconds
99.79% <= 28 milliseconds
99.81% <= 29 milliseconds
99.84% <= 30 milliseconds
99.85% <= 31 milliseconds
99.86% <= 32 milliseconds
99.86% <= 33 milliseconds
99.87% <= 34 milliseconds
99.87% <= 35 milliseconds
99.88% <= 36 milliseconds
99.89% <= 37 milliseconds
99.89% <= 38 milliseconds
99.89% <= 39 milliseconds
99.89% <= 40 milliseconds
99.90% <= 42 milliseconds
99.90% <= 43 milliseconds
99.91% <= 44 milliseconds
99.91% <= 49 milliseconds
99.91% <= 53 milliseconds
99.91% <= 54 milliseconds
99.92% <= 55 milliseconds
99.92% <= 56 milliseconds
99.93% <= 57 milliseconds
99.93% <= 62 milliseconds
99.93% <= 63 milliseconds
99.93% <= 65 milliseconds
99.93% <= 66 milliseconds
99.93% <= 67 milliseconds
99.94% <= 68 milliseconds
99.94% <= 74 milliseconds
99.94% <= 76 milliseconds
99.94% <= 77 milliseconds
99.94% <= 78 milliseconds
99.95% <= 79 milliseconds
99.95% <= 81 milliseconds
99.95% <= 82 milliseconds
99.95% <= 83 milliseconds
99.95% <= 85 milliseconds
99.96% <= 86 milliseconds
99.96% <= 91 milliseconds
99.96% <= 92 milliseconds
99.97% <= 93 milliseconds
99.97% <= 95 milliseconds
99.97% <= 96 milliseconds
99.98% <= 97 milliseconds
99.99% <= 98 milliseconds
99.99% <= 99 milliseconds
99.99% <= 100 milliseconds
99.99% <= 101 milliseconds
99.99% <= 126 milliseconds
100.00% <= 157 milliseconds
100.00% <= 158 milliseconds
100.00% <= 168 milliseconds
100.00% <= 169 milliseconds
100.00% <= 169 milliseconds
241382.62 requests per second
關閉THP:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
測試關閉THPredis性能:
/usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000
====== SET ======
10000000 requests completed in 26.72 seconds
50 parallel clients
3 bytes payload
keep alive: 1
99.70% <= 1 milliseconds
99.71% <= 2 milliseconds
100.00% <= 3 milliseconds
100.00% <= 3 milliseconds
374321.53 requests per second
關閉THP後,redis的響應時間明顯縮短,建議關閉THP,需要重啓redis進程