記錄redis "Connection timed out"處理(關於THP的處理)

記錄redis "Connection timed out"處理

最近程序報錯:Failed to connect to redis: Connection timed out

發現THP的有關信息,決定做測試及修改

1.page 與 Huge Pages

page:

一般而言,內存管理的最小塊級單位叫做page,一個page是4096bytes,1M的內存會有256個page,1GB的話就會有256,000個page。

 CPU通過內置的內存管理單元維護着page表記錄。

Huge pages:

Huge Pages就是大小爲2M到1GB的內存page,主要用於管理數千兆的內存,比如1GB的page對於1TB的內存來說是相對比較合適的。

HugePage廣泛啓用開始於Kernal 2.6,一些版本下2.4內核也可以是用。在操作系統Linux環境中,內存是以頁Page的方式進行分配,默認大小爲4K。如果需要比較大的內存空間,則需要進行頻繁的頁分配和管理尋址動作。

HugePage是傳統4K Page的替代方案。顧名思義,是用HugePage可以讓我們有更大的內存分頁大小。無論是HugePage還是傳統的正常Page,這個過程都涉及到OS內存尋址過程

當一個進程訪問內存的時候,並不是直接進行內存位置訪問,是需要通過Page Table進行轉移變換。在使用HugePage的情況下,PageTable具有了額外的屬性,就是判斷該頁記錄是HugePage還是Regular Page。

2.THP

THP(Transparent Huge Pages)是一個使管理Huge Pages自動化的抽象層。

Linux本身的頁大小是固定的4KB,在2.6.38內核新增了THP,透明地支持huge page(2MB)的使用,並且默認開啓。

THP優點:

(1) 減少page fault。一次page fault可以加載更大的內存塊。

(2) 更小的頁表。相同的內存大小,需要更少的頁。

(3) 由於頁表更小,虛擬地址到物理地址的翻譯也更快。

THP缺點:

(1)降低分配內存效率。需要大塊、連續內存塊,內核線程會比較激進的進行compaction,解決內存碎片,加劇鎖爭用。

(2)降低IO吞吐。由於swapable huge page,在swap時需要切分成原有的4K的頁。Oracle的測試數據顯示會降低30%的IO吞吐。

3.測試性能

查看系統版本:

uname -r
 Linux ckl-master1 2.6.32-504.el6.x86_64

查看當前的Huge pages設置:

  grep Huge /proc/meminfo 
AnonHugePages:   5971968 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

查看是否開啓了Huge pages:

  cat /sys/kernel/mm/transparent_hugepage/enabled
 [always] madvise never

測試THP開啓redis性能:

 /usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000
	====== SET ======
	  10000000 requests completed in 41.43 seconds
	  50 parallel clients
	  3 bytes payload
	  keep alive: 1

	98.62% <= 1 milliseconds
	98.62% <= 2 milliseconds
	98.63% <= 4 milliseconds
	98.63% <= 5 milliseconds
	98.63% <= 7 milliseconds
	98.63% <= 12 milliseconds
	98.64% <= 13 milliseconds
	98.64% <= 14 milliseconds
	98.65% <= 15 milliseconds
	98.66% <= 16 milliseconds
	98.68% <= 17 milliseconds
	98.69% <= 18 milliseconds
	98.74% <= 19 milliseconds
	98.83% <= 20 milliseconds
	98.96% <= 21 milliseconds
	99.13% <= 22 milliseconds
	99.32% <= 23 milliseconds
	99.49% <= 24 milliseconds
	99.60% <= 25 milliseconds
	99.67% <= 26 milliseconds
	99.74% <= 27 milliseconds
	99.79% <= 28 milliseconds
	99.81% <= 29 milliseconds
	99.84% <= 30 milliseconds
	99.85% <= 31 milliseconds
	99.86% <= 32 milliseconds
	99.86% <= 33 milliseconds
	99.87% <= 34 milliseconds
	99.87% <= 35 milliseconds
	99.88% <= 36 milliseconds
	99.89% <= 37 milliseconds
	99.89% <= 38 milliseconds
	99.89% <= 39 milliseconds
	99.89% <= 40 milliseconds
	99.90% <= 42 milliseconds
	99.90% <= 43 milliseconds
	99.91% <= 44 milliseconds
	99.91% <= 49 milliseconds
	99.91% <= 53 milliseconds
	99.91% <= 54 milliseconds
	99.92% <= 55 milliseconds
	99.92% <= 56 milliseconds
	99.93% <= 57 milliseconds
	99.93% <= 62 milliseconds
	99.93% <= 63 milliseconds
	99.93% <= 65 milliseconds
	99.93% <= 66 milliseconds
	99.93% <= 67 milliseconds
	99.94% <= 68 milliseconds
	99.94% <= 74 milliseconds
	99.94% <= 76 milliseconds
	99.94% <= 77 milliseconds
	99.94% <= 78 milliseconds
	99.95% <= 79 milliseconds
	99.95% <= 81 milliseconds
	99.95% <= 82 milliseconds
	99.95% <= 83 milliseconds
	99.95% <= 85 milliseconds
	99.96% <= 86 milliseconds
	99.96% <= 91 milliseconds
	99.96% <= 92 milliseconds
	99.97% <= 93 milliseconds
	99.97% <= 95 milliseconds
	99.97% <= 96 milliseconds
	99.98% <= 97 milliseconds
	99.99% <= 98 milliseconds
	99.99% <= 99 milliseconds
	99.99% <= 100 milliseconds
	99.99% <= 101 milliseconds
	99.99% <= 126 milliseconds
	100.00% <= 157 milliseconds
	100.00% <= 158 milliseconds
	100.00% <= 168 milliseconds
	100.00% <= 169 milliseconds
	100.00% <= 169 milliseconds
	241382.62 requests per second

關閉THP:

  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  cat /sys/kernel/mm/transparent_hugepage/enabled   
  always madvise [never]

測試關閉THPredis性能:

/usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000
	====== SET ======
	  10000000 requests completed in 26.72 seconds
	  50 parallel clients
	  3 bytes payload
	  keep alive: 1

	99.70% <= 1 milliseconds
	99.71% <= 2 milliseconds
	100.00% <= 3 milliseconds
	100.00% <= 3 milliseconds
	374321.53 requests per second

關閉THP後,redis的響應時間明顯縮短,建議關閉THP,需要重啓redis進程

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章