记录redis "Connection timed out"处理(关于THP的处理)

记录redis "Connection timed out"处理

最近程序报错:Failed to connect to redis: Connection timed out

发现THP的有关信息,决定做测试及修改

1.page 与 Huge Pages

page:

一般而言,内存管理的最小块级单位叫做page,一个page是4096bytes,1M的内存会有256个page,1GB的话就会有256,000个page。

 CPU通过内置的内存管理单元维护着page表记录。

Huge pages:

Huge Pages就是大小为2M到1GB的内存page,主要用于管理数千兆的内存,比如1GB的page对于1TB的内存来说是相对比较合适的。

HugePage广泛启用开始于Kernal 2.6,一些版本下2.4内核也可以是用。在操作系统Linux环境中,内存是以页Page的方式进行分配,默认大小为4K。如果需要比较大的内存空间,则需要进行频繁的页分配和管理寻址动作。

HugePage是传统4K Page的替代方案。顾名思义,是用HugePage可以让我们有更大的内存分页大小。无论是HugePage还是传统的正常Page,这个过程都涉及到OS内存寻址过程

当一个进程访问内存的时候,并不是直接进行内存位置访问,是需要通过Page Table进行转移变换。在使用HugePage的情况下,PageTable具有了额外的属性,就是判断该页记录是HugePage还是Regular Page。

2.THP

THP(Transparent Huge Pages)是一个使管理Huge Pages自动化的抽象层。

Linux本身的页大小是固定的4KB,在2.6.38内核新增了THP,透明地支持huge page(2MB)的使用,并且默认开启。

THP优点:

(1) 减少page fault。一次page fault可以加载更大的内存块。

(2) 更小的页表。相同的内存大小,需要更少的页。

(3) 由于页表更小,虚拟地址到物理地址的翻译也更快。

THP缺点:

(1)降低分配内存效率。需要大块、连续内存块,内核线程会比较激进的进行compaction,解决内存碎片,加剧锁争用。

(2)降低IO吞吐。由于swapable huge page,在swap时需要切分成原有的4K的页。Oracle的测试数据显示会降低30%的IO吞吐。

3.测试性能

查看系统版本:

uname -r
 Linux ckl-master1 2.6.32-504.el6.x86_64

查看当前的Huge pages设置:

  grep Huge /proc/meminfo 
AnonHugePages:   5971968 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

查看是否开启了Huge pages:

  cat /sys/kernel/mm/transparent_hugepage/enabled
 [always] madvise never

测试THP开启redis性能:

 /usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000
	====== SET ======
	  10000000 requests completed in 41.43 seconds
	  50 parallel clients
	  3 bytes payload
	  keep alive: 1

	98.62% <= 1 milliseconds
	98.62% <= 2 milliseconds
	98.63% <= 4 milliseconds
	98.63% <= 5 milliseconds
	98.63% <= 7 milliseconds
	98.63% <= 12 milliseconds
	98.64% <= 13 milliseconds
	98.64% <= 14 milliseconds
	98.65% <= 15 milliseconds
	98.66% <= 16 milliseconds
	98.68% <= 17 milliseconds
	98.69% <= 18 milliseconds
	98.74% <= 19 milliseconds
	98.83% <= 20 milliseconds
	98.96% <= 21 milliseconds
	99.13% <= 22 milliseconds
	99.32% <= 23 milliseconds
	99.49% <= 24 milliseconds
	99.60% <= 25 milliseconds
	99.67% <= 26 milliseconds
	99.74% <= 27 milliseconds
	99.79% <= 28 milliseconds
	99.81% <= 29 milliseconds
	99.84% <= 30 milliseconds
	99.85% <= 31 milliseconds
	99.86% <= 32 milliseconds
	99.86% <= 33 milliseconds
	99.87% <= 34 milliseconds
	99.87% <= 35 milliseconds
	99.88% <= 36 milliseconds
	99.89% <= 37 milliseconds
	99.89% <= 38 milliseconds
	99.89% <= 39 milliseconds
	99.89% <= 40 milliseconds
	99.90% <= 42 milliseconds
	99.90% <= 43 milliseconds
	99.91% <= 44 milliseconds
	99.91% <= 49 milliseconds
	99.91% <= 53 milliseconds
	99.91% <= 54 milliseconds
	99.92% <= 55 milliseconds
	99.92% <= 56 milliseconds
	99.93% <= 57 milliseconds
	99.93% <= 62 milliseconds
	99.93% <= 63 milliseconds
	99.93% <= 65 milliseconds
	99.93% <= 66 milliseconds
	99.93% <= 67 milliseconds
	99.94% <= 68 milliseconds
	99.94% <= 74 milliseconds
	99.94% <= 76 milliseconds
	99.94% <= 77 milliseconds
	99.94% <= 78 milliseconds
	99.95% <= 79 milliseconds
	99.95% <= 81 milliseconds
	99.95% <= 82 milliseconds
	99.95% <= 83 milliseconds
	99.95% <= 85 milliseconds
	99.96% <= 86 milliseconds
	99.96% <= 91 milliseconds
	99.96% <= 92 milliseconds
	99.97% <= 93 milliseconds
	99.97% <= 95 milliseconds
	99.97% <= 96 milliseconds
	99.98% <= 97 milliseconds
	99.99% <= 98 milliseconds
	99.99% <= 99 milliseconds
	99.99% <= 100 milliseconds
	99.99% <= 101 milliseconds
	99.99% <= 126 milliseconds
	100.00% <= 157 milliseconds
	100.00% <= 158 milliseconds
	100.00% <= 168 milliseconds
	100.00% <= 169 milliseconds
	100.00% <= 169 milliseconds
	241382.62 requests per second

关闭THP:

  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  cat /sys/kernel/mm/transparent_hugepage/enabled   
  always madvise [never]

测试关闭THPredis性能:

/usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000
	====== SET ======
	  10000000 requests completed in 26.72 seconds
	  50 parallel clients
	  3 bytes payload
	  keep alive: 1

	99.70% <= 1 milliseconds
	99.71% <= 2 milliseconds
	100.00% <= 3 milliseconds
	100.00% <= 3 milliseconds
	374321.53 requests per second

关闭THP后,redis的响应时间明显缩短,建议关闭THP,需要重启redis进程

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章