Working With NUMA/CPU Pinning
Sep8th, 201611:06 am| Comments
The term CPU pinning / process affinity / NUMA generally boils down to the same idea that In a multi socket system, application best performance can be achieved by allowing application threads to get execute on the CPU core which is as close as to its memory bank. In most of the cases Linux process scheduler is intelligent enough to do this , however if you do this manually by yourself , it’s most likely that you will enjoy luxury of increased application performance. Here are some of my notes describing steps required for process affinity setup
Verify how application (radosgw in my case) threads being executed currently. The 5th column psr
which represents processor core.
1 |
|
NUMA requires the following
- Multi socket system
- NUMA feature should be enabled from BIOS or UEFI ( most modern motherboards have this feature enabled by default )
First verify if NUMA configuration is available at OS level by looking for numa
in /var/log/dmesg
or /var/log/messages
1 2 3 4 5 6 7 |
|
To make use of NUMA node pinning, identify your NUMA zones , using numactl
CLI
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
These 2 commands should give you enough information about your server’s CPU and NUMA configuration.
In my environment, I have a 2 socket server and each socket has 10 cores , so 2 x 10 = 20 cores. On top of it, system has Hyper Threading support which makes 20 cores to 40 cores. These 40 cores are divided into two numa zones - node0 with CPUs 0-9 and 20-29 - node1 with CPUs 10-19 and 30-39 Verify on what cores your service is entitled to run
1 2 3 |
|
Based on the above output, my application (radosgw) can execute on any core between 0-39. I want to do CPU Affinity with the following configuration - radosgw process from CPU’s 0-39 to CPU’s 10-19,30-39 ( allocating whole socket-2 ) Now its the time to tune your application and instruct it to use a specific CPU or NUMA node. For this we need to modify service configuration of your application and add CPU pinning parameters there.
Edit your applications ( in my case radosgw ) systemctl service configuration files /usr/lib/systemd/system/ceph-radosgw\@.service
and add the following section to [Service] section of unit file
1 |
|
Reload changes
1 2 3 4 5 6 7 8 9 10 11 12 |
|
Verify your daemon is getting pinned to the correct CPU cores
1 2 |
|
Another way of doing this or may more easier way is to use numactl command which can bind application to entire NUMA node which includes both CPU and Memory bindings
1 |
|
Image source: stackoverflow.com , Thanks
Hope this is useful for you :)
Posted by Karan SinghSep8th, 201611:06 ambenchmarking, ceph