postgresql shared_buffers 講解
什麼是shred_buffer,我們爲什麼需要shared_buffers?
1.在數據庫系統中,我們主要關注磁盤io,大多數oltp工作負載都是隨機io,因此從磁盤獲取非常慢。
2.爲了解決這個問題,postgres將數據緩存在RAM中,以此來提高性能,即使ssd的情況下RAM也要快很多。
3.shared_buffers是一個8KB的數組,postgres在從磁盤中查詢數據前,會先查找shared_buffers的頁,如果命中,就直接返回,避免從磁盤查詢。
shared_buffers存儲什麼?
1.表數據
2.索引,索引也存儲在8K塊中。
3.執行計劃,存儲基於會話的執行計劃,會話結束,緩存的計劃也就被丟棄。
什麼時候加載shared_buffers?
1.在訪問數據時,數據會先加載到os緩存,然後再加載到shared_buffers,這個加載過程可能是一些查詢,也可以使用pg_prewarm預熱緩存。
2.當然也可能同時存在os和shared_buffers兩份一樣的緩存(雙緩存)。
3.查找到的時候會先在shared_buffers查找是否有緩存,如果沒有再到os緩存查找,最後再從磁盤獲取。
4.os緩存使用簡單的LRU(移除最近最久未使用的緩存),而數據庫採用的優化的時鐘掃描,即緩存使用頻率高的會被保存,低的被移除。
shared_buffers設置的合理範圍
1.windows服務器有用範圍是64MB到512MB,默認128MB
2.linux服務器建議設置爲25%,亞馬遜服務器設置爲75%(避免雙緩存,數據會存儲在os和shared_buffers兩份)
os緩存的重要性
數據寫入時,從內存到磁盤,這個頁面就會被標記爲髒頁,一旦被標記爲髒頁,它就會被刷新到os緩存,然後寫入磁盤。所以如果os高速緩存大小較小,則它不能重新排序寫入並優化io,這對於繁重的寫入來說非常致命,因此os的緩存大小也非常重要。給予shared_buffers太大或太小都會損害性能。
查看shared_buffers,os緩存
這裏需要使用到兩個插件,pg_bufferscache系統已經自帶可以直接創建擴展,pgfincore需要安裝詳細的步驟
pg_buffered表佔用緩存大小
pg_buffer_percent:表佔用整個緩存的佔比
percent_of_relation:緩存數據和該表數據佔比
os_cache_mb:表佔用os系統緩存大小
os_cache_percent_of_relation:os緩存和表佔比
rel_size:整個表大小
pgbench=# select c.relname,pg_size_pretty(count(*) * 8192) as pg_buffered,
round(100.0 * count(*) / (select setting from pg_settings where name='shared_buffers')::integer,1) as pgbuffer_percent,
round(100.0*count(*)*8192 / pg_table_size(c.oid),1) as percent_of_relation,
(select round( sum(pages_mem) * 4 /1024,0 ) from pgfincore(c.relname::text) ) as os_cache_MB ,
round(100 * ( select sum(pages_mem)*4096 from pgfincore(c.relname::text) )/ pg_table_size(c.oid),1) as os_cache_percent_of_relation,
pg_size_pretty(pg_table_size(c.oid)) as rel_size
from pg_class c
inner join pg_buffercache b on b.relfilenode=c.relfilenode
inner join pg_database d on (b.reldatabase=d.oid and d.datname=current_database()
and c.relnamespace=(select oid from pg_namespace where nspname='public'))
group by c.oid,c.relname
order by 3 desc limit 30;
relname | pg_buffered | pgbuffer_percent | percent_of_relation | os_cache_mb | os_cache_percent_of_relation | rel_size
-----------------------+-------------+------------------+---------------------+-------------+------------------------------+----------
pgbench_accounts | 471 MB | 1.9 | 7.3 | 495 | 7.7 | 6416 MB
pgbench_accounts_pkey | 139 MB | 0.6 | 13.0 | 274 | 25.6 | 1071 MB
pgbench_history | 2704 kB | 0.0 | 86.9 | 3 | 99.2 | 3112 kB
pgbench_branches_pkey | 56 kB | 0.0 | 100.0 | 0 | 100.0 | 56 kB
pgbench_tellers_pkey | 240 kB | 0.0 | 100.0 | 0 | 100.0 | 240 kB
pgbench_branches | 2968 kB | 0.0 | 70.7 | 4 | 99.2 | 4200 kB
pgbench_tellers | 608 kB | 0.0 | 100.0 | 1 | 94.7 | 608 kB
(7 rows)
--表緩存預熱
pgbench=# select pg_prewarm('pgbench_accounts', 'buffer', 'main');
pg_prewarm
------------
820956
(1 row)
--索引預熱:
pgbench=# select pg_prewarm('pgbench_accounts_pkey', 'buffer', 'main');
pg_prewarm
------------
137099
(1 row)
--預熱後查看緩存
pgbench=# select c.relname,pg_size_pretty(count(*) * 8192) as pg_buffered,
round(100.0 * count(*) / (select setting from pg_settings where name='shared_buffers')::integer,1) as pgbuffer_percent,
round(100.0*count(*)*8192 / pg_table_size(c.oid),1) as percent_of_relation,
(select round( sum(pages_mem) * 4 /1024,0 ) from pgfincore(c.relname::text) ) as os_cache_MB ,
round(100 * ( select sum(pages_mem)*4096 from pgfincore(c.relname::text) )/ pg_table_size(c.oid),1) as os_cache_percent_of_relation,
pg_size_pretty(pg_table_size(c.oid)) as rel_size
from pg_class c
inner join pg_buffercache b on b.relfilenode=c.relfilenode
inner join pg_database d on (b.reldatabase=d.oid and d.datname=current_database()
and c.relnamespace=(select oid from pg_namespace where nspname='public'))
group by c.oid,c.relname
order by 3 desc limit 30;
relname | pg_buffered | pgbuffer_percent | percent_of_relation | os_cache_mb | os_cache_percent_of_relation | rel_size
-----------------------+-------------+------------------+---------------------+-------------+------------------------------+----------
pgbench_accounts | 6414 MB | 26.1 | 100.0 | 6414 | 100.0 | 6416 MB
pgbench_accounts_pkey | 139 MB | 0.6 | 13.0 | 274 | 25.6 | 1071 MB
pgbench_history | 2704 kB | 0.0 | 86.9 | 3 | 99.2 | 3112 kB
pgbench_branches_pkey | 56 kB | 0.0 | 100.0 | 0 | 100.0 | 56 kB
pgbench_tellers_pkey | 240 kB | 0.0 | 100.0 | 0 | 100.0 | 240 kB
pgbench_branches | 2968 kB | 0.0 | 70.7 | 4 | 99.2 | 4200 kB
pgbench_tellers | 608 kB | 0.0 | 100.0 | 1 | 94.7 | 608 kB
(7 rows)
可以看到將數據加載至shared_buffers,並且os也緩存了一份。正常情況os不應該緩存這麼多的數據。
如何設定shared_buffers?
使用pg_buffercache可查看緩存使用情況,以及命中次數和髒塊
--1.緩存命中數
pgbench=# select usagecount,count(*),isdirty from pg_buffercache group by isdirty, usagecount order by isdirty, usagecount ;
usagecount | count | isdirty
------------+---------+---------
1 | 6651 | f
2 | 762250 | f
3 | 54684 | f
4 | 12630 | f
5 | 3940 | f
| 2305573 |
(6 rows)
--2.數據在緩存中佔比
pgbench=# SELECT
c.relname,pg_size_pretty(count(*) * 8192) as buffered,
round(100.0 * count(*) /(SELECT setting FROM pg_settings WHERE name='shared_buffers')::integer,1)AS buffers_percent,
round(100.0 * count(*) * 8192 /pg_relation_size(c.oid),1)AS percent_of_relation
FROM pg_class c
INNER JOIN pg_buffercache b ON b.relfilenode = c.relfilenode
INNER JOIN pg_database d ON (b.reldatabase = d.oid AND d.datname = current_database())
GROUP BY c.oid,c.relname
ORDER BY 3 DESC LIMIT 10;
relname | buffered | buffers_percent | percent_of_relation
-----------------------+------------+-----------------+---------------------
pgbench_accounts | 6414 MB | 26.1 | 100.0
pgbench_accounts_pkey | 1071 MB | 4.4 | 100.0
pg_amop | 56 kB | 0.0 | 87.5
pg_cast | 16 kB | 0.0 | 100.0
pg_constraint | 8192 bytes | 0.0 | 100.0
pg_index | 32 kB | 0.0 | 100.0
pg_opclass | 16 kB | 0.0 | 66.7
pg_namespace | 8192 bytes | 0.0 | 100.0
pg_operator | 120 kB | 0.0 | 100.0
pg_amproc | 40 kB | 0.0 | 100.0
(10 rows)
緩存中存儲了完整的表,和索引,佔總緩存的30%,佔比很低緩存剩餘很多。
1.如果大量的usagecount都是4或者5,那表明shared_buffers不夠,應該擴大shared_buffers;
2.如果大量的usagecount都是0或者1,那表明shared_buffers過大,應該減小shared_buffers;
每當共享內存中使用一個塊時,他就會增加一次時鐘掃描算法,範圍從1-5。4和5標識極高的使用數據塊,高使用可能會保留在shared_buffers中(有空間),如果需要更高使用率的空間,則低使用率的塊將被移除,一般簡單的插入或者更新會將使用次數設置爲1。
緩存佔比低。可以確定的是如果我們的數據集非常小,那麼設置較大的shared_buffers,沒什麼區別。
pgbench性能測試(shared_buffers 128MB,4GB,8GB,24GB)
PostgreSQL默認測試腳本,含UPDATE、INSERT還有SELECT等操作。通過修改shared_buffers大小來測試tps。
數據庫版本:PostgreSQL 10.4 (ArteryBase 5.0.0, Thunisoft)
操作系統配置:CentOS Linux release 7 ,32GB內存,8 cpu
測試參數:初始化5000w數據:pgbench -i -s 500 -h localhost -U sa -d pgbench
測試方法:pgbench -c 500 -t 20 -n -r pgbench 模擬500客戶端,每個客戶端20個事務,每種配置參數執行三次,記錄tps值。
數據庫物理大小:數據庫總大小7503 MB,其中表總大小pgbench_accounts:7487 MB,索引pgbench_accounts_pkey :1071 MB
測試腳本:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
9.478 BEGIN;
14.575 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
6.758 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
130.573 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
786.933 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
5.355 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
1242.835 END;
未預熱緩存測試結果:
序號 | 第一次 | 第二次 | 第三次 | 平均值(tps)
-----------------------------+---------+---------+---------+---------
shared_buffers=128MB(默認)| 249 | 126 | 145 | 173
shared_buffers=4GB | 357 | 357 | 373 | 362
shared_buffers=8GB | 362 | 363 | 415 | 380
shared_buffers=24GB | 378 | 368 | 397 | 381
shared_buffers設置爲8GB(25%)和設置爲24GB(75%)差別不大。
預熱緩存測試結果:
序號 | 第一次 | 第二次 | 第三次 | 平均值(tps)
-----------------------------+---------+---------+---------+---------
shared_buffers=128MB(默認)| 211 | 194 | 207 | 204
shared_buffers=4GB | 1225 | 1288 | 1321 | 1278
shared_buffers=8GB | 1176 | 1291 | 1144 | 1203
shared_buffers=24GB | 1285 | 1250 | 1309 | 1281
當shared_buffers=4GB時,數據6GB不能完全裝下,所以優先預熱索引,將索引加載到緩存,然後在加載數據。可以看到最終shared_buffers=4GB的tps和8GB,24GB表現差別不大。
內存結構
1.本地內存:work_mem,maintenance_work_mem,temp_buffer,進程分配
2.共享內存:shared_buffers,wal buffers,commitLog buffer
本地內存*max_connections+共享內存+服務器使用內存<=總內存
小結
1.大多數情況設置shared_buffers爲內存的25%,當然爲了最優可以根據命中,以及緩存佔比調整。
2.設置shared_buffers爲75%和25%相差不大,也和數據量一共只有7G+有關係。但是os系統緩存同樣重要,而設置爲75%,可能會超過總內存。
3.設置所有的緩存需要注意不要超過總內存大小。
4.在預熱數據的過程中可以考慮先做索引的預熱,因爲要做索引的情況加載索引會比較慢。