公司短信服務器下發併發在2000條/s,需要先把短信寫入到本地磁盤然後發送。同事反應發送短信慢,有事發送失敗。
通過查看磁盤io發現:
[root@wmonstermt1] ~# iostat -x 1
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 34.5 63.6 141.8 1370.5 2143 737.3 30
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 1.0 50.0 4.0 1243.7 143 1256 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 29.0 0.0 1038.3 2267 782 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
可以發現服務時間和等待時間超長。說明磁盤IO不夠用了。如何解決?臨時解決方案:
查看當前raid卡陣列IO策略:
[root@wmonstermt1] ~# MegaCli64 -LDInfo -Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 2.178 TB
Sector Size : 512
Parity Size : 557.861 GB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 5
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
PI type: No PI
Is VD Cached: No
更改Raid卡策略吧:
[root@wmonstermt3] ~# MegaCli64 -LDSetProp WB -L0 -a0
[root@wmonstermt3] ~# MegaCli64 -LDSetProp CachedBadBBU -Lall -aALL
[root@wmonstermt3] ~# MegaCli64 -LDInfo -Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 2.178 TB
Sector Size : 512
Parity Size : 557.861 GB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 5
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
PI type: No PI
Is VD Cached: No
[root@wmonstermt3] ~# iostat -x 1
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 17.5 57.0 77.0 1348.1 0 537.2 28
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 1.0 0.0 127.9 0 0.1 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 4.0 0.0 511.6 0 0.1 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 5.0 0.0 447.6 0 0.1 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
發現效果槓槓滴。