一、環境說明：
此次演示是dellr620 物理服務器，系統爲centos7.6 x86_64位
硬件相關的信息如下：
[root@mysql-redis scripts]# sh hw_msg1.sh
+--------------------------------------------------------------+
| This Machine's Hyper-Threading is Enabled(recommend disable) |
+--------------------------------------------------------------+
Systembit : 64
MEM info : 20*4096 MB
Disk_totle :
Pro_SN_name : Product Name: PowerEdge R620 Serial Number: 5RV2962
System name : \S
Board_SN name: Product Name: Serial Number:
CPU model : 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
CPU_phy_num : 2
CPU_core_num : 8
CPU_logic_num: 16
Netcard info : Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
+++++++++++++++++++++++++++++++++++
<<raid informations>>
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 893.75 GB
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
--
Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 223.0 GB
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
+++++++++++++++++++++++++++++++++++
<<Virtual Drive Information>>
Device Id: 3
PD Type: SATA
Raw Size: 894.252 GB [0x6fc81ab0 Sectors]
Inquiry Data: 56AB609UKCDU TOSHIBA Q300. SAFM12.3
Device Speed: 6.0Gb/s
Drive Temperature : N/A
Device Id: 0
PD Type: SATA
Raw Size: 223.570 GB [0x1bf244b0 Sectors]
Inquiry Data: PHDV723500E8240AGN INTEL SSDSC2BB240G7 N2010112
Device Speed: 6.0Gb/s
Drive Temperature :25C (77.00 F)二、安裝fio
1.redhat linux系統直接yum 安裝
yum install fio.x86_64
centos7.6系統上直接yum 安裝的話，默認的版本是fio.3.7
2.源碼編譯安裝
爲了測試異步io，需要安裝libaio
yum install libaio-devel
wget -P /data/soft http://github.com/axboe/fio/archive/fio-3.7.tar.gz
tar xf /data/soft/fio.3.7.tar.gz -C /usr/local/
cd /usr/local/fio.3.7
./configure
make && make install三、重要參數介紹
Fio各個版本的參數不盡相同，這裏只列出日常較爲廣發使用的重要參數
filename=/dev/sdb1 或者/data/soft/test1 ： 指定測試硬盤名稱或者測試文件名稱（切忌選擇系統設備測試，會導致系統奔潰的）
direct=1: 測試過程繞過機器自帶的緩衝區，使得測試的結果更準確。
rw=randwrite: 隨機寫。 rw可取值：randwrite/randread/read/write/randrw/trim/randtrim/trimwrite ;分別是隨機寫，隨機讀，順序讀，順序寫，混合隨機讀寫，
bs=16k : 單次I/O塊大小爲16KB,單次I/O大小對IOPS的影響比較大，一般來說要想得到最大的IOPS，bs越小越好。當然，這也和文件系統最小快的大小有關。一般linux下的文件分區最小快的大小爲512B,1KB,2KB,4KB,8KB等可以根據文件系統類型進行選擇。
size=5g : 測試文件大小爲5GB
ioengine=libaio: I/O引擎使用libaio,libaio是異步的I/O引擎，一次提交一批I/O,然後等待這批I/O完成，這種方式減少了交互的次數，效率更好。FIO 至少支持13種不通的I/O引擎。sync,mmap,libaio,network等
iodepth=4 : I/O隊列深度，主要是根據設備的並行度來調整。通常有兩種I/O訪問方式。也就是同步I/O，和異步I/O.同步I/O一次只能發送一個I/o請求，等待內核完成纔可以返回。這對於單個線程來說iodepth總是小於1.想要iodepth大於1，可以通過多線程併發執行做到。異步I/O一次提交多個I/O請求。等I/O完成或者間隔一段時間收割一次。iodepth一般用在異步I/O模型（libaio）,用於指定一次發起多少個I/O請求。Flash存儲一般有多個並行的單元。支持多個I/O併發執行。較合適異步的I/O
-numjobs=4: 使用4個線程對硬盤設備進行壓測 對於Raid設備，如raid1,raid5或者並行高的設備，可以考慮適當加大測試線程數-numjobs來測試。
runtime=1000: 測試時間爲1000s.和參數size共同起作用，屬於雙限制，即達到任何一個限制都停止測試。
rwmixread=70: 在混合讀寫模式下，讀佔70%
group_reporting: 設置顯示結果，指定group_reporting 將彙總每個進程的信息。
-name iops_randwrite： 改測試命名爲 iops_randwrite
--output TestResult.log: 測試結果指定輸出到文件TestResult.log 中
四、測試io性能
可以使用fio模擬測試各種不同類型的io請求，入隨機寫，順序寫，隨機讀，順序讀。混合隨機讀寫
4.1隨機寫：
time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite --output TestResult.log 命令介紹：
採用4個線程對 /data/soft/test1文件做持續時間爲60s，隊列深度爲1，塊大小爲4kb的direct y異步隨機寫（libaio）測試，該測試命名爲iops_randwrite.輸出結果不安4個jobs展示，而是按照group 組彙總 ，這樣得出的在此壓力下從此文件系統的隨機寫的IOPS。對於Raid設備，如raid1,raid5或者並行高的設備，可以考慮適當加大測試線程數-numjobs來測試。
--output TestResult.log 輸出到指定的日誌
下面是隨機寫具體執行過程：
[root@mysql-redis soft]# time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite
iops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 processes
iops_randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=68.2MiB/s][r=0,w=17.5k IOPS][eta 00m:00s]
iops_randwrite: (groupid=0, jobs=4): err= 0: pid=13505: Thu May 21 18:50:33 2020
write: IOPS=39.2k, BW=153MiB/s (160MB/s)(9182MiB/60001msec)
slat (usec): min=6, max=27813, avg=17.42, stdev=36.21
clat (nsec): min=1177, max=123217k, avg=82183.17, stdev=323538.76
lat (usec): min=53, max=123230, avg=99.84, stdev=325.69
clat percentiles (usec):
| 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 57], 20.00th=[ 62],
| 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 79],
| 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 103], 95.00th=[ 117],
| 99.00th=[ 161], 99.50th=[ 206], 99.90th=[ 330], 99.95th=[ 619],
| 99.99th=[14615]
bw ( KiB/s): min=17536, max=45304, per=25.12%, avg=39354.01, stdev=5145.44, samples=476
iops : min= 4384, max=11326, avg=9838.50, stdev=1286.36, samples=476
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.22%
lat (usec) : 100=84.07%, 250=11.50%, 500=0.12%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.33%, sys=19.34%, ctx=2824889, majf=0, minf=1185
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2350467,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9182MiB (9628MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=0/2349456, merge=0/0, ticks=0/167181, in_queue=166516, util=96.46%
real 1m0.706s
user 0m11.167s
sys 0m47.017s4.2隨機讀：
time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 關於命令參數說明：
命令介紹：採用2個線程對 /data/soft/test1文件做持續時間爲60s，隊列深度爲1，塊大小爲4kb的direct y異步隨機讀（libaio）測試，該測試命名爲iops_randread.輸出結果不安2個jobs展示，而是按照group 組彙總 ，這樣得出的在此壓力下從此文件系統的隨機讀的IOPS
下面是隨機讀具體執行過程：
[root@mysql-redis39 ~]# time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread
iops_randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=36.7MiB/s,w=0KiB/s][r=9389,w=0 IOPS][eta 00m:00s]
iops_randread: (groupid=0, jobs=2): err= 0: pid=24566: Fri May 22 06:21:46 2020
read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec)
slat (usec): min=7, max=1480, avg=23.09, stdev= 8.09
clat (usec): min=8, max=123984, avg=185.97, stdev=570.65
lat (usec): min=136, max=123999, avg=209.50, stdev=570.70
clat percentiles (usec):
| 1.00th=[ 139], 5.00th=[ 141], 10.00th=[ 145], 20.00th=[ 169],
| 30.00th=[ 174], 40.00th=[ 178], 50.00th=[ 182], 60.00th=[ 188],
| 70.00th=[ 198], 80.00th=[ 204], 90.00th=[ 212], 95.00th=[ 219],
| 99.00th=[ 255], 99.50th=[ 273], 99.90th=[ 322], 99.95th=[ 537],
| 99.99th=[ 1549]
bw ( KiB/s): min=13824, max=19616, per=49.99%, avg=18713.87, stdev=1060.66, samples=238
iops : min= 3456, max= 4904, avg=4678.43, stdev=265.16, samples=238
lat (usec) : 10=0.01%, 100=0.01%, 250=98.87%, 500=1.08%, 750=0.02%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.01%, 20=0.01%, 250=0.01%
cpu : usr=4.11%, sys=15.44%, ctx=561546, majf=0, minf=498
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=561503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=2193MiB (2300MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=560393/2, merge=0/0, ticks=102314/0, in_queue=101910, util=97.80%
real 1m0.706s
user 0m5.649s
sys 0m19.196s以上需要關注的指標：
read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec) io=2193MiB (2300MB) 這些參數可以瞭解到iops和通途量
lat (usec): min=136, max=123999, avg=209.50, stdev=570.70 這些參數也要注意：io延遲平均在209.5微妙。
4.3順序寫：
time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write
說明：採用4個線程對/data/soft/test3文件做持續60s，隊列深度爲1的塊大小爲1M的direct異步順序寫（libaio）壓力測試該測試命名爲bw_write.輸出結果不安4個job分別展示，而是按照group彙總展示。
[root@mysql-redis ~]# time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write
bw_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 5 processes
bw_write: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [W(5)][100.0%][r=0KiB/s,w=184MiB/s][r=0,w=184 IOPS][eta 00m:00s]
bw_write: (groupid=0, jobs=5): err= 0: pid=26278: Fri May 22 06:44:38 2020
write: IOPS=319, BW=320MiB/s (335MB/s)(18.7GiB/60009msec)
slat (usec): min=56, max=303, avg=140.02, stdev=29.86
clat (msec): min=3, max=1092, avg=15.49, stdev=67.68
lat (msec): min=3, max=1093, avg=15.63, stdev=67.68
4.4順序讀：
time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read [root@mysql-redis ~]# time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read
bw_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=259MiB/s,w=0KiB/s][r=259,w=0 IOPS][eta 00m:00s]
bw_read: (groupid=0, jobs=2): err= 0: pid=26789: Fri May 22 06:51:01 2020
read: IOPS=255, BW=256MiB/s (268MB/s)(10.0GiB/40034msec)
slat (usec): min=95, max=1149, avg=245.64, stdev=42.23
clat (msec): min=4, max=130, avg= 7.57, stdev= 2.98
lat (msec): min=4, max=130, avg= 7.81, stdev= 2.984.5混合讀寫
time fio -rw=randrw -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw命令介紹：
2個線程對/data/soft/test2 文件做持續時間爲60s，隊列深度爲1，塊大小爲16kb的direct異步混合讀寫的（libaio）壓力測試，讀寫比僞7:3，測試被命名爲iops_randrw
rwmixwrite=30 意思是在混合讀寫的模式下，寫佔30%。
-rwmixread=70 意思是在混合讀寫的模式下，讀佔70%。
混合讀寫具體輸出結果：
[root@mysql-redis ~]# time fio -rw=randrw -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw
iops_randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=62.0MiB/s,w=27.2MiB/s][r=4030,w=1740 IOPS][eta 00m:00s]
iops_randrw: (groupid=0, jobs=2): err= 0: pid=27095: Fri May 22 06:55:08 2020
read: IOPS=3931, BW=61.4MiB/s (64.4MB/s)(3686MiB/60001msec)
slat (usec): min=8, max=6353, avg=36.20, stdev=19.77
clat (nsec): min=1648, max=123466k, avg=346175.56, stdev=725094.33
lat (usec): min=12, max=123516, avg=382.99, stdev=725.55
clat percentiles (usec):
| 1.00th=[ 215], 5.00th=[ 241], 10.00th=[ 255], 20.00th=[ 277],
| 30.00th=[ 293], 40.00th=[ 310], 50.00th=[ 322], 60.00th=[ 334],
| 70.00th=[ 351], 80.00th=[ 375], 90.00th=[ 412], 95.00th=[ 449],
| 99.00th=[ 1221], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1827],
| 99.99th=[16712]
bw ( KiB/s): min=23328, max=33792, per=49.97%, avg=31431.39, stdev=1649.11, samples=238
iops : min= 1458, max= 2112, avg=1964.44, stdev=103.06, samples=238
write: IOPS=1686, BW=26.4MiB/s (27.6MB/s)(1581MiB/60001msec)
slat (usec): min=12, max=2005, avg=42.06, stdev=16.09
clat (usec): min=3, max=123270, avg=225.73, stdev=432.84
lat (usec): min=86, max=123320, avg=268.44, stdev=433.42
clat percentiles (usec):
| 1.00th=[ 78], 5.00th=[ 85], 10.00th=[ 100], 20.00th=[ 129],
| 30.00th=[ 149], 40.00th=[ 174], 50.00th=[ 202], 60.00th=[ 233],
| 70.00th=[ 265], 80.00th=[ 306], 90.00th=[ 359], 95.00th=[ 408],
| 99.00th=[ 510], 99.50th=[ 1090], 99.90th=[ 1549], 99.95th=[ 1614],
| 99.99th=[ 5080]
bw ( KiB/s): min= 9152, max=15329, per=49.97%, avg=13484.37, stdev=892.91, samples=238
iops : min= 572, max= 958, avg=842.76, stdev=55.80, samples=238
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=3.06%
lat (usec) : 250=22.22%, 500=72.91%, 750=0.76%, 1000=0.05%
lat (msec) : 2=0.96%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.64%, sys=15.91%, ctx=346233, majf=0, minf=266
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=235884,101198,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=61.4MiB/s (64.4MB/s), 61.4MiB/s-61.4MiB/s (64.4MB/s-64.4MB/s), io=3686MiB (3865MB), run=60001-60001msec
WRITE: bw=26.4MiB/s (27.6MB/s), 26.4MiB/s-26.4MiB/s (27.6MB/s-27.6MB/s), io=1581MiB (1658MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=274477/101294, merge=0/0, ticks=88860/19119, in_queue=107576, util=95.82%
real 1m0.690s
user 0m6.300s
sys 0m19.718s4.6SSD性能測試配置文件
配置文件內容如下：
[root@mysql-redis scripts]# cat /data/soft/ssd-test.fio
##2020-05-24
[global]
bs=4k
ioengine=libaio
iodepth=4
numjobs=1
group_reporting
size=5g
direct=1
runtime=60
directory=/data/soft
filename=ssd.test.file
#inflate-log=/data/soft/TestResult.log
#output=/data/soft/TestResult.log
[seq-read]
name=iops_seqread
rw=read
stonewall
[rand-read]
name=iops_randread
rw=randread
stonewall
[seq-write]
name=iops_seqwrite
rw=write
stonewall
[rand-write]
name=iops_randwrite
rw=randwrite
stonewall
[randrw]
name=iops_randrw
rwmixread=70
rw=randrw
stonewall下面是fio測試指定配置文件輸出的內容：
[root@mysql-redis soft]# fio ssd-test.fio
iops_seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randread: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_seqwrite: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randwrite: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randrw: (g=4): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.7
Starting 5 processes
iops_seqread: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [_(4),m(1)][90.5%][r=47.8MiB/s,w=20.7MiB/s][r=12.2k,w=5308 IOPS][eta 00m:21s]
iops_seqread: (groupid=0, jobs=1): err= 0: pid=2142: Sun May 24 20:06:20 2020
read: IOPS=57.5k, BW=225MiB/s (236MB/s)(5120MiB/22782msec)
slat (usec): min=3, max=1943, avg= 5.13, stdev= 2.21
clat (usec): min=25, max=12467, avg=63.13, stdev=202.65
lat (usec): min=40, max=12472, avg=68.41, stdev=202.72
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 56],
| 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 57],
| 70.00th=[ 57], 80.00th=[ 58], 90.00th=[ 59], 95.00th=[ 62],
| 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 2073], 99.95th=[ 5080],
| 99.99th=[10814]
bw ( KiB/s): min=43579, max=193812, per=75.80%, avg=174446.82, stdev=47419.87, samples=45
iops : min=10894, max=48453, avg=43611.36, stdev=11855.06, samples=45
lat (usec) : 50=3.39%, 100=96.25%, 250=0.08%, 500=0.07%, 750=0.04%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.06%, 4=0.04%, 10=0.05%, 20=0.01%
cpu : usr=19.83%, sys=39.29%, ctx=1222141, majf=0, minf=114
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_randread: (groupid=1, jobs=1): err= 0: pid=2170: Sun May 24 20:06:20 2020
read: IOPS=21.7k, BW=84.9MiB/s (89.0MB/s)(5094MiB/60001msec)
slat (usec): min=3, max=1068, avg=12.62, stdev= 6.14
clat (usec): min=42, max=38568, avg=167.76, stdev=315.76
lat (usec): min=62, max=38588, avg=180.75, stdev=316.07
clat percentiles (usec):
| 1.00th=[ 130], 5.00th=[ 135], 10.00th=[ 137], 20.00th=[ 141],
| 30.00th=[ 143], 40.00th=[ 147], 50.00th=[ 151], 60.00th=[ 155],
| 70.00th=[ 161], 80.00th=[ 172], 90.00th=[ 192], 95.00th=[ 206],
| 99.00th=[ 251], 99.50th=[ 273], 99.90th=[ 3982], 99.95th=[ 7308],
| 99.99th=[14091]
bw ( KiB/s): min= 5397, max=74196, per=79.00%, avg=68682.74, stdev=15713.88, samples=119
iops : min= 1349, max=18549, avg=17170.32, stdev=3928.48, samples=119
lat (usec) : 50=0.01%, 100=0.02%, 250=98.94%, 500=0.82%, 750=0.02%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.03%, 4=0.04%, 10=0.07%, 20=0.02%, 50=0.01%
cpu : usr=11.52%, sys=37.46%, ctx=668232, majf=0, minf=319
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1304064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_seqwrite: (groupid=2, jobs=1): err= 0: pid=2231: Sun May 24 20:06:20 2020
write: IOPS=47.1k, BW=184MiB/s (193MB/s)(5120MiB/27848msec)
slat (usec): min=4, max=139, avg= 7.50, stdev= 2.96
clat (usec): min=27, max=15374, avg=76.14, stdev=278.53
lat (usec): min=49, max=15381, avg=83.78, stdev=279.26
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 59], 10.00th=[ 60], 20.00th=[ 61],
| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 67],
| 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 73], 95.00th=[ 74],
| 99.00th=[ 81], 99.50th=[ 89], 99.90th=[ 3326], 99.95th=[ 3392],
| 99.99th=[15139]
bw ( KiB/s): min= 3769, max=175262, per=80.83%, avg=152166.87, stdev=43009.10, samples=55
iops : min= 942, max=43815, avg=38041.35, stdev=10752.22, samples=55
lat (usec) : 50=0.24%, 100=99.43%, 250=0.07%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.21%, 10=0.01%, 20=0.03%
cpu : usr=11.79%, sys=43.48%, ctx=652945, majf=0, minf=95
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_randwrite: (groupid=3, jobs=1): err= 0: pid=2262: Sun May 24 20:06:20 2020
write: IOPS=47.2k, BW=185MiB/s (194MB/s)(5120MiB/27743msec)
slat (usec): min=4, max=415, avg= 7.49, stdev= 2.82
clat (usec): min=26, max=21032, avg=75.63, stdev=276.91
lat (usec): min=48, max=21058, avg=83.27, stdev=277.60
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 62],
| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 67],
| 70.00th=[ 68], 80.00th=[ 70], 90.00th=[ 73], 95.00th=[ 74],
| 99.00th=[ 80], 99.50th=[ 86], 99.90th=[ 3326], 99.95th=[ 3392],
| 99.99th=[15139]
bw ( KiB/s): min= 3175, max=142274, per=68.21%, avg=128901.20, stdev=33671.41, samples=55
iops : min= 793, max=35568, avg=32224.95, stdev=8417.81, samples=55
lat (usec) : 50=0.21%, 100=99.50%, 250=0.04%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.20%, 10=0.01%, 20=0.02%, 50=0.01%
cpu : usr=12.97%, sys=43.05%, ctx=654916, majf=0, minf=258
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_randrw: (groupid=4, jobs=1): err= 0: pid=2292: Sun May 24 20:06:20 2020
read: IOPS=11.3k, BW=44.3MiB/s (46.5MB/s)(2659MiB/60001msec)
slat (usec): min=4, max=594, avg=13.08, stdev= 6.88
clat (usec): min=61, max=45073, avg=244.45, stdev=441.39
lat (usec): min=134, max=45115, avg=257.84, stdev=441.61
clat percentiles (usec):
| 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 176],
| 30.00th=[ 190], 40.00th=[ 202], 50.00th=[ 212], 60.00th=[ 221],
| 70.00th=[ 233], 80.00th=[ 258], 90.00th=[ 330], 95.00th=[ 359],
| 99.00th=[ 474], 99.50th=[ 1270], 99.90th=[ 6521], 99.95th=[10028],
| 99.99th=[17695]
bw ( KiB/s): min= 3072, max=51824, per=99.92%, avg=45336.55, stdev=11250.66, samples=119
iops : min= 768, max=12958, avg=11334.12, stdev=2812.66, samples=119
write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(1139MiB/60001msec)
slat (usec): min=6, max=874, avg=16.94, stdev= 7.42
clat (usec): min=40, max=31265, avg=192.80, stdev=413.19
lat (usec): min=56, max=31306, avg=210.09, stdev=413.32
clat percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 78],
| 30.00th=[ 145], 40.00th=[ 169], 50.00th=[ 188], 60.00th=[ 202],
| 70.00th=[ 212], 80.00th=[ 227], 90.00th=[ 251], 95.00th=[ 277],
| 99.00th=[ 619], 99.50th=[ 1205], 99.90th=[ 5997], 99.95th=[ 9634],
| 99.99th=[16581]
bw ( KiB/s): min= 1184, max=22176, per=99.94%, avg=19428.64, stdev=4847.41, samples=119
iops : min= 296, max= 5544, avg=4857.13, stdev=1211.85, samples=119
lat (usec) : 50=0.12%, 100=7.22%, 250=74.36%, 500=17.22%, 750=0.24%
lat (usec) : 1000=0.10%
lat (msec) : 2=0.52%, 4=0.06%, 10=0.11%, 20=0.04%, 50=0.01%
cpu : usr=7.88%, sys=29.46%, ctx=340101, majf=0, minf=374
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=680630,291634,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=5120MiB (5369MB), run=22782-22782msec
Run status group 1 (all jobs):
READ: bw=84.9MiB/s (89.0MB/s), 84.9MiB/s-84.9MiB/s (89.0MB/s-89.0MB/s), io=5094MiB (5341MB), run=60001-60001msec
Run status group 2 (all jobs):
WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5120MiB (5369MB), run=27848-27848msec
Run status group 3 (all jobs):
WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5120MiB (5369MB), run=27743-27743msec
Run status group 4 (all jobs):
READ: bw=44.3MiB/s (46.5MB/s), 44.3MiB/s-44.3MiB/s (46.5MB/s-46.5MB/s), io=2659MiB (2788MB), run=60001-60001msec
WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=1139MiB (1195MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=3294785/2912828, merge=0/3, ticks=430527/238028, in_queue=666582, util=99.09%
[root@mysql-redis soft]# 五、總結
關於fio測試硬盤的建議：
一、建議使用順序I/O和較大的blocksize 來測試設備的通吐量和延遲
二、建議使用隨機I/O和較小的blocksize來測試設備的IOPS和延遲
三、在配置numjobs和iodepth 測試底層存儲性能時，建議要深入瞭解應用到底採用的是同步的io還是異步的io（是多進程併發i/o請求還是一次提交一批的i/o的請求）
提示：對fio壓測 磁盤性能感興趣的話，可以參考下ezfio(直接在github.com上搜索ezfio)
重要提示：fio是 支持對裸設備直接進行I/O測試的，在測試裸設備時，一定要避開系統所在的裸設備，否則會導致系統奔潰。
例如：
[root@mysql-redis soft]# df -h / /boot
文件系統 容量 已用 可用 已用% 掛載點
/dev/sdb5 207G 7.5G 200G 4% /
/dev/sdb1 297M 120M 177M 41% /boot/dev/sdb /dev/sdb5 /dev/sdb1 這些設備是絕對禁止進行fio測試的，
同時在測試其他的設備裸設備時，一定要提前確定好此裸設備上是否有重要 的數據，一定要重要的設備提前做好備份。
以上就是我這邊對fio測試硬盤的簡單學習總結，僅供參考，如有網友直接複製應用造成損失，本博文不負擔任何責任。
參考文檔：
http://www.ssdfans.com/?p=6087
https://linux.die.net/man/1/fio
https://blog.csdn.net/don_chiang709/article/details/92628623