一、环境说明：
此次演示是dellr620 物理服务器，系统为centos7.6 x86_64位
硬件相关的信息如下：
[root@mysql-redis scripts]# sh hw_msg1.sh
+--------------------------------------------------------------+
| This Machine's Hyper-Threading is Enabled(recommend disable) |
+--------------------------------------------------------------+
Systembit : 64
MEM info : 20*4096 MB
Disk_totle :
Pro_SN_name : Product Name: PowerEdge R620 Serial Number: 5RV2962
System name : \S
Board_SN name: Product Name: Serial Number:
CPU model : 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
CPU_phy_num : 2
CPU_core_num : 8
CPU_logic_num: 16
Netcard info : Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
+++++++++++++++++++++++++++++++++++
<<raid informations>>
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 893.75 GB
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
--
Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 223.0 GB
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
+++++++++++++++++++++++++++++++++++
<<Virtual Drive Information>>
Device Id: 3
PD Type: SATA
Raw Size: 894.252 GB [0x6fc81ab0 Sectors]
Inquiry Data: 56AB609UKCDU TOSHIBA Q300. SAFM12.3
Device Speed: 6.0Gb/s
Drive Temperature : N/A
Device Id: 0
PD Type: SATA
Raw Size: 223.570 GB [0x1bf244b0 Sectors]
Inquiry Data: PHDV723500E8240AGN INTEL SSDSC2BB240G7 N2010112
Device Speed: 6.0Gb/s
Drive Temperature :25C (77.00 F)二、安装fio
1.redhat linux系统直接yum 安装
yum install fio.x86_64
centos7.6系统上直接yum 安装的话，默认的版本是fio.3.7
2.源码编译安装
为了测试异步io，需要安装libaio
yum install libaio-devel
wget -P /data/soft http://github.com/axboe/fio/archive/fio-3.7.tar.gz
tar xf /data/soft/fio.3.7.tar.gz -C /usr/local/
cd /usr/local/fio.3.7
./configure
make && make install三、重要参数介绍
Fio各个版本的参数不尽相同，这里只列出日常较为广发使用的重要参数
filename=/dev/sdb1 或者/data/soft/test1 ： 指定测试硬盘名称或者测试文件名称（切忌选择系统设备测试，会导致系统奔溃的）
direct=1: 测试过程绕过机器自带的缓冲区，使得测试的结果更准确。
rw=randwrite: 随机写。 rw可取值：randwrite/randread/read/write/randrw/trim/randtrim/trimwrite ;分别是随机写，随机读，顺序读，顺序写，混合随机读写，
bs=16k : 单次I/O块大小为16KB,单次I/O大小对IOPS的影响比较大，一般来说要想得到最大的IOPS，bs越小越好。当然，这也和文件系统最小快的大小有关。一般linux下的文件分区最小快的大小为512B,1KB,2KB,4KB,8KB等可以根据文件系统类型进行选择。
size=5g : 测试文件大小为5GB
ioengine=libaio: I/O引擎使用libaio,libaio是异步的I/O引擎，一次提交一批I/O,然后等待这批I/O完成，这种方式减少了交互的次数，效率更好。FIO 至少支持13种不通的I/O引擎。sync,mmap,libaio,network等
iodepth=4 : I/O队列深度，主要是根据设备的并行度来调整。通常有两种I/O访问方式。也就是同步I/O，和异步I/O.同步I/O一次只能发送一个I/o请求，等待内核完成才可以返回。这对於单个线程来说iodepth总是小于1.想要iodepth大于1，可以通过多线程并发执行做到。异步I/O一次提交多个I/O请求。等I/O完成或者间隔一段时间收割一次。iodepth一般用在异步I/O模型（libaio）,用于指定一次发起多少个I/O请求。Flash存储一般有多个并行的单元。支持多个I/O并发执行。较合适异步的I/O
-numjobs=4: 使用4个线程对硬盘设备进行压测 对于Raid设备，如raid1,raid5或者并行高的设备，可以考虑适当加大测试线程数-numjobs来测试。
runtime=1000: 测试时间为1000s.和参数size共同起作用，属于双限制，即达到任何一个限制都停止测试。
rwmixread=70: 在混合读写模式下，读占70%
group_reporting: 设置显示结果，指定group_reporting 将汇总每个进程的信息。
-name iops_randwrite： 改测试命名为 iops_randwrite
--output TestResult.log: 测试结果指定输出到文件TestResult.log 中
四、测试io性能
可以使用fio模拟测试各种不同类型的io请求，入随机写，顺序写，随机读，顺序读。混合随机读写
4.1随机写：
time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite --output TestResult.log 命令介绍：
采用4个线程对 /data/soft/test1文件做持续时间为60s，队列深度为1，块大小为4kb的direct y异步随机写（libaio）测试，该测试命名为iops_randwrite.输出结果不安4个jobs展示，而是按照group 组汇总 ，这样得出的在此压力下从此文件系统的随机写的IOPS。对于Raid设备，如raid1,raid5或者并行高的设备，可以考虑适当加大测试线程数-numjobs来测试。
--output TestResult.log 输出到指定的日志
下面是随机写具体执行过程：
[root@mysql-redis soft]# time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite
iops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 processes
iops_randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=68.2MiB/s][r=0,w=17.5k IOPS][eta 00m:00s]
iops_randwrite: (groupid=0, jobs=4): err= 0: pid=13505: Thu May 21 18:50:33 2020
write: IOPS=39.2k, BW=153MiB/s (160MB/s)(9182MiB/60001msec)
slat (usec): min=6, max=27813, avg=17.42, stdev=36.21
clat (nsec): min=1177, max=123217k, avg=82183.17, stdev=323538.76
lat (usec): min=53, max=123230, avg=99.84, stdev=325.69
clat percentiles (usec):
| 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 57], 20.00th=[ 62],
| 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 79],
| 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 103], 95.00th=[ 117],
| 99.00th=[ 161], 99.50th=[ 206], 99.90th=[ 330], 99.95th=[ 619],
| 99.99th=[14615]
bw ( KiB/s): min=17536, max=45304, per=25.12%, avg=39354.01, stdev=5145.44, samples=476
iops : min= 4384, max=11326, avg=9838.50, stdev=1286.36, samples=476
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.22%
lat (usec) : 100=84.07%, 250=11.50%, 500=0.12%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.33%, sys=19.34%, ctx=2824889, majf=0, minf=1185
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2350467,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9182MiB (9628MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=0/2349456, merge=0/0, ticks=0/167181, in_queue=166516, util=96.46%
real 1m0.706s
user 0m11.167s
sys 0m47.017s4.2随机读：
time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 关于命令参数说明：
命令介绍：采用2个线程对 /data/soft/test1文件做持续时间为60s，队列深度为1，块大小为4kb的direct y异步随机读（libaio）测试，该测试命名为iops_randread.输出结果不安2个jobs展示，而是按照group 组汇总 ，这样得出的在此压力下从此文件系统的随机读的IOPS
下面是随机读具体执行过程：
[root@mysql-redis39 ~]# time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread
iops_randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=36.7MiB/s,w=0KiB/s][r=9389,w=0 IOPS][eta 00m:00s]
iops_randread: (groupid=0, jobs=2): err= 0: pid=24566: Fri May 22 06:21:46 2020
read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec)
slat (usec): min=7, max=1480, avg=23.09, stdev= 8.09
clat (usec): min=8, max=123984, avg=185.97, stdev=570.65
lat (usec): min=136, max=123999, avg=209.50, stdev=570.70
clat percentiles (usec):
| 1.00th=[ 139], 5.00th=[ 141], 10.00th=[ 145], 20.00th=[ 169],
| 30.00th=[ 174], 40.00th=[ 178], 50.00th=[ 182], 60.00th=[ 188],
| 70.00th=[ 198], 80.00th=[ 204], 90.00th=[ 212], 95.00th=[ 219],
| 99.00th=[ 255], 99.50th=[ 273], 99.90th=[ 322], 99.95th=[ 537],
| 99.99th=[ 1549]
bw ( KiB/s): min=13824, max=19616, per=49.99%, avg=18713.87, stdev=1060.66, samples=238
iops : min= 3456, max= 4904, avg=4678.43, stdev=265.16, samples=238
lat (usec) : 10=0.01%, 100=0.01%, 250=98.87%, 500=1.08%, 750=0.02%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.01%, 20=0.01%, 250=0.01%
cpu : usr=4.11%, sys=15.44%, ctx=561546, majf=0, minf=498
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=561503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=2193MiB (2300MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=560393/2, merge=0/0, ticks=102314/0, in_queue=101910, util=97.80%
real 1m0.706s
user 0m5.649s
sys 0m19.196s以上需要关注的指标：
read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec) io=2193MiB (2300MB) 这些参数可以了解到iops和通途量
lat (usec): min=136, max=123999, avg=209.50, stdev=570.70 这些参数也要注意：io延迟平均在209.5微妙。
4.3顺序写：
time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write
说明：采用4个线程对/data/soft/test3文件做持续60s，队列深度为1的块大小为1M的direct异步顺序写（libaio）压力测试该测试命名为bw_write.输出结果不安4个job分别展示，而是按照group汇总展示。
[root@mysql-redis ~]# time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write
bw_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 5 processes
bw_write: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [W(5)][100.0%][r=0KiB/s,w=184MiB/s][r=0,w=184 IOPS][eta 00m:00s]
bw_write: (groupid=0, jobs=5): err= 0: pid=26278: Fri May 22 06:44:38 2020
write: IOPS=319, BW=320MiB/s (335MB/s)(18.7GiB/60009msec)
slat (usec): min=56, max=303, avg=140.02, stdev=29.86
clat (msec): min=3, max=1092, avg=15.49, stdev=67.68
lat (msec): min=3, max=1093, avg=15.63, stdev=67.68
4.4顺序读：
time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read [root@mysql-redis ~]# time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read
bw_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=259MiB/s,w=0KiB/s][r=259,w=0 IOPS][eta 00m:00s]
bw_read: (groupid=0, jobs=2): err= 0: pid=26789: Fri May 22 06:51:01 2020
read: IOPS=255, BW=256MiB/s (268MB/s)(10.0GiB/40034msec)
slat (usec): min=95, max=1149, avg=245.64, stdev=42.23
clat (msec): min=4, max=130, avg= 7.57, stdev= 2.98
lat (msec): min=4, max=130, avg= 7.81, stdev= 2.984.5混合读写
time fio -rw=randrw -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw命令介绍：
2个线程对/data/soft/test2 文件做持续时间为60s，队列深度为1，块大小为16kb的direct异步混合读写的（libaio）压力测试，读写比伪7:3，测试被命名为iops_randrw
rwmixwrite=30 意思是在混合读写的模式下，写占30%。
-rwmixread=70 意思是在混合读写的模式下，读占70%。
混合读写具体输出结果：
[root@mysql-redis ~]# time fio -rw=randrw -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw
iops_randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=62.0MiB/s,w=27.2MiB/s][r=4030,w=1740 IOPS][eta 00m:00s]
iops_randrw: (groupid=0, jobs=2): err= 0: pid=27095: Fri May 22 06:55:08 2020
read: IOPS=3931, BW=61.4MiB/s (64.4MB/s)(3686MiB/60001msec)
slat (usec): min=8, max=6353, avg=36.20, stdev=19.77
clat (nsec): min=1648, max=123466k, avg=346175.56, stdev=725094.33
lat (usec): min=12, max=123516, avg=382.99, stdev=725.55
clat percentiles (usec):
| 1.00th=[ 215], 5.00th=[ 241], 10.00th=[ 255], 20.00th=[ 277],
| 30.00th=[ 293], 40.00th=[ 310], 50.00th=[ 322], 60.00th=[ 334],
| 70.00th=[ 351], 80.00th=[ 375], 90.00th=[ 412], 95.00th=[ 449],
| 99.00th=[ 1221], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1827],
| 99.99th=[16712]
bw ( KiB/s): min=23328, max=33792, per=49.97%, avg=31431.39, stdev=1649.11, samples=238
iops : min= 1458, max= 2112, avg=1964.44, stdev=103.06, samples=238
write: IOPS=1686, BW=26.4MiB/s (27.6MB/s)(1581MiB/60001msec)
slat (usec): min=12, max=2005, avg=42.06, stdev=16.09
clat (usec): min=3, max=123270, avg=225.73, stdev=432.84
lat (usec): min=86, max=123320, avg=268.44, stdev=433.42
clat percentiles (usec):
| 1.00th=[ 78], 5.00th=[ 85], 10.00th=[ 100], 20.00th=[ 129],
| 30.00th=[ 149], 40.00th=[ 174], 50.00th=[ 202], 60.00th=[ 233],
| 70.00th=[ 265], 80.00th=[ 306], 90.00th=[ 359], 95.00th=[ 408],
| 99.00th=[ 510], 99.50th=[ 1090], 99.90th=[ 1549], 99.95th=[ 1614],
| 99.99th=[ 5080]
bw ( KiB/s): min= 9152, max=15329, per=49.97%, avg=13484.37, stdev=892.91, samples=238
iops : min= 572, max= 958, avg=842.76, stdev=55.80, samples=238
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=3.06%
lat (usec) : 250=22.22%, 500=72.91%, 750=0.76%, 1000=0.05%
lat (msec) : 2=0.96%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.64%, sys=15.91%, ctx=346233, majf=0, minf=266
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=235884,101198,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=61.4MiB/s (64.4MB/s), 61.4MiB/s-61.4MiB/s (64.4MB/s-64.4MB/s), io=3686MiB (3865MB), run=60001-60001msec
WRITE: bw=26.4MiB/s (27.6MB/s), 26.4MiB/s-26.4MiB/s (27.6MB/s-27.6MB/s), io=1581MiB (1658MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=274477/101294, merge=0/0, ticks=88860/19119, in_queue=107576, util=95.82%
real 1m0.690s
user 0m6.300s
sys 0m19.718s4.6SSD性能测试配置文件
配置文件内容如下：
[root@mysql-redis scripts]# cat /data/soft/ssd-test.fio
##2020-05-24
[global]
bs=4k
ioengine=libaio
iodepth=4
numjobs=1
group_reporting
size=5g
direct=1
runtime=60
directory=/data/soft
filename=ssd.test.file
#inflate-log=/data/soft/TestResult.log
#output=/data/soft/TestResult.log
[seq-read]
name=iops_seqread
rw=read
stonewall
[rand-read]
name=iops_randread
rw=randread
stonewall
[seq-write]
name=iops_seqwrite
rw=write
stonewall
[rand-write]
name=iops_randwrite
rw=randwrite
stonewall
[randrw]
name=iops_randrw
rwmixread=70
rw=randrw
stonewall下面是fio测试指定配置文件输出的内容：
[root@mysql-redis soft]# fio ssd-test.fio
iops_seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randread: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_seqwrite: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randwrite: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randrw: (g=4): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.7
Starting 5 processes
iops_seqread: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [_(4),m(1)][90.5%][r=47.8MiB/s,w=20.7MiB/s][r=12.2k,w=5308 IOPS][eta 00m:21s]
iops_seqread: (groupid=0, jobs=1): err= 0: pid=2142: Sun May 24 20:06:20 2020
read: IOPS=57.5k, BW=225MiB/s (236MB/s)(5120MiB/22782msec)
slat (usec): min=3, max=1943, avg= 5.13, stdev= 2.21
clat (usec): min=25, max=12467, avg=63.13, stdev=202.65
lat (usec): min=40, max=12472, avg=68.41, stdev=202.72
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 56],
| 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 57],
| 70.00th=[ 57], 80.00th=[ 58], 90.00th=[ 59], 95.00th=[ 62],
| 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 2073], 99.95th=[ 5080],
| 99.99th=[10814]
bw ( KiB/s): min=43579, max=193812, per=75.80%, avg=174446.82, stdev=47419.87, samples=45
iops : min=10894, max=48453, avg=43611.36, stdev=11855.06, samples=45
lat (usec) : 50=3.39%, 100=96.25%, 250=0.08%, 500=0.07%, 750=0.04%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.06%, 4=0.04%, 10=0.05%, 20=0.01%
cpu : usr=19.83%, sys=39.29%, ctx=1222141, majf=0, minf=114
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_randread: (groupid=1, jobs=1): err= 0: pid=2170: Sun May 24 20:06:20 2020
read: IOPS=21.7k, BW=84.9MiB/s (89.0MB/s)(5094MiB/60001msec)
slat (usec): min=3, max=1068, avg=12.62, stdev= 6.14
clat (usec): min=42, max=38568, avg=167.76, stdev=315.76
lat (usec): min=62, max=38588, avg=180.75, stdev=316.07
clat percentiles (usec):
| 1.00th=[ 130], 5.00th=[ 135], 10.00th=[ 137], 20.00th=[ 141],
| 30.00th=[ 143], 40.00th=[ 147], 50.00th=[ 151], 60.00th=[ 155],
| 70.00th=[ 161], 80.00th=[ 172], 90.00th=[ 192], 95.00th=[ 206],
| 99.00th=[ 251], 99.50th=[ 273], 99.90th=[ 3982], 99.95th=[ 7308],
| 99.99th=[14091]
bw ( KiB/s): min= 5397, max=74196, per=79.00%, avg=68682.74, stdev=15713.88, samples=119
iops : min= 1349, max=18549, avg=17170.32, stdev=3928.48, samples=119
lat (usec) : 50=0.01%, 100=0.02%, 250=98.94%, 500=0.82%, 750=0.02%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.03%, 4=0.04%, 10=0.07%, 20=0.02%, 50=0.01%
cpu : usr=11.52%, sys=37.46%, ctx=668232, majf=0, minf=319
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1304064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_seqwrite: (groupid=2, jobs=1): err= 0: pid=2231: Sun May 24 20:06:20 2020
write: IOPS=47.1k, BW=184MiB/s (193MB/s)(5120MiB/27848msec)
slat (usec): min=4, max=139, avg= 7.50, stdev= 2.96
clat (usec): min=27, max=15374, avg=76.14, stdev=278.53
lat (usec): min=49, max=15381, avg=83.78, stdev=279.26
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 59], 10.00th=[ 60], 20.00th=[ 61],
| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 67],
| 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 73], 95.00th=[ 74],
| 99.00th=[ 81], 99.50th=[ 89], 99.90th=[ 3326], 99.95th=[ 3392],
| 99.99th=[15139]
bw ( KiB/s): min= 3769, max=175262, per=80.83%, avg=152166.87, stdev=43009.10, samples=55
iops : min= 942, max=43815, avg=38041.35, stdev=10752.22, samples=55
lat (usec) : 50=0.24%, 100=99.43%, 250=0.07%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.21%, 10=0.01%, 20=0.03%
cpu : usr=11.79%, sys=43.48%, ctx=652945, majf=0, minf=95
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_randwrite: (groupid=3, jobs=1): err= 0: pid=2262: Sun May 24 20:06:20 2020
write: IOPS=47.2k, BW=185MiB/s (194MB/s)(5120MiB/27743msec)
slat (usec): min=4, max=415, avg= 7.49, stdev= 2.82
clat (usec): min=26, max=21032, avg=75.63, stdev=276.91
lat (usec): min=48, max=21058, avg=83.27, stdev=277.60
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 62],
| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 67],
| 70.00th=[ 68], 80.00th=[ 70], 90.00th=[ 73], 95.00th=[ 74],
| 99.00th=[ 80], 99.50th=[ 86], 99.90th=[ 3326], 99.95th=[ 3392],
| 99.99th=[15139]
bw ( KiB/s): min= 3175, max=142274, per=68.21%, avg=128901.20, stdev=33671.41, samples=55
iops : min= 793, max=35568, avg=32224.95, stdev=8417.81, samples=55
lat (usec) : 50=0.21%, 100=99.50%, 250=0.04%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.20%, 10=0.01%, 20=0.02%, 50=0.01%
cpu : usr=12.97%, sys=43.05%, ctx=654916, majf=0, minf=258
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
iops_randrw: (groupid=4, jobs=1): err= 0: pid=2292: Sun May 24 20:06:20 2020
read: IOPS=11.3k, BW=44.3MiB/s (46.5MB/s)(2659MiB/60001msec)
slat (usec): min=4, max=594, avg=13.08, stdev= 6.88
clat (usec): min=61, max=45073, avg=244.45, stdev=441.39
lat (usec): min=134, max=45115, avg=257.84, stdev=441.61
clat percentiles (usec):
| 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 176],
| 30.00th=[ 190], 40.00th=[ 202], 50.00th=[ 212], 60.00th=[ 221],
| 70.00th=[ 233], 80.00th=[ 258], 90.00th=[ 330], 95.00th=[ 359],
| 99.00th=[ 474], 99.50th=[ 1270], 99.90th=[ 6521], 99.95th=[10028],
| 99.99th=[17695]
bw ( KiB/s): min= 3072, max=51824, per=99.92%, avg=45336.55, stdev=11250.66, samples=119
iops : min= 768, max=12958, avg=11334.12, stdev=2812.66, samples=119
write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(1139MiB/60001msec)
slat (usec): min=6, max=874, avg=16.94, stdev= 7.42
clat (usec): min=40, max=31265, avg=192.80, stdev=413.19
lat (usec): min=56, max=31306, avg=210.09, stdev=413.32
clat percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 78],
| 30.00th=[ 145], 40.00th=[ 169], 50.00th=[ 188], 60.00th=[ 202],
| 70.00th=[ 212], 80.00th=[ 227], 90.00th=[ 251], 95.00th=[ 277],
| 99.00th=[ 619], 99.50th=[ 1205], 99.90th=[ 5997], 99.95th=[ 9634],
| 99.99th=[16581]
bw ( KiB/s): min= 1184, max=22176, per=99.94%, avg=19428.64, stdev=4847.41, samples=119
iops : min= 296, max= 5544, avg=4857.13, stdev=1211.85, samples=119
lat (usec) : 50=0.12%, 100=7.22%, 250=74.36%, 500=17.22%, 750=0.24%
lat (usec) : 1000=0.10%
lat (msec) : 2=0.52%, 4=0.06%, 10=0.11%, 20=0.04%, 50=0.01%
cpu : usr=7.88%, sys=29.46%, ctx=340101, majf=0, minf=374
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=680630,291634,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=5120MiB (5369MB), run=22782-22782msec
Run status group 1 (all jobs):
READ: bw=84.9MiB/s (89.0MB/s), 84.9MiB/s-84.9MiB/s (89.0MB/s-89.0MB/s), io=5094MiB (5341MB), run=60001-60001msec
Run status group 2 (all jobs):
WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5120MiB (5369MB), run=27848-27848msec
Run status group 3 (all jobs):
WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5120MiB (5369MB), run=27743-27743msec
Run status group 4 (all jobs):
READ: bw=44.3MiB/s (46.5MB/s), 44.3MiB/s-44.3MiB/s (46.5MB/s-46.5MB/s), io=2659MiB (2788MB), run=60001-60001msec
WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=1139MiB (1195MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=3294785/2912828, merge=0/3, ticks=430527/238028, in_queue=666582, util=99.09%
[root@mysql-redis soft]# 五、总结
关于fio测试硬盘的建议：
一、建议使用顺序I/O和较大的blocksize 来测试设备的通吐量和延迟
二、建议使用随机I/O和较小的blocksize来测试设备的IOPS和延迟
三、在配置numjobs和iodepth 测试底层存储性能时，建议要深入了解应用到底采用的是同步的io还是异步的io（是多进程并发i/o请求还是一次提交一批的i/o的请求）
提示：对fio压测 磁盘性能感兴趣的话，可以参考下ezfio(直接在github.com上搜索ezfio)
重要提示：fio是 支持对裸设备直接进行I/O测试的，在测试裸设备时，一定要避开系统所在的裸设备，否则会导致系统奔溃。
例如：
[root@mysql-redis soft]# df -h / /boot
文件系统 容量 已用 可用 已用% 挂载点
/dev/sdb5 207G 7.5G 200G 4% /
/dev/sdb1 297M 120M 177M 41% /boot/dev/sdb /dev/sdb5 /dev/sdb1 这些设备是绝对禁止进行fio测试的，
同时在测试其他的设备裸设备时，一定要提前确定好此裸设备上是否有重要 的数据，一定要重要的设备提前做好备份。
以上就是我这边对fio测试硬盘的简单学习总结，仅供参考，如有网友直接复制应用造成损失，本博文不负担任何责任。
参考文档：
http://www.ssdfans.com/?p=6087
https://linux.die.net/man/1/fio
https://blog.csdn.net/don_chiang709/article/details/92628623