fio --name=test_write --blocksize=4k --direct=1 --buffered=0 --ioengine=libaio --loops=1000 --runtime=300 --iodepth=256 --rw=randwrite --filename=/dev/sdd
fio --name=test_read --blocksize=4k --direct=1 --buffered=0 --ioengine=libaio --loops=1000 --runtime=300 --iodepth=256 --rw=randread --filename=/dev/sdd
RAID 0 - 2 диска
Random write
test_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=4444KiB/s][w=1111 IOPS][eta 00m:00s]
test_write: (groupid=0, jobs=1): err= 0: pid=1373: Mon Apr 5 19:29:28 2021
write: IOPS=1265, BW=5061KiB/s (5182kB/s)(1484MiB/300300msec); 0 zone resets
slat (nsec): min=2747, max=91707, avg=7673.84, stdev=4380.03
clat (usec): min=1494, max=894298, avg=202335.90, stdev=169010.64
lat (usec): min=1497, max=894311, avg=202343.77, stdev=169011.88
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 157], 50.00th=[ 220], 60.00th=[ 264],
| 70.00th=[ 305], 80.00th=[ 347], 90.00th=[ 418], 95.00th=[ 485],
| 99.00th=[ 609], 99.50th=[ 659], 99.90th=[ 760], 99.95th=[ 802],
| 99.99th=[ 852]
bw ( KiB/s): min= 808, max=276616, per=100.00%, avg=5061.27, stdev=19722.93, samples=600
iops : min= 202, max=69154, avg=1265.30, stdev=4930.69, samples=600
lat (msec) : 2=0.04%, 4=13.23%, 10=18.81%, 20=0.10%, 50=0.39%
lat (msec) : 100=2.27%, 250=21.75%, 500=39.27%, 750=4.02%, 1000=0.12%
cpu : usr=0.65%, sys=1.11%, ctx=215677, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,379924,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=5061KiB/s (5182kB/s), 5061KiB/s-5061KiB/s (5182kB/s-5182kB/s), io=1484MiB (1556MB), run=300300-300300msec
Disk stats (read/write):
sdd: ios=51/379713, merge=0/0, ticks=40/76788750, in_queue=76818560, util=100.00%
Random read
test_read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][0.0%][eta 1825d:16h:56m:03s]
test_read: (groupid=0, jobs=1): err= 0: pid=1477: Mon Apr 5 19:34:36 2021
read: IOPS=695, BW=2780KiB/s (2847kB/s)(822MiB/302654msec)
slat (usec): min=2, max=64145, avg=118.03, stdev=1190.41
clat (usec): min=22, max=3751.8k, avg=368158.17, stdev=229672.84
lat (usec): min=31, max=3751.8k, avg=368276.73, stdev=229680.31
clat percentiles (usec):
| 1.00th=[ 38], 5.00th=[ 162530], 10.00th=[ 202376],
| 20.00th=[ 233833], 30.00th=[ 256902], 40.00th=[ 283116],
| 50.00th=[ 312476], 60.00th=[ 350225], 70.00th=[ 400557],
| 80.00th=[ 471860], 90.00th=[ 608175], 95.00th=[ 759170],
| 99.00th=[1182794], 99.50th=[1417675], 99.90th=[2734687],
| 99.95th=[2868904], 99.99th=[3137340]
bw ( KiB/s): min= 2224, max=15416, per=100.00%, avg=2801.29, stdev=535.08, samples=600
iops : min= 556, max= 3854, avg=700.30, stdev=133.77, samples=600
lat (usec) : 50=3.42%, 100=0.27%, 250=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.02%, 50=0.01%
lat (msec) : 100=0.02%, 250=23.36%, 500=55.57%, 750=12.07%, 1000=3.29%
cpu : usr=1.04%, sys=2.45%, ctx=190713, majf=0, minf=265
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=210371,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=2780KiB/s (2847kB/s), 2780KiB/s-2780KiB/s (2847kB/s-2847kB/s), io=822MiB (862MB), run=302654-302654msec
Disk stats (read/write):
sdd: ios=210410/0, merge=0/0, ticks=76934763/0, in_queue=76956280, util=100.00%
RAID 10 - 4 диска
Random write
test_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=6346KiB/s][w=1586 IOPS][eta 00m:00s]
test_write: (groupid=0, jobs=1): err= 0: pid=1192: Mon Apr 5 19:49:45 2021
write: IOPS=1997, BW=7990KiB/s (8182kB/s)(2342MiB/300194msec); 0 zone resets
slat (usec): min=2, max=5661, avg= 7.99, stdev= 8.39
clat (usec): min=1482, max=477844, avg=128143.60, stdev=81267.20
lat (usec): min=1496, max=477852, avg=128151.78, stdev=81268.15
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 96], 40.00th=[ 125], 50.00th=[ 142], 60.00th=[ 157],
| 70.00th=[ 174], 80.00th=[ 190], 90.00th=[ 222], 95.00th=[ 251],
| 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 409], 99.95th=[ 439],
| 99.99th=[ 447]
bw ( KiB/s): min= 2840, max=265984, per=100.00%, avg=7991.55, stdev=19325.62, samples=600
iops : min= 710, max=66496, avg=1997.87, stdev=4831.41, samples=600
lat (msec) : 2=0.03%, 4=7.07%, 10=13.49%, 20=0.23%, 50=1.41%
lat (msec) : 100=8.96%, 250=63.81%, 500=5.00%
cpu : usr=1.05%, sys=2.00%, ctx=413082, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,599662,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=7990KiB/s (8182kB/s), 7990KiB/s-7990KiB/s (8182kB/s-8182kB/s), io=2342MiB (2456MB), run=300194-300194msec
Disk stats (read/write):
sdd: ios=51/599538, merge=0/0, ticks=5/76796212, in_queue=76810976, util=100.00%
Random read
test_read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=7368KiB/s][r=1842 IOPS][eta 00m:00s]
test_read: (groupid=0, jobs=1): err= 0: pid=1299: Mon Apr 5 19:54:51 2021
read: IOPS=1376, BW=5505KiB/s (5637kB/s)(1614MiB/300192msec)
slat (usec): min=2, max=95445, avg=84.76, stdev=1188.69
clat (usec): min=21, max=2583.3k, avg=185910.01, stdev=225305.68
lat (usec): min=30, max=2583.3k, avg=185995.32, stdev=225370.59
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 10290], 10.00th=[ 20317],
| 20.00th=[ 39584], 30.00th=[ 60556], 40.00th=[ 84411],
| 50.00th=[ 112722], 60.00th=[ 149947], 70.00th=[ 200279],
| 80.00th=[ 283116], 90.00th=[ 434111], 95.00th=[ 599786],
| 99.00th=[1098908], 99.50th=[1367344], 99.90th=[2021655],
| 99.95th=[2332034], 99.99th=[2499806]
bw ( KiB/s): min= 1480, max=21504, per=100.00%, avg=5504.83, stdev=2366.39, samples=600
iops : min= 370, max= 5376, avg=1376.20, stdev=591.60, samples=600
lat (usec) : 50=1.62%, 100=0.16%, 250=0.01%
lat (msec) : 2=0.01%, 4=0.13%, 10=2.90%, 20=5.07%, 50=15.18%
lat (msec) : 100=20.76%, 250=30.74%, 500=15.90%, 750=4.62%, 1000=1.59%
cpu : usr=1.93%, sys=4.12%, ctx=356979, majf=0, minf=266
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=413160,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=5505KiB/s (5637kB/s), 5505KiB/s-5505KiB/s (5637kB/s-5637kB/s), io=1614MiB (1692MB), run=300192-300192msec
Disk stats (read/write):
sdd: ios=413092/0, merge=0/0, ticks=76699528/0, in_queue=76723088, util=100.00%
RAID 10 - 6 диска
Random write
test_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=9096KiB/s][w=2274 IOPS][eta 00m:00s]
test_write: (groupid=0, jobs=1): err= 0: pid=1165: Mon Apr 5 20:07:44 2021
write: IOPS=2632, BW=10.3MiB/s (10.8MB/s)(3087MiB/300174msec); 0 zone resets
slat (usec): min=2, max=41109, avg= 8.44, stdev=58.97
clat (usec): min=1635, max=366375, avg=97233.33, stdev=53529.19
lat (usec): min=1640, max=366381, avg=97241.99, stdev=53530.13
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 53],
| 30.00th=[ 81], 40.00th=[ 96], 50.00th=[ 107], 60.00th=[ 115],
| 70.00th=[ 126], 80.00th=[ 138], 90.00th=[ 159], 95.00th=[ 178],
| 99.00th=[ 220], 99.50th=[ 234], 99.90th=[ 266], 99.95th=[ 275],
| 99.99th=[ 334]
bw ( KiB/s): min= 5184, max=265120, per=100.00%, avg=10532.46, stdev=18719.19, samples=600
iops : min= 1296, max=66280, avg=2633.11, stdev=4679.80, samples=600
lat (msec) : 2=0.03%, 4=4.89%, 10=10.89%, 20=0.34%, 50=3.31%
lat (msec) : 100=24.55%, 250=55.74%, 500=0.25%
cpu : usr=1.52%, sys=2.79%, ctx=589423, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,790202,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=10.3MiB/s (10.8MB/s), 10.3MiB/s-10.3MiB/s (10.8MB/s-10.8MB/s), io=3087MiB (3237MB), run=300174-300174msec
Disk stats (read/write):
sdd: ios=51/790005, merge=0/0, ticks=6/76793907, in_queue=76809220, util=100.00%
Random read
test_read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=10.1MiB/s][r=2579 IOPS][eta 00m:00s]
test_read: (groupid=0, jobs=1): err= 0: pid=1271: Mon Apr 5 20:12:48 2021
read: IOPS=1988, BW=7955KiB/s (8146kB/s)(2332MiB/300176msec)
slat (usec): min=2, max=405210, avg=73.43, stdev=2141.88
clat (usec): min=20, max=2686.4k, avg=128637.50, stdev=211237.65
lat (usec): min=32, max=2858.4k, avg=128711.36, stdev=211412.80
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 6390], 10.00th=[ 9634],
| 20.00th=[ 17695], 30.00th=[ 28443], 40.00th=[ 42206],
| 50.00th=[ 60556], 60.00th=[ 84411], 70.00th=[ 121111],
| 80.00th=[ 185598], 90.00th=[ 308282], 95.00th=[ 450888],
| 99.00th=[1010828], 99.50th=[1451230], 99.90th=[2432697],
| 99.95th=[2466251], 99.99th=[2499806]
bw ( KiB/s): min= 232, max=20848, per=100.00%, avg=7956.28, stdev=3303.60, samples=600
iops : min= 58, max= 5212, avg=1989.06, stdev=825.90, samples=600
lat (usec) : 50=1.09%, 100=0.09%, 250=0.01%
lat (msec) : 2=0.02%, 4=0.75%, 10=8.57%, 20=11.87%, 50=22.20%
lat (msec) : 100=20.26%, 250=21.19%, 500=9.89%, 750=2.26%, 1000=0.78%
cpu : usr=2.60%, sys=5.35%, ctx=502930, majf=0, minf=266
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=596987,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=7955KiB/s (8146kB/s), 7955KiB/s-7955KiB/s (8146kB/s-8146kB/s), io=2332MiB (2445MB), run=300176-300176msec
Disk stats (read/write):
sdd: ios=597013/0, merge=0/0, ticks=76681940/0, in_queue=76684352, util=100.00%
RAID 10 - 8 диска
Random write
test_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=11.8MiB/s][w=3009 IOPS][eta 00m:00s]
test_write: (groupid=0, jobs=1): err= 0: pid=1207: Mon Apr 5 20:34:05 2021
write: IOPS=3294, BW=12.9MiB/s (13.5MB/s)(3862MiB/300140msec); 0 zone resets
slat (usec): min=2, max=22164, avg= 7.65, stdev=23.81
clat (usec): min=1539, max=284418, avg=77697.47, stdev=39317.72
lat (usec): min=1544, max=284425, avg=77705.33, stdev=39318.27
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 49],
| 30.00th=[ 66], 40.00th=[ 77], 50.00th=[ 84], 60.00th=[ 90],
| 70.00th=[ 97], 80.00th=[ 107], 90.00th=[ 123], 95.00th=[ 138],
| 99.00th=[ 169], 99.50th=[ 184], 99.90th=[ 213], 99.95th=[ 224],
| 99.99th=[ 234]
bw ( KiB/s): min= 7688, max=256760, per=100.00%, avg=13179.08, stdev=18284.54, samples=600
iops : min= 1922, max=64190, avg=3294.76, stdev=4571.14, samples=600
lat (msec) : 2=0.02%, 4=2.08%, 10=10.72%, 20=0.70%, 50=7.03%
lat (msec) : 100=53.46%, 250=25.99%, 500=0.01%
cpu : usr=1.68%, sys=3.27%, ctx=790992, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,988753,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=12.9MiB/s (13.5MB/s), 12.9MiB/s-12.9MiB/s (13.5MB/s-13.5MB/s), io=3862MiB (4050MB), run=300140-300140msec
Disk stats (read/write):
sdd: ios=51/988627, merge=0/0, ticks=6/76794574, in_queue=76813300, util=100.00%
Random read
test_read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=12.5MiB/s][r=3190 IOPS][eta 00m:00s]
test_read: (groupid=0, jobs=1): err= 0: pid=1315: Mon Apr 5 20:39:13 2021
read: IOPS=2686, BW=10.5MiB/s (11.0MB/s)(3151MiB/300166msec)
slat (usec): min=2, max=401131, avg=56.34, stdev=1656.28
clat (usec): min=22, max=2603.8k, avg=95209.07, stdev=162089.53
lat (usec): min=32, max=2681.2k, avg=95265.78, stdev=162260.29
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 16],
| 30.00th=[ 25], 40.00th=[ 35], 50.00th=[ 48], 60.00th=[ 65],
| 70.00th=[ 90], 80.00th=[ 132], 90.00th=[ 222], 95.00th=[ 326],
| 99.00th=[ 693], 99.50th=[ 1028], 99.90th=[ 2265], 99.95th=[ 2433],
| 99.99th=[ 2467]
bw ( KiB/s): min= 192, max=24568, per=100.00%, avg=10749.53, stdev=3799.85, samples=600
iops : min= 48, max= 6142, avg=2687.36, stdev=949.95, samples=600
lat (usec) : 50=0.78%, 100=0.08%, 250=0.01%
lat (msec) : 2=0.01%, 4=0.61%, 10=9.44%, 20=14.13%, 50=26.40%
lat (msec) : 100=21.74%, 250=18.60%, 500=6.23%, 750=1.11%, 1000=0.34%
cpu : usr=3.22%, sys=6.45%, ctx=663778, majf=0, minf=267
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=806533,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s), io=3151MiB (3304MB), run=300166-300166msec
Disk stats (read/write):
sdd: ios=806420/0, merge=0/0, ticks=76655708/0, in_queue=76680652, util=100.00%
Sofware RAID 10 - 8 disks
Тот же контроллер, те же диски, идентичные тесты. Диски в контроллере распределены по одному в логические диски в RAID 0.
mdadm --create /dev/md0 --level=10 --raid-devices=8 /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk
Дождался пока отребилдится (2.5 часа).
Random write
test_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=10.7MiB/s][w=2726 IOPS][eta 00m:00s]
test_write: (groupid=0, jobs=1): err= 0: pid=28273: Tue Apr 6 17:46:25 2021
write: IOPS=2905, BW=11.4MiB/s (11.9MB/s)(3406MiB/300075msec); 0 zone resets
slat (usec): min=5, max=165309, avg=64.75, stdev=1520.62
clat (usec): min=99, max=984911, avg=88019.42, stdev=64149.80
lat (usec): min=115, max=984929, avg=88084.65, stdev=64147.22
clat percentiles (usec):
| 1.00th=[ 783], 5.00th=[ 6915], 10.00th=[ 13960], 20.00th=[ 25822],
| 30.00th=[ 43779], 40.00th=[ 62653], 50.00th=[ 80217], 60.00th=[ 95945],
| 70.00th=[114820], 80.00th=[141558], 90.00th=[175113], 95.00th=[204473],
| 99.00th=[270533], 99.50th=[299893], 99.90th=[362808], 99.95th=[408945],
| 99.99th=[549454]
bw ( KiB/s): min= 7233, max=79464, per=99.97%, avg=11620.05, stdev=5787.96, samples=600
iops : min= 1808, max=19866, avg=2904.95, stdev=1447.00, samples=600
lat (usec) : 100=0.01%, 250=0.12%, 500=0.44%, 750=0.40%, 1000=0.33%
lat (msec) : 2=0.97%, 4=1.16%, 10=2.91%, 20=9.79%, 50=17.08%
lat (msec) : 100=28.93%, 250=36.17%, 500=1.70%, 750=0.02%, 1000=0.01%
cpu : usr=1.97%, sys=6.42%, ctx=587321, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,872013,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=11.4MiB/s (11.9MB/s), 11.4MiB/s-11.4MiB/s (11.9MB/s-11.9MB/s), io=3406MiB (3572MB), run=300075-300075msec
Disk stats (read/write):
md0: ios=48/871528, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=6/244791, aggrmerge=0/0, aggrticks=10/14296073, aggrin_queue=13771795, aggrutil=98.29%
sdf: ios=1/244498, merge=0/0, ticks=22/16728408, in_queue=15952712, util=98.02%
sdd: ios=20/244438, merge=0/0, ticks=5/13289034, in_queue=12874136, util=97.81%
sdk: ios=2/244934, merge=0/0, ticks=7/10311635, in_queue=10197032, util=97.43%
sdi: ios=0/245295, merge=0/0, ticks=0/13093438, in_queue=12912240, util=97.97%
sdg: ios=0/244498, merge=0/0, ticks=0/13894552, in_queue=13531028, util=97.73%
sde: ios=1/244438, merge=0/0, ticks=6/10837612, in_queue=10736384, util=97.18%
sdj: ios=25/244934, merge=0/0, ticks=39/16292011, in_queue=15303284, util=98.15%
sdh: ios=1/245295, merge=0/0, ticks=5/19921900, in_queue=18667544, util=98.29%
Random read
test_read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=12.6MiB/s][r=3238 IOPS][eta 00m:00s]
test_read: (groupid=0, jobs=1): err= 0: pid=28353: Tue Apr 6 17:51:33 2021
read: IOPS=2811, BW=10.0MiB/s (11.5MB/s)(3297MiB/300193msec)
slat (usec): min=5, max=117, avg=23.73, stdev=12.17
clat (usec): min=19, max=3079.4k, avg=91014.77, stdev=153271.21
lat (usec): min=37, max=3079.4k, avg=91038.95, stdev=153271.59
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 13],
| 30.00th=[ 19], 40.00th=[ 28], 50.00th=[ 40], 60.00th=[ 58],
| 70.00th=[ 84], 80.00th=[ 129], 90.00th=[ 224], 95.00th=[ 347],
| 99.00th=[ 709], 99.50th=[ 902], 99.90th=[ 1770], 99.95th=[ 2198],
| 99.99th=[ 2500]
bw ( KiB/s): min= 240, max=15072, per=100.00%, avg=11249.52, stdev=2536.47, samples=600
iops : min= 60, max= 3768, avg=2812.35, stdev=634.11, samples=600
lat (usec) : 20=0.01%, 50=0.41%, 100=0.08%, 250=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=1.18%, 10=13.65%, 20=16.75%, 50=24.16%
lat (msec) : 100=18.10%, 250=17.17%, 500=6.06%, 750=1.56%, 1000=0.47%
cpu : usr=3.62%, sys=9.65%, ctx=679553, majf=0, minf=266
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=844053,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=10.0MiB/s (11.5MB/s), 10.0MiB/s-10.0MiB/s (11.5MB/s-11.5MB/s), io=3297MiB (3457MB), run=300193-300193msec
Disk stats (read/write):
md0: ios=844101/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=105512/1, aggrmerge=0/0, aggrticks=9601937/0, aggrin_queue=9332516, aggrutil=100.00%
sdf: ios=105585/1, merge=0/0, ticks=7433407/0, in_queue=7433068, util=99.44%
sdd: ios=105559/1, merge=0/0, ticks=7499274/0, in_queue=7497784, util=99.64%
sdk: ios=106131/1, merge=0/0, ticks=13735893/0, in_queue=13708692, util=100.00%
sdi: ios=106099/1, merge=0/0, ticks=19380582/0, in_queue=17255912, util=99.98%
sdg: ios=105294/1, merge=0/0, ticks=7138768/0, in_queue=7137960, util=99.79%
sde: ios=105127/1, merge=0/0, ticks=7820366/0, in_queue=7820172, util=99.59%
sdj: ios=105049/1, merge=0/0, ticks=7447432/0, in_queue=7446616, util=99.92%
sdh: ios=105259/1, merge=0/0, ticks=6359775/0, in_queue=6359924, util=99.68%
Странно. Ожидал в софтверном рейде увидеть большую производительность. А увидел меньшую. Причем top не показывал сколько-нибудь значимую загрузку по IO. Есть сомнения в корректности теста.
