Foreman Maintain

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1622616

Description of problem:

Inconsistency over fio utility benchmarking

Version-Release number of selected component (if applicable):

satellite-installer-6.4.0.7-1.beta.el7sat.noarch
fio-3.1-2.el7.x86_64

How reproducible:

Steps to Reproduce:

1. # foreman-maintain upgrade check --target-version 6.4

Actual results:

pre-upgrade-step fails because we are using a VM:

- Check for recommended disk speed of pulp, mongodb, pgsql dir.:
- Finished

Disk speed : 24 MB/sec [FAIL]
Slow disk detected /var/lib/pulp mounted on /dev/mapper/vg_app1-lv_pulp.
Actual disk speed: 24 MB/sec
Expected disk speed: 80 MB/sec.

Expected results:

- Fio test looks extremely inconsistent while calculating Disk Speed, please review additional information for detailed analysis.

Additional info:

Fio results:

-----------
root@server1:~# sudo fio --name=job1 --rw=read --size=1g --directory=/var/lib/pulp --direct=1

Run status group 0 (all jobs):
READ: bw=24.0MiB/s (26.2MB/s), 24.0MiB/s-24.0MiB/s (26.2MB/s-26.2MB/s), io=1024MiB (1074MB), run=40996-40996msec
-----------

fio test on an other server that has both local disk and a SAN disk. The SAN disk reports the same read performance.

root@server2 ~$ sudo fio --name=job1 --rw=read --size=1g --direct=1 --directory=/hana/log

Run status group 0 (all jobs):
READ: bw=30.8MiB/s (32.3MB/s), 30.8MiB/s-30.8MiB/s (32.3MB/s-32.3MB/s), io=1024MiB (1074MB), run=33264-33264msec

root@server2 ~$ sudo fio --name=job1 --rw=read --size=1g --direct=1 --directory=/var/tmp

Run status group 0 (all jobs):
READ: bw=92.0MiB/s (96.5MB/s), 92.0MiB/s-92.0MiB/s (96.5MB/s-96.5MB/s), io=1024MiB (1074MB), run=11129-11129msec

Executing on the same hardware setup with different fio parameters gives a difference view

---

Also testing different block sizes on the VM gives already different results:

root@server1:~# sudo fio --name=job1 --rw=read --bs=4k --size=1g --directory=/var/lib/pulp --direct=1

Run status group 0 (all jobs):
READ: bw=24.0MiB/s (26.2MB/s), 24.0MiB/s-24.0MiB/s (26.2MB/s-26.2MB/s), io=1024MiB (1074MB), run=41011-41011msec

-----------
root@server1:~# sudo fio --name=job1 --rw=read --bs=8k --size=1g --directory=/var/lib/pulp --direct=1

Run status group 0 (all jobs):
READ: bw=46.6MiB/s (48.9MB/s), 46.6MiB/s-46.6MiB/s (48.9MB/s-48.9MB/s), io=1024MiB (1074MB), run=21954-21954msec

-----------
root@server1:~# sudo fio --name=job1 --rw=read --bs=16k --size=1g --directory=/var/lib/pulp --direct=1

Run status group 0 (all jobs):
READ: bw=84.3MiB/s (88.4MB/s), 84.3MiB/s-84.3MiB/s (88.4MB/s-88.4MB/s), io=1024MiB (1074MB), run=12141-12141msec

---

Review the currently IO tests needs clarification on exact :

E.g. bandwidth of latency and different load blocksizes and if sequential or random/

The difference in physical hardware were in the current test mode, the local physcial-HDD was outpefroming the enterprise class AllFlash SAN.

Putting in a bit more randomness and more jibs shows that the SAN beats the local physical-HDD by a factor 8x.

---

root@server2 ~$ sudo fio --name=randread --rw=randread --direct=1 --bs=8k --numjobs=16 --size=1G --runtime=30 --group_reporting --directory=/var/tmp

Run status group 0 (all jobs):
READ: bw=40.4MiB/s (42.4MB/s), 40.4MiB/s-40.4MiB/s (42.4MB/s-42.4MB/s), io=1215MiB (1274MB), run=30048-30048msec

-----------
root@server2 ~$ sudo fio --name=randread --rw=randread --direct=1 --bs=8k --numjobs=16 --size=1G --runtime=30 --group_reporting --directory=/hana/log

Run status group 0 (all jobs):
READ: bw=291MiB/s (306MB/s), 291MiB/s-291MiB/s (306MB/s-306MB/s), io=8745MiB (9169MB), run=30002-30002msec
-----------

Based on the above results it looks like, fio test can not be trusted for real workload simulation.