Repository navigation
perf: run one fio job per data disk in perf_premium_datadisks - #4847
SrikanthMyakam (SRIKKANTH) wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
An incompatible lsblk query breaks FreeBSD data-disk tests before fio starts, while integer division changes the requested concurrency and requires corresponding fixes to job distribution and region sizing.
2 open issues
What changed in this PR
This PR improves multi-disk fio performance by assigning jobs to individual disks instead of sharing every disk across all jobs.
Changes:
- Adds opt-in per-file fio jobs, offset increments, and total queue-depth reporting.
- Enables per-disk jobs in data-disk benchmarks and bounds job regions by partition capacity.
- Adds unit tests for command generation and queue-depth accounting.
| File | Description |
|---|---|
selftests/test_fio_tool.py |
Tests per-file commands, default behavior, and queue-depth reporting. |
lisa/tools/fio.py |
Adds per-file job sections and offset support. |
lisa/microsoft/testsuites/performance/common.py |
Distributes jobs across disks and limits per-job regions. |
Key Test Cases:
perf_premium_datadisks_io|perf_premium_datadisks_1024k|perf_premium_datadisks_4k_io_uring
Impacted LISA Features:
Disk
Tested Azure Marketplace Images:
- canonical ubuntu-24_04-lts server latest
Review effort: Balanced
perf: run one fio job per data disk in perf_premium_datadisks perf_premium_datadisks passed all data disk partitions to every fio job (colon-joined --filename). With that layout fio stalls writes before they reach the devices: on Standard_D128ds_v6 with 64 Premium SSD disks, random/sequential 4K writes plateaued at ~34-48k IOPS (17-23% of the 204,800 uncached IOPS limit) regardless of numjobs, CPU set, queue depth, working-set size or wbt, while only ~144 of 8,192 outstanding writes were in flight at the disks. With one job per disk (same total jobs and outstanding I/O) the same VM reached 204k read and 192k write IOPS (99.8% / 93.6% of the limit). - Fio: add opt-in per_file_jobs mode that emits global options followed by one --name/--filename section per file, plus offset_increment support. qdepth stays the total outstanding I/O (iodepth x total jobs). - perf_disk: add per_disk_jobs to split numjob across disks and give each job its own region. - perf_premium_datadisks: enable per_disk_jobs and cap the per-job region by the smallest partition so jobs on a disk do not overlap. - Add selftests for the per-file command, the default command and qdepth accounting. Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
32cdf18 to
889ec8e
Compare
Preserve total fio concurrency with balanced per-file jobs and size regions for the busiest disk. Skip workloads with fewer jobs than disks before partitioning. Query byte-accurate disk sizes through Linux and FreeBSD tool implementations. Remove fio selftests as requested by review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AI Test Case SelectionSelected 65 test case(s): perf_nested_kvm_storage_singledisk,perf_nested_kvm_storage_multidisk,perf_nested_hyperv_storage_singledisk,perf_nested_hyperv_storage_multidisk,perf_nested_kvm_ntttcp_private_bridge,perf_nested_kvm_ntttcp_different_l1_nat,perf_nested_hyperv_ntttcp_different_l1_nat,perf_nested_kvm_netperf_pps_nat,perf_tcp_latency_synthetic,perf_tcp_latency_sriov,perf_tcp_single_pps_synthetic,perf_tcp_single_pps_sriov,perf_tcp_max_pps_synthetic,perf_tcp_max_pps_sriov,perf_tcp_ntttcp_128_connections_synthetic,perf_tcp_ntttcp_synthetic,perf_tcp_ntttcp_sriov,perf_udp_1k_ntttcp_synthetic,perf_udp_1k_ntttcp_sriov,perf_tcp_iperf_synthetic,perf_tcp_iperf_sriov,perf_udp_iperf_synthetic,perf_udp_iperf_sriov,perf_sockperf_latency_tcp_sriov,perf_sockperf_latency_udp_sriov,perf_sockperf_latency_udp_synthetic,perf_sockperf_latency_tcp_synthetic,perf_sockperf_latency_tcp_sriov_busy_poll,perf_sockperf_latency_udp_sriov_busy_poll,perf_sockperf_latency_udp_synthetic_busy_poll,perf_sockperf_latency_tcp_synthetic_busy_poll,perf_tcp_iperf_passthrough_host_guest,perf_udp_iperf_passthrough_host_guest,perf_tcp_single_pps_passthrough_host_guest,perf_tcp_max_pps_passthrough_host_guest,perf_tcp_ntttcp_passthrough_host_guest,perf_udp_1k_ntttcp_passthrough_host_guest,perf_tcp_iperf_passthrough_two_guest,perf_udp_iperf_passthrough_two_guest,perf_tcp_single_pps_passthrough_two_guest,perf_tcp_max_pps_passthrough_two_guest,perf_tcp_ntttcp_passthrough_two_guest,perf_udp_1k_ntttcp_passthrough_two_guest,perf_nvme,perf_nvme_io_uring,perf_messaging,perf_epoll,perf_ultra_datadisks_4k,perf_ultra_datadisks_1024k,perf_premiumv2_datadisks_4k,perf_premiumv2_datadisks_1024k,perf_premium_datadisks_4k,perf_premium_datadisks_1024k,perf_premium_datadisks_4k_io_uring,perf_premium_datadisks_1024k_io_uring,perf_resource_disk_1024k,perf_resource_disk_4k,perf_standardssd_datadisks_4k,perf_standardssd_datadisks_1024k,perf_premium_datadisks_io,perf_storage_over_nfs_sriov_tcp_4k,perf_storage_over_nfs_sriov_udp_4k,perf_storage_over_nfs_synthetic_tcp_4k,perf_storage_over_nfs_synthetic_udp_4k,perf_storage_generic_fio_test Marketplace image: Result: Canceled |
perf: run one fio job per data disk in perf_premium_datadisks
The current fio tool struggles to push full IOPS on large VM sizes/ VM sizes with high number of remote storage IOPS.
RCA:
perf_premium_datadisks passed all data disk partitions to every fio job (colon-joined --filename). With that layout fio stalls writes before they reach the devices: on Standard_D128ds_v6 with 64 Premium SSD disks, random/sequential 4K writes plateaued at ~34-48k IOPS (17-23% of the 204,800 uncached IOPS limit) regardless of numjobs, CPU set, queue depth, working-set size or wbt, while only ~144 of 8,192 outstanding writes were in flight at the disks.
With one job per disk (same total jobs and outstanding I/O) the same VM reached 204k read and 192k write IOPS (99.8% / 93.6% of the limit).
Description
Related Issue
Type of Change
Checklist
Test Validation
Key Test Cases:
Impacted LISA Features:
Tested Azure Marketplace Images:
Test Results