You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add Sriov.network_servicing_validation to exercise one accelerated-to-synthetic-to-accelerated networking transition between two Linux VMs under continuous, paced UDP and TCP traffic.
Require exactly one Azure NIC per VM. Check the Azure NIC inventory on every VM before changing any firewall or Accelerated Networking setting; skip multi-NIC environments with an explicit reason. The synthetic guest interface and its paired VF belong to the same Azure NIC.
Establish a two-minute baseline, observe each confirmed synthetic/restored path for at least 30 seconds, and let both 1 Mbit/sec streams complete their 1,140-second duration.
Require zero receiver-reported UDP loss, matching sent/received datagram counts, TCP completion and interval progress, unchanged internal addresses/routes, and no new detected kernel errors.
Submit AN changes without nested guest-state waits, poll VF state with bounded waits, restore AN before SSH-dependent cleanup, preserve the original test exception, and save traffic and kernel diagnostics.
Report offered-load throughput and recovery timing diagnostically. This simulates the guest network-path transition; it does not initiate host servicing or measure network capacity.
Use priority 4 to keep this longer continuity scenario out of P0-P3 runs; explicit or P4 selection remains supported.
Why this testcase is needed
The existing verify_sriov_disable_enable testcase verifies that Accelerated Networking can be toggled and that SR-IOV devices and drivers are healthy before and after the toggles. It does not verify what happens to ongoing application traffic during those transitions.
network_servicing_validation adds coverage for traffic continuity while the VF is detached, traffic falls back to the synthetic path, and the VF returns. It can detect packet loss or traffic stalls during the transition that successful AN toggles and final SR-IOV health checks alone would miss. This complements the existing testcase by checking workload continuity, not just successful network reconfiguration.
Reuse of existing LISA functionality
Existing LISA APIs are reused where their behavior matches this scenario. The private servicing helpers coordinate the additional continuity checks rather than introduce another traffic generator or AN implementation.
Area
Existing functionality reused
Why servicing-specific logic remains
Traffic and AN changes
Iperf3.run_as_server_async, Iperf3.run_as_client_async, and NetworkInterface.switch_sriov
Coordinate the same UDP/TCP streams across the accelerated, synthetic, and restored phases without restarting them.
VF state and receive progress
NIC discovery, Nics.get_packets for stable-path samples, LISA tools, and check_till_timeout
Transition probes share explicit deadlines; Nics.get_packets can reload NICs and does not accept a per-call timeout.
Cleanup and diagnostics
Lsof, Kill.by_pid, process result capture, and Dmesg
Stop only this testcase's port-scoped iperf processes gracefully, restore AN before SSH-dependent cleanup, and preserve the original test failure. The existing blanket cleanup_iperf3 would also stop unrelated iperf workloads.
Traffic validation and reporting
Existing iperf JSON output
Validate receiver packet loss and interval progress, and calculate phase-specific throughput/recovery. Existing iperf performance helpers aggregate whole-run metrics rather than provide these transition-specific checks.
Validation Results
Sriov.network_servicing_validation passed on both images below using two Standard_D48s_v6 VMs in westus3, with Accelerated Networking enabled and one Azure NIC per VM. Each run used a 120-second baseline and uninterrupted 1,140-second UDP/TCP streams at 1 Mbit/sec per stream.
Validation
Ubuntu 24.04
SLES 15 SP7
Exact image
canonical ubuntu-24_04-lts server 24.04.202610210
suse sles-15-sp7 gen2 2026.10.14
Kernel
7.0.0-1017-azure
6.4.0-150700.53.81-azure
Public LISA revision tested
f4c25516ba17a097e6caa6a22b84a5962c13e6a1
c897e4b76a5abde5d8ba047d29d7077f4f2b7dc5
Result
PASSED
PASSED
UDP sent / received / lost
118,751 / 118,751 / 0
118,751 / 118,751 / 0
TCP sent / received bytes
142,501,200 / 142,501,200
142,501,200 / 142,501,200
TCP retransmissions
0
2
All full traffic intervals made progress
Yes
Yes
All sender/receiver processes exited with code 0
Yes
Yes
AN disable confirmed
4.172 seconds
6.656 seconds
VF restored after enable request
6.656 seconds
6.265 seconds
VF receive progress after enable request
6.875 seconds
6.671 seconds
Five-interval 90% throughput recovery confirmed after enable
Approximately 13 seconds
Approximately 13 seconds
Internal addresses and routes unchanged
Both VMs
Both VMs
New detected kernel errors
None
None
Resource group cleanup
Independently verified deleted
Independently verified deleted
The Ubuntu rerun validates the cleanup-order, restored-phase-boundary, and testcase-specific iperf cleanup fixes. The SLES run predates those fixes and used the earlier restored-phase boundary. Both runs predate the new explicit single-NIC requirement and precondition check in the current revision 6d8b655e776ce11d9c498382dbc4916f3c38cb2d; this latest guard has passed focused local validation but has not yet been exercised in Azure.
The two SLES TCP retransmissions did not cause a stalled full interval or a byte-count mismatch. Zero loss is required for receiver-reported UDP; zero TCP retransmissions is not a test requirement.
The two-minute baseline retains pre-transition offered-load samples while reducing the traffic duration by three minutes. The transition timeouts, 30-second path observations, and traffic validation gates remain unchanged. These are continuity tests, not capacity or long-duration soak tests.
Recovery timing is diagnostic and confirms five consecutive qualifying reporting intervals at >=90% of baseline. It is not a measurement of outage duration.
Local checks
Reproduced the absence of a single-NIC selection constraint and early multi-NIC skip before the change.
All nine focused single-NIC checks passed with both normal and optimized Python: exact provisioning requirements, compatibility with multi-NIC-capable VM sizes provisioned with one NIC, synthetic/VF pairing, checking both VMs before side effects, receiver/sender multi-NIC skips, zero-NIC rejection, inventory-query error propagation, unchanged preparation for other cases, and no blanket iperf cleanup after a servicing precondition skip.
The prior 14 focused checks for cleanup ordering, exception preservation, skip safety, restored-phase reporting, unchanged traffic budgets, and port-scoped cleanup passed with normal and optimized Python.
Black, Flake8, focused mypy, and testcase import/registration passed. No permanent unit tests were added.
Pylint introduces no new diagnostics. Two existing E0606 findings in unrelated ethtool tests (reverted_settings and restored) reproduce on the pre-fix branch.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
NIC discovery fails in IPv6 configurations because it uses IPv4-only data, and VF polling can exceed its transition budget because remote commands are not bounded by the remaining deadline.
Adds network-servicing validation to LISA’s SR-IOV suite, exercising an accelerated-to-synthetic-to-accelerated transition between two Azure Linux VMs under continuous traffic.
Changes:
Adds paced UDP and TCP streams with baseline, synthetic-path, and restored-path observations.
Validates traffic continuity, UDP loss, addresses, routes, and kernel errors.
Adds state restoration, diagnostic collection, and throughput/recovery reporting without capacity thresholds.
Selected 25 test case(s): verify_bbr3_available,verify_bbr3_set_and_restore,verify_bbr3_applies_to_live_tcp_flow,verify_bbr3_kernel_config_runtime_consistency,verify_inet_diag_destroy,verify_inet_diag_enabled,verify_lwtunnel_bpf_support,validate_netvsc_reload,verify_network_interface_reload_via_ip_link,verify_ringbuffer_settings_change,verify_device_channels_change,verify_device_enabled_features,verify_device_gro_lro_settings_change,verify_device_rss_hash_key_change,verify_device_rx_hash_level_change,verify_device_msg_level_change,verify_device_statistics,verify_synthetic_provision_with_max_nics,verify_synthetic_provision_with_max_nics_reboot,verify_synthetic_provision_with_max_nics_reboot_from_platform,verify_synthetic_provision_with_max_nics_stop_start_from_platform,verify_synthetic_add_max_nics_one_time_after_provision,verify_synthetic_add_max_nics_one_by_one_after_provision,verify_xfrm_interface,verify_xfrm_interface_load_unload
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
An iperf timeout can leave the process result uncached, causing a subsequent assertion that interrupts collection of the remaining traffic diagnostics.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Cache timed-out iperf results before saving diagnostics
Major: If an iperf process times out here, Process._wait_result() recycles its backend and raises before caching its result. _servicing_save_buffers() then calls wait_result() on that same process, hits an assertion, and stops collecting the remaining traffic diagnostics. Pass raise_on_timeout=False here so the result is cached and saved before the existing exit-code assertion reports the failure.
Selected 25 test case(s): verify_bbr3_available,verify_bbr3_set_and_restore,verify_bbr3_applies_to_live_tcp_flow,verify_bbr3_kernel_config_runtime_consistency,verify_inet_diag_destroy,verify_inet_diag_enabled,verify_lwtunnel_bpf_support,validate_netvsc_reload,verify_network_interface_reload_via_ip_link,verify_ringbuffer_settings_change,verify_device_channels_change,verify_device_enabled_features,verify_device_gro_lro_settings_change,verify_device_rss_hash_key_change,verify_device_rx_hash_level_change,verify_device_msg_level_change,verify_device_statistics,verify_synthetic_provision_with_max_nics,verify_synthetic_provision_with_max_nics_reboot,verify_synthetic_provision_with_max_nics_reboot_from_platform,verify_synthetic_provision_with_max_nics_stop_start_from_platform,verify_synthetic_add_max_nics_one_time_after_provision,verify_synthetic_add_max_nics_one_by_one_after_provision,verify_xfrm_interface,verify_xfrm_interface_load_unload
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Cleanup can exhaust the case budget on SSH-dependent commands before AN restoration is submitted, while premature restored-phase timing can misreport throughput and recovery.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
A busy-port skip can still kill existing iperf3 workloads through blanket cleanup, and the configured validation durations are shorter than those specified in the PR description.
Selected 25 test case(s): verify_bbr3_available,verify_bbr3_set_and_restore,verify_bbr3_applies_to_live_tcp_flow,verify_bbr3_kernel_config_runtime_consistency,verify_inet_diag_destroy,verify_inet_diag_enabled,verify_lwtunnel_bpf_support,validate_netvsc_reload,verify_network_interface_reload_via_ip_link,verify_ringbuffer_settings_change,verify_device_channels_change,verify_device_enabled_features,verify_device_gro_lro_settings_change,verify_device_rss_hash_key_change,verify_device_rx_hash_level_change,verify_device_msg_level_change,verify_device_statistics,verify_synthetic_provision_with_max_nics,verify_synthetic_provision_with_max_nics_reboot,verify_synthetic_provision_with_max_nics_reboot_from_platform,verify_synthetic_provision_with_max_nics_stop_start_from_platform,verify_synthetic_add_max_nics_one_time_after_provision,verify_synthetic_add_max_nics_one_by_one_after_provision,verify_xfrm_interface,verify_xfrm_interface_load_unload
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Cleanup can disable SR-IOV on an originally accelerated primary NIC when a synthetic secondary NIC causes the saved baseline state to be false, so per-NIC settings must be preserved or mixed-NIC receivers skipped.
Selected 25 test case(s): verify_bbr3_available,verify_bbr3_set_and_restore,verify_bbr3_applies_to_live_tcp_flow,verify_bbr3_kernel_config_runtime_consistency,verify_inet_diag_destroy,verify_inet_diag_enabled,verify_lwtunnel_bpf_support,validate_netvsc_reload,verify_network_interface_reload_via_ip_link,verify_ringbuffer_settings_change,verify_device_channels_change,verify_device_enabled_features,verify_device_gro_lro_settings_change,verify_device_rss_hash_key_change,verify_device_rx_hash_level_change,verify_device_msg_level_change,verify_device_statistics,verify_synthetic_provision_with_max_nics,verify_synthetic_provision_with_max_nics_reboot,verify_synthetic_provision_with_max_nics_reboot_from_platform,verify_synthetic_provision_with_max_nics_stop_start_from_platform,verify_synthetic_add_max_nics_one_time_after_provision,verify_synthetic_add_max_nics_one_by_one_after_provision,verify_xfrm_interface,verify_xfrm_interface_load_unload
Selected 25 test case(s): verify_bbr3_available,verify_bbr3_set_and_restore,verify_bbr3_applies_to_live_tcp_flow,verify_bbr3_kernel_config_runtime_consistency,verify_inet_diag_destroy,verify_inet_diag_enabled,verify_lwtunnel_bpf_support,validate_netvsc_reload,verify_network_interface_reload_via_ip_link,verify_ringbuffer_settings_change,verify_device_channels_change,verify_device_enabled_features,verify_device_gro_lro_settings_change,verify_device_rss_hash_key_change,verify_device_rx_hash_level_change,verify_device_msg_level_change,verify_device_statistics,verify_synthetic_provision_with_max_nics,verify_synthetic_provision_with_max_nics_reboot,verify_synthetic_provision_with_max_nics_reboot_from_platform,verify_synthetic_provision_with_max_nics_stop_start_from_platform,verify_synthetic_add_max_nics_one_time_after_provision,verify_synthetic_add_max_nics_one_by_one_after_provision,verify_xfrm_interface,verify_xfrm_interface_load_unload
Selected 25 test case(s): verify_bbr3_available,verify_bbr3_set_and_restore,verify_bbr3_applies_to_live_tcp_flow,verify_bbr3_kernel_config_runtime_consistency,verify_inet_diag_destroy,verify_inet_diag_enabled,verify_lwtunnel_bpf_support,validate_netvsc_reload,verify_network_interface_reload_via_ip_link,verify_ringbuffer_settings_change,verify_device_channels_change,verify_device_enabled_features,verify_device_gro_lro_settings_change,verify_device_rss_hash_key_change,verify_device_rx_hash_level_change,verify_device_msg_level_change,verify_device_statistics,verify_synthetic_provision_with_max_nics,verify_synthetic_provision_with_max_nics_reboot,verify_synthetic_provision_with_max_nics_reboot_from_platform,verify_synthetic_provision_with_max_nics_stop_start_from_platform,verify_synthetic_add_max_nics_one_time_after_provision,verify_synthetic_add_max_nics_one_by_one_after_provision,verify_xfrm_interface,verify_xfrm_interface_load_unload
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sriov.network_servicing_validationto exercise one accelerated-to-synthetic-to-accelerated networking transition between two Linux VMs under continuous, paced UDP and TCP traffic.Why this testcase is needed
The existing
verify_sriov_disable_enabletestcase verifies that Accelerated Networking can be toggled and that SR-IOV devices and drivers are healthy before and after the toggles. It does not verify what happens to ongoing application traffic during those transitions.network_servicing_validationadds coverage for traffic continuity while the VF is detached, traffic falls back to the synthetic path, and the VF returns. It can detect packet loss or traffic stalls during the transition that successful AN toggles and final SR-IOV health checks alone would miss. This complements the existing testcase by checking workload continuity, not just successful network reconfiguration.Reuse of existing LISA functionality
Existing LISA APIs are reused where their behavior matches this scenario. The private servicing helpers coordinate the additional continuity checks rather than introduce another traffic generator or AN implementation.
Iperf3.run_as_server_async,Iperf3.run_as_client_async, andNetworkInterface.switch_sriovNics.get_packetsfor stable-path samples, LISA tools, andcheck_till_timeoutNics.get_packetscan reload NICs and does not accept a per-call timeout.Lsof,Kill.by_pid, process result capture, andDmesgcleanup_iperf3would also stop unrelated iperf workloads.Validation Results
Sriov.network_servicing_validationpassed on both images below using twoStandard_D48s_v6VMs inwestus3, with Accelerated Networking enabled and one Azure NIC per VM. Each run used a 120-second baseline and uninterrupted 1,140-second UDP/TCP streams at 1 Mbit/sec per stream.canonical ubuntu-24_04-lts server 24.04.202610210suse sles-15-sp7 gen2 2026.10.147.0.0-1017-azure6.4.0-150700.53.81-azuref4c25516ba17a097e6caa6a22b84a5962c13e6a1c897e4b76a5abde5d8ba047d29d7077f4f2b7dc5The Ubuntu rerun validates the cleanup-order, restored-phase-boundary, and testcase-specific iperf cleanup fixes. The SLES run predates those fixes and used the earlier restored-phase boundary. Both runs predate the new explicit single-NIC requirement and precondition check in the current revision
6d8b655e776ce11d9c498382dbc4916f3c38cb2d; this latest guard has passed focused local validation but has not yet been exercised in Azure.The two SLES TCP retransmissions did not cause a stalled full interval or a byte-count mismatch. Zero loss is required for receiver-reported UDP; zero TCP retransmissions is not a test requirement.
The two-minute baseline retains pre-transition offered-load samples while reducing the traffic duration by three minutes. The transition timeouts, 30-second path observations, and traffic validation gates remain unchanged. These are continuity tests, not capacity or long-duration soak tests.
Recovery timing is diagnostic and confirms five consecutive qualifying reporting intervals at >=90% of baseline. It is not a measurement of outage duration.
Local checks
E0606findings in unrelated ethtool tests (reverted_settingsandrestored) reproduce on the pre-fix branch.