Skip to content

dpdk-stdlib-tcp: Contract types, TcpState, Clock, and IsnGenerator - #80

Merged
gspivey merged 2 commits into
developmentfrom
agent/tcp-contract-types-state-clock-isn
Jun 13, 2026
Merged

gspivey merged 2 commits into
developmentfrom
agent/tcp-contract-types-state-clock-isn

Conversation

@gspivey

@gspivey gspivey commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Roadmap Item

Implements #16: dpdk-stdlib-tcp: Contract types, TcpState, Clock, and IsnGenerator

Spec tasks: 5.1, 5.2, 5.3, 5.4

Changes

Four new source modules in dpdk-stdlib-tcp/src/:

  • state.rs — TcpState enum (11 states per RFC 9293) + FourTuple (Hash, Eq, serializable to bytes)
  • clock.rs — Clock trait + SystemClock + MockClock (with advance()/set() for deterministic testing)
  • isn.rs — IsnGenerator per RFC 6528 (128-bit per-boot secret via getrandom, SipHash-2-4 of FourTuple, M = elapsed µs / 4)
  • contract.rs — ConnectionHandle (SPSC rings, atomic state, condvar wake, AtomicWaker), EngineCommand enum, SocketOption enum, CommandSender (wraps mpsc + EngineWakeup signal), OneshotSender/OneshotReceiver, EngineWakeup, AtomicWaker

Updated lib.rs with module declarations and Cargo.toml with getrandom + siphasher dependencies.

Tests

  • 23 new unit tests covering all new types
  • All 65 TCP tests pass (55 unit + 10 property)
  • Full workspace build and test pass (800+ tests)

Tradeoffs

  • AtomicWaker uses Mutex<Option<Waker>> for simplicity (no unsafe, no external dep). Sufficient for MVP; can be replaced with lock-free implementation if contention is measured.
  • EngineWakeup uses AtomicBool + Condvar (portable). Linux eventfd optimization deferred to engine loop implementation.
  • OneshotSender/OneshotReceiver avoid tokio dependency per spec (CI enforces no tokio dep in dpdk-stdlib-tcp).

Implement spec tasks 5.1, 5.2, 5.3, 5.4 for roadmap item #16.

New modules:
- state.rs: TcpState enum (11 states) + FourTuple
- clock.rs: Clock trait, SystemClock, MockClock (deterministic testing)
- isn.rs: IsnGenerator per RFC 6528 (SipHash-2-4 + per-boot secret)
- contract.rs: ConnectionHandle, EngineCommand, SocketOption,
  CommandSender, OneshotSender/Receiver, EngineWakeup, AtomicWaker

Dependencies added: getrandom, siphasher.
All 65 TCP tests pass (55 unit + 10 property).
@github-actions

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: a2203962

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 14.6M 14.0M 1.0x 71
RX recv_from 64B 4.4M 5.7M 0.8x 175
TX send_to 1400B 2.1M 2.1M 1.0x 476
RX recv_from 1400B 1.3M 1.4M 0.9x 689

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 11.0M 10.9M 1.0x 92
RX recv_from (IPv6) 64B 5.1M 7.0M 0.7x 141
TX send_to (IPv6) 1400B 3.4M 3.4M 1.0x 295
RX recv_from (IPv6) 1400B 1.4M 1.4M 0.9x 695

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 14.6M 11.0M 1.32x
RX recv_from (sync) 64B 4.4M 5.1M 0.87x
TX send_to (sync) 1400B 2.1M 3.4M 0.62x
RX recv_from (sync) 1400B 1.3M 1.4M 0.97x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.3% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: a2203962

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.4M 10.9M 1.0x 91
RX recv_from 64B 3.4M 4.6M 0.7x 218
TX send_to 1400B 1.6M 1.6M 1.0x 612
RX recv_from 1400B 982.8K 1.1M 0.9x 936

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 8.6M 8.4M 1.0x 119
RX recv_from (IPv6) 64B 3.9M 5.5M 0.7x 182
TX send_to (IPv6) 1400B 2.7M 2.6M 1.0x 377
RX recv_from (IPv6) 1400B 1.0M 1.1M 0.9x 920

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.4M 8.6M 1.32x
RX recv_from (sync) 64B 3.4M 3.9M 0.86x
TX send_to (sync) 1400B 1.6M 2.7M 0.61x
RX recv_from (sync) 1400B 982.8K 1.0M 0.97x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.3% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0348613fa0dfcb45d (DPDK ENI: 10.0.1.220)
  • Receiver: i-0a067775c463afa3f (DPDK ENI: 10.0.1.134)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0f9edfb7f483bd2fd (DPDK ENI: 10.0.1.52)
  • Receiver: i-0c311a47ea475115a (DPDK ENI: 10.0.1.38)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

1 similar comment
@github-actions

Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions

Copy link
Copy Markdown

✅ Integration Tests Passed — Graviton (run)

Branch: 80/merge | Commit: a2203962

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.134:9000 (MAC: 02:e3:1b:69:26:49)
echo listening on 10.0.1.134:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.220:9000 (MAC: 02:b8:f9:97:ed:dd)
echo listening on 10.0.1.220:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.134:9000: 'arp-probe #1'
Test complete
[2026-06-13T18:45:05Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T18:45:05Z] INFO: Test: udp_send_receive
[2026-06-13T18:45:06Z] INFO: UDP send/receive succeeded
[2026-06-13T18:45:06Z] INFO: Test: echo_roundtrip
[2026-06-13T18:45:06Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T18:45:06Z] INFO: Test: payload_integrity
[2026-06-13T18:45:07Z] INFO: Response received, checking payload match...
[2026-06-13T18:45:07Z] INFO: Payload integrity verified (found in response)
[2026-06-13T18:45:07Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-13T18:45:07Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Deploy

Deploying PerfTestStack...
Configs: plain-rust,rust-dpdk,tokio-dpdk,native-dpdk
Packet sizes: 64,512,1400,8500

@github-actions

Copy link
Copy Markdown

✅ Integration Tests Passed (Run 27475357505)

Branch: 80/merge | Commit: a2203962

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.38:9000 (MAC: 02:7e:0b:ed:e0:37)
echo listening on 10.0.1.38:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.52:9000 (MAC: 02:18:ce:44:21:39)
echo listening on 10.0.1.52:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.38:9000: 'arp-probe #1'
Test complete
[2026-06-13T18:47:03Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T18:47:03Z] INFO: Test: udp_send_receive
[2026-06-13T18:47:04Z] INFO: UDP send/receive succeeded
[2026-06-13T18:47:04Z] INFO: Test: echo_roundtrip
[2026-06-13T18:47:05Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T18:47:05Z] INFO: Test: payload_integrity
[2026-06-13T18:47:05Z] INFO: Response received, checking payload match...
[2026-06-13T18:47:05Z] INFO: Payload integrity verified (found in response)
[2026-06-13T18:47:05Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-13T18:47:05Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-06-13T18:54:04Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-06-13T18:54:04Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-06-13T18:54:04Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-06-13T18:54:04Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-06-13T18:53:04Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-06-13T18:53:04Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-06-13T18:53:05Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-06-13T18:53:05Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.38:9000 (MAC: 02:7e:0b:ed:e0:37)
echo listening on 10.0.1.38:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.52:9000 (MAC: 02:18:ce:44:21:39)
echo listening on 10.0.1.52:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-06-13T18:42:30Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.38:9000
Bind address: 10.0.1.52:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.52:32768 (MAC: 02:18:ce:44:21:39)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.38:9000: 'arp-probe #1'
Test complete
[2026-06-13T18:42:31Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T18:42:31Z] INFO: Test: udp_send_receive
[2026-06-13T18:42:33Z] INFO: UDP send/receive succeeded
[2026-06-13T18:42:33Z] INFO: Test: echo_roundtrip
[2026-06-13T18:42:34Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T18:42:34Z] INFO: Test: payload_integrity
[2026-06-13T18:42:35Z] INFO: Response received, checking payload match...
[2026-06-13T18:42:35Z] INFO: Payload integrity verified (found in response)
[2026-06-13T18:42:35Z] INFO: Test: jumbo_diagnostics
[2026-06-13T18:42:35Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-06-13T18:42:35Z] INFO: Interface MTU:
  9001
  65536
[2026-06-13T18:42:35Z] INFO:   ens5: MTU=9001
[2026-06-13T18:42:35Z] INFO:   lo: MTU=65536
[2026-06-13T18:42:35Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-06-13T18:42:35Z] INFO: DPDK port config (from echo server log):
[2026-06-13T18:42:35Z] INFO:   (no MTU info in echo log)
[2026-06-13T18:42:35Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-06-13T18:42:35Z] INFO: Test: jumbo_echo_8000
[2026-06-13T18:42:36Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.38:9000
Bind address: 10.0.1.52:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.52:32768 (MAC: 02:18:ce:44:21:39)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.38:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.38:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.38:9000 (expected 8000, OK)
Test complete
[2026-06-13T18:42:37Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-06-13T18:42:37Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-06-13T18:42:37Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-06-13T18:47:03Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.38:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.38:9000: 'arp-probe #1'
Test complete
[2026-06-13T18:47:03Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T18:47:03Z] INFO: Test: udp_send_receive
[2026-06-13T18:47:04Z] INFO: UDP send/receive succeeded
[2026-06-13T18:47:04Z] INFO: Test: echo_roundtrip
[2026-06-13T18:47:05Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T18:47:05Z] INFO: Test: payload_integrity
[2026-06-13T18:47:05Z] INFO: Response received, checking payload match...
[2026-06-13T18:47:05Z] INFO: Payload integrity verified (found in response)
[2026-06-13T18:47:05Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-13T18:47:05Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-06-13T18:54:04Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-06-13T18:54:04Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-06-13T18:54:04Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-06-13T18:54:04Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-06-13T18:53:02Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-06-13T18:53:02Z] INFO: Local IP: 10.0.1.52, Peer IP: 10.0.1.38, Port: 9000
[2026-06-13T18:53:02Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.103       0x1         0x2         02:4c:d8:d5:55:2d     *        ens5
10.0.1.224       0x1         0x2         02:d0:01:18:73:5f     *        ens5
10.0.1.38        0x1         0x2         02:7e:0b:ed:e0:37     *        ens5
10.0.1.52        0x1         0x2         02:18:ce:44:21:39     *        ens5
10.0.1.1         0x1         0x2         02:8f:5c:d0:52:ef     *        ens5
[2026-06-13T18:53:02Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-06-13T18:53:02Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-06-13T18:53:02Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1916464 Jun 13 18:39 /opt/dpdk-stdlib/target/release/test-client
[2026-06-13T18:53:02Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.38 --port 9000 --bind-ip 10.0.1.52 --count 10 --delay 200
[2026-06-13T18:53:04Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.38:9000
Bind address: 10.0.1.52:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.52:32768 (MAC: 02:18:ce:44:21:39)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.38:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-06-13T18:53:04Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-06-13T18:53:04Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-06-13T18:53:05Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-06-13T18:53:05Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0c311a47ea475115a (receiver)
Command ID: bf059da1-62f2-472b-b63a-92a01ac16140

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0c311a47ea475115a (receiver)
Command ID: 53c7a4e0-d05c-4d14-bdfb-09383a06b88b

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0c311a47ea475115a (receiver)
Command ID: 86875e35-52b4-47c7-b6e7-8cae1cb9775d

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0c311a47ea475115a (receiver)
Command ID: c2a49694-f4ef-4ca8-bea8-4f08f3072987

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.52) ===
Status: InProgress
Instance: i-0f9edfb7f483bd2fd (sender)
Command ID: 83b11008-34d2-4fc7-8eea-a5c9f8b43102

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0f9edfb7f483bd2fd (sender)
Command ID: cdc63cfa-1983-46d0-9d03-ccaa5928986c

=== STDOUT ===


=== STDERR ===


Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:bc:1f:fa:e6:41 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0ad13571c4a2a10c3
    altname device-number-0.0
    inet 10.0.1.57/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2134sec preferred_lft 2134sec
    inet6 fe80::bc:1fff:fefa:e641/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ea:c2:15:5a:87 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-00902e0dcd361abe4
    altname device-number-0.0
    inet 10.0.1.187/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2157sec preferred_lft 2157sec
    inet6 fe80::ea:c2ff:fe15:5a87/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-06-13T18:41:51Z
hostname: ip-10-0-1-57.ec2.internal
kernel: 6.18.33-63.124.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:bc:1f:fa:e6:41 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0ad13571c4a2a10c3
    altname device-number-0.0
    inet 10.0.1.57/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3165sec preferred_lft 3165sec
    inet6 fe80::bc:1fff:fefa:e641/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:8f:5c:d0:52:ef REACHABLE 
10.0.1.103 dev ens5 lladdr 02:4c:d8:d5:55:2d REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.57 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.57 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.57 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.57 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:7e:0b:ed:e0:37/ 02:bc:1f:fa:e6:41/ 

--- ENI: 02:7e:0b:ed:e0:37/ ---
  device-number: 1
  local-ipv4s: 10.0.1.38
  subnet-id: subnet-038c4fcadf0bb663e
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:bc:1f:fa:e6:41/ ---
  device-number: 0
  local-ipv4s: 10.0.1.57
  subnet-id: subnet-038c4fcadf0bb663e
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:8f:5c:d0:52:ef REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.57 ens5
Unicast reply from 10.0.1.1 [02:8F:5C:D0:52:EF]  0.540ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jun 13 18:41 .
drwxr-xr-x. 14 root root     3100 Jun 13 18:41 ..
crw-------.  1 root root 243,   0 Jun 13 18:41 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jun 13 18:34 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.053743] printk: legacy console [ttyS0] enabled
[    0.054770] x2apic enabled
[    0.058999] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.059108] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.068862] audit: type=2000 audit(1781375668.857:1): state=initialized audit_enabled=0 res=1
[    0.088447] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.158643] ACPI: Interpreter enabled
[    0.158689] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.177266] pci 0000:00:05.0: enabling Extended Tags
[    0.228324] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.233722] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.237183] IPI shorthand broadcast: enabled
[    5.165595] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    5.198871] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    5.347955] VFIO - User Level meta-driver version: 0.3
[    5.805299] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    5.826826] ena 0000:00:05.0: ENA device version: 0.10
[    5.827551] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    5.930480] ena 0000:00:05.0: ENA Large LLQ is disabled
[    5.943299] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:bc:1f:fa:e6:41
[    6.009178] ena 0000:00:05.0 ens5: renamed from eth0
[  391.556351] pci 0000:00:06.0: enabling Extended Tags
[  391.560353] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  391.573171] ena 0000:00:06.0: ENA device version: 0.10
[  391.573925] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  391.675108] ena 0000:00:06.0: ENA Large LLQ is disabled
[  391.687908] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:7e:0b:ed:e0:37
[  391.695073] ena 0000:00:06.0 ens6: renamed from eth0
[  429.160893] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  429.162326] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-06-13T18:41:43Z
hostname: ip-10-0-1-187.ec2.internal
kernel: 6.18.33-63.124.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ea:c2:15:5a:87 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-00902e0dcd361abe4
    altname device-number-0.0
    inet 10.0.1.187/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3174sec preferred_lft 3174sec
    inet6 fe80::ea:c2ff:fe15:5a87/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.103 dev ens5 lladdr 02:4c:d8:d5:55:2d REACHABLE 
10.0.1.224 dev ens5 lladdr 02:d0:01:18:73:5f REACHABLE 
10.0.1.1 dev ens5 lladdr 02:8f:5c:d0:52:ef REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.187 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.187 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.187 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.187 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:18:ce:44:21:39/ 02:ea:c2:15:5a:87/ 

--- ENI: 02:18:ce:44:21:39/ ---
  device-number: 1
  local-ipv4s: 10.0.1.52
  subnet-id: subnet-038c4fcadf0bb663e
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:ea:c2:15:5a:87/ ---
  device-number: 0
  local-ipv4s: 10.0.1.187
  subnet-id: subnet-038c4fcadf0bb663e
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:8f:5c:d0:52:ef REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.187 ens5
Unicast reply from 10.0.1.1 [02:8F:5C:D0:52:EF]  0.535ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jun 13 18:41 .
drwxr-xr-x. 14 root root     3100 Jun 13 18:41 ..
crw-------.  1 root root 243,   0 Jun 13 18:41 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jun 13 18:34 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.053043] printk: legacy console [ttyS0] enabled
[    0.054072] x2apic enabled
[    0.058324] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.058433] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.068177] audit: type=2000 audit(1781375669.571:1): state=initialized audit_enabled=0 res=1
[    0.087768] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.157864] ACPI: Interpreter enabled
[    0.158017] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.176334] pci 0000:00:05.0: enabling Extended Tags
[    0.228581] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.239496] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.245050] IPI shorthand broadcast: enabled
[    4.904865] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    4.939807] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    5.009563] VFIO - User Level meta-driver version: 0.3
[    5.712912] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    5.724192] ena 0000:00:05.0: ENA device version: 0.10
[    5.724947] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    5.824779] ena 0000:00:05.0: ENA Large LLQ is disabled
[    5.837740] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:ea:c2:15:5a:87
[    6.013031] ena 0000:00:05.0 ens5: renamed from eth0
[  392.164620] pci 0000:00:06.0: enabling Extended Tags
[  392.168629] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  392.179260] ena 0000:00:06.0: ENA device version: 0.10
[  392.180020] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  392.279234] ena 0000:00:06.0: ENA Large LLQ is disabled
[  392.291095] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:18:ce:44:21:39
[  392.298823] ena 0000:00:06.0 ens6: renamed from eth0
[  421.782419] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  421.783859] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.068259] pid_max: default: 32768 minimum: 301
[    0.180949] iommu: Default domain type: Translated
[    0.180949] NetLabel:  unlabeled traffic allowed by default
[    0.202251] PCI: CLS 0 bytes, default 64
[    0.256551] nvme nvme0: 2/0/0 default/read/poll queues
[    0.439312] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.439439] systemd[1]: No hostname configured, using default hostname.
[    0.505266] systemd[1]: Queued start job for default target initrd.target.
[    3.540033] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  465.930341] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:7998)
[  739.396903] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8438)
[ 1336.159439] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9601)

sender-dmesg-crashes.log

[    0.067575] pid_max: default: 32768 minimum: 301
[    0.180053] iommu: Default domain type: Translated
[    0.180053] NetLabel:  unlabeled traffic allowed by default
[    0.201616] PCI: CLS 0 bytes, default 64
[    0.256502] nvme nvme0: 2/0/0 default/read/poll queues
[    0.438926] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.439048] systemd[1]: No hostname configured, using default hostname.
[    0.505605] systemd[1]: Queued start job for default target initrd.target.
[    3.821171] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 1157.071467] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9314)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  465.928771] vfio-pci 0000:00:06.0: reset done
[  465.930341] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:7998)
[  465.931739] vfio-pci 0000:00:06.0: resetting
[  466.148636] vfio-pci 0000:00:06.0: reset done
[  545.626500] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  546.636113] ena 0000:00:06.0: ENA device version: 0.10
[  546.636883] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  546.736070] ena 0000:00:06.0: ENA Large LLQ is disabled
[  546.748053] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:7e:0b:ed:e0:37
[  546.756577] ena 0000:00:06.0 ens6: renamed from eth0
[  566.340652] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  566.342059] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  739.178147] vfio-pci 0000:00:06.0: resetting
[  739.395348] vfio-pci 0000:00:06.0: reset done
[  739.396903] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8438)
[  739.398316] vfio-pci 0000:00:06.0: resetting
[  739.615226] vfio-pci 0000:00:06.0: reset done
[  813.722005] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  814.732751] ena 0000:00:06.0: ENA device version: 0.10
[  814.733506] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  814.832723] ena 0000:00:06.0: ENA Large LLQ is disabled
[  814.844678] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:7e:0b:ed:e0:37
[  814.852673] ena 0000:00:06.0 ens6: renamed from eth0
[ 1256.430997] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1256.432402] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1335.937012] vfio-pci 0000:00:06.0: resetting
[ 1336.157879] vfio-pci 0000:00:06.0: reset done
[ 1336.159439] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9601)
[ 1336.160871] vfio-pci 0000:00:06.0: resetting
[ 1336.377742] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[ 1157.072901] vfio-pci 0000:00:06.0: resetting
[ 1157.289753] vfio-pci 0000:00:06.0: reset done
[ 1218.621949] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1219.635036] ena 0000:00:06.0: ENA device version: 0.10
[ 1219.635806] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1219.736994] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1219.749182] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:18:ce:44:21:39
[ 1219.760363] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1248.971882] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1248.973247] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1351.022879] vfio-pci 0000:00:06.0: resetting
[ 1351.237158] vfio-pci 0000:00:06.0: reset done
[ 1351.238708] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9754)
[ 1351.240268] vfio-pci 0000:00:06.0: resetting
[ 1351.457007] vfio-pci 0000:00:06.0: reset done
[ 1351.744711] vfio-pci 0000:00:06.0: resetting
[ 1351.956990] vfio-pci 0000:00:06.0: reset done
[ 1351.958558] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9776)
[ 1351.960143] vfio-pci 0000:00:06.0: resetting
[ 1352.176965] vfio-pci 0000:00:06.0: reset done
[ 1353.466342] vfio-pci 0000:00:06.0: resetting
[ 1353.687039] vfio-pci 0000:00:06.0: reset done
[ 1353.688606] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9799)
[ 1353.690157] vfio-pci 0000:00:06.0: resetting
[ 1353.906983] vfio-pci 0000:00:06.0: reset done
[ 1355.186551] vfio-pci 0000:00:06.0: resetting
[ 1355.407090] vfio-pci 0000:00:06.0: reset done
[ 1355.408583] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9822)
[ 1355.410058] vfio-pci 0000:00:06.0: resetting
[ 1355.626982] vfio-pci 0000:00:06.0: reset done

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Instances Ready

  • TRex: i-0d553fde1b6769308 (10.0.1.224)
  • DUT: i-011fe211a50a14eb7 (10.0.1.214)
  • Instance type: unknown

@github-actions

Copy link
Copy Markdown

[Perf] Stage: TRex Config

Starting TRex configuration (MAC discovery + NIC binding)...

@github-actions

Copy link
Copy Markdown

[Perf] Stage: TRex Config OK

  • TX: 0000:00:06.0 MAC: 02:a7:26:b1:09:a5
  • RX: 0000:00:07.0 MAC: 02:8c:1c:f3:c5:ed
  • Gateway MAC: 02:6e:15:d1:ab:5d
    Starting TRex server...

@github-actions

Copy link
Copy Markdown

[Perf] Stage: TRex Started

TRex server running. Beginning benchmarks...

@github-actions

Copy link
Copy Markdown

[Perf] DUT Ready

DUT instance i-011fe211a50a14eb7 SSM working, build complete.

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (1/4)

Running plain-rust benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: plain-rust preflight

TRex API OK: 2 ports
  Port 0: 02:a7:26:b1:09:a5
  Port 1: 02:8c:1c:f3:c5:ed
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: plain-rust result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:a7:26:b1:09:a5

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,997 pps, Drop: 0.7167%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,985 pps, Drop: 0.2899%, Lat avg: 257.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 381,095 pps, Drop: 45.5579%, Lat avg: 673.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 227.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,978 pps, Drop: 0.73%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,815 pps, Drop: 0.3385%, Lat avg: 241.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 398,868 pps, Drop: 43.0189%, Lat avg: 0.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 216.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,990 pps, Drop: 0.7217%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,770 pps, Drop: 0.3513%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 475,846 pps, RX: 447,288 pps, Drop: 6.0015%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 38,549 pps, Drop: 44.9296%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 78,352 pps, RX: 77,780 pps, Drop: 0.7302%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 78,286 pps, RX: 77,905 pps, Drop: 0.4862%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/plain-rust.json
EXIT_CODE=0

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (2/4)

Running rust-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: rust-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:a7:26:b1:09:a5
  Port 1: 02:8c:1c:f3:c5:ed
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: rust-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:a7:26:b1:09:a5

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7142%, Lat avg: 203.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,990 pps, Drop: 0.2886%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 658,996 pps, Drop: 5.8577%, Lat avg: 1045.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 167.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7142%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,990 pps, Drop: 0.2886%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 620,084 pps, Drop: 11.4167%, Lat avg: 2952.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7142%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 349,000 pps, Drop: 0.2857%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 475,764 pps, RX: 466,142 pps, Drop: 2.0224%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 78,340 pps, RX: 77,724 pps, Drop: 0.7862%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 78,336 pps, RX: 77,900 pps, Drop: 0.5568%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/rust-dpdk.json
EXIT_CODE=0

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (3/4)

Running tokio-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: tokio-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:a7:26:b1:09:a5
  Port 1: 02:8c:1c:f3:c5:ed
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: tokio-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:a7:26:b1:09:a5

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7142%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 311,544 pps, Drop: 10.9874%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 312,244 pps, Drop: 55.3937%, Lat avg: 0.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7142%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 231,845 pps, Drop: 33.7586%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 231,659 pps, Drop: 66.9058%, Lat avg: 0.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4285%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7142%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 150,532 pps, Drop: 56.9908%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 476,267 pps, RX: 160,320 pps, Drop: 66.3381%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 55,327 pps, Drop: 20.9612%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 78,350 pps, RX: 58,906 pps, Drop: 24.8167%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 78,329 pps, RX: 59,005 pps, Drop: 24.6708%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/tokio-dpdk.json
EXIT_CODE=0

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (4/4)

Running native-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: native-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:a7:26:b1:09:a5
  Port 1: 02:8c:1c:f3:c5:ed
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: native-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:a7:26:b1:09:a5

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 128.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 140,000 pps, Drop: 0.0%, Lat avg: 138.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 349,973 pps, Drop: 0.0076%, Lat avg: 148.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 656,738 pps, Drop: 6.1802%, Lat avg: 488.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 123.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 140,000 pps, Drop: 0.0%, Lat avg: 117.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 349,978 pps, Drop: 0.0062%, Lat avg: 161.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 598,345 pps, Drop: 14.5222%, Lat avg: 956.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 122.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 140,000 pps, Drop: 0.0%, Lat avg: 145.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 349,998 pps, Drop: 0.0005%, Lat avg: 133.0 us
  Target: 700,000 pps ... TX: 475,188 pps, RX: 460,415 pps, Drop: 3.1089%, Lat avg: 2528.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 138.0 us
  Target: 140,000 pps ... TX: 78,293 pps, RX: 78,266 pps, Drop: 0.0349%, Lat avg: 13847.0 us
  Target: 350,000 pps ... TX: 78,285 pps, RX: 77,960 pps, Drop: 0.4144%, Lat avg: 14172.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/native-dpdk.json
EXIT_CODE=0

@github-actions

Copy link
Copy Markdown

[Perf] Diag: testpmd log

testpmd output (last 30 lines)
=== testpmd stats (last 30 lines) ===
  ############################################################################
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 113446482      RX-dropped: 0             RX-total: 113446482
  RX-error: 403557
  RX-nombufs: 0             
  TX-packets: 113446102      TX-dropped: 380           TX-total: 113446482
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 113446482      RX-dropped: 0             RX-total: 113446482
  TX-packets: 113446102      TX-dropped: 380           TX-total: 113446482
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Shutting down port 0...
ena_rx_queue_release(): Rx queue 0:0 released
ena_tx_queue_release(): Tx queue 0:0 released
Closing ports...
Port 0 is closed
Done

Bye...
=== ENA port stats ===
N/A (vfio-pci)

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Results

[19:55:06] INFO Generating markdown summary...

Performance Test Results — unknown

Commit: 22c0cd5c
Timestamp: 2026-06-13T19:55:06.707635+00:00

64B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% — — — — 128.0 468.0 35.8 35.8
native-dpdk 140,000 140,000 140,000 0.00% — — — — 138.0 534.0 71.7 71.7
native-dpdk 350,000 350,000 349,973 0.01% — — — — 148.0 2004.0 179.2 179.2
native-dpdk 700,000 700,000 656,738 6.18% — — — — 488.0 3030.0 358.4 336.2
rust-dpdk 70,000 70,000 69,000 1.43% 0 31,692 0 0 0.0 0.0 35.8 35.3
rust-dpdk 140,000 140,000 139,000 0.71% 0 41,914 0 0 203.0 0.0 71.7 71.2
rust-dpdk 350,000 350,000 348,990 0.29% 0 33,566 0 0 0.0 0.0 179.2 178.7
rust-dpdk 700,000 700,000 658,996 5.86% 0 41,844 0 0 1045.0 0.0 358.4 337.4
tokio-dpdk 70,000 70,000 69,000 1.43% 0 31,978 0 0 0.0 0.0 35.8 38.6
tokio-dpdk 140,000 140,000 139,000 0.71% 0 38,242 0 0 0.0 0.0 71.7 77.8
tokio-dpdk 350,000 350,000 311,544 10.99% 0 22,113 0 0 0.0 0.0 179.2 174.5
tokio-dpdk 700,000 700,000 312,244 55.39% 0 24,750 0 0 0.0 0.0 358.4 174.9
plain-rust 70,000 70,000 69,000 1.43% — — — — 0.0 0.0 35.8 35.3
plain-rust 140,000 140,000 138,997 0.72% — — — — 0.0 0.0 71.7 71.2
plain-rust 350,000 350,000 348,985 0.29% — — — — 257.0 0.0 179.2 178.7
plain-rust 700,000 700,000 381,095 45.56% — — — — 673.0 0.0 358.4 195.1

512B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% — — — — 123.0 464.0 286.7 286.7
native-dpdk 140,000 140,000 140,000 0.00% — — — — 117.0 432.0 573.4 573.4
native-dpdk 350,000 350,000 349,978 0.01% — — — — 161.0 2397.0 1433.6 1433.5
native-dpdk 700,000 700,000 598,345 14.52% — — — — 956.0 5046.0 2867.2 2450.8
rust-dpdk 70,000 70,000 69,000 1.43% 0 35,521 0 0 167.0 0.0 286.7 282.6
rust-dpdk 140,000 140,000 139,000 0.71% 0 31,936 0 0 0.0 0.0 573.4 569.3
rust-dpdk 350,000 350,000 348,990 0.29% 0 41,089 0 0 0.0 0.0 1433.6 1429.5
rust-dpdk 700,000 700,000 620,084 11.42% 0 33,325 0 0 2952.0 0.0 2867.2 2539.9
tokio-dpdk 70,000 70,000 69,000 1.43% 0 35,814 0 0 0.0 0.0 286.7 285.9
tokio-dpdk 140,000 140,000 139,000 0.71% 0 30,758 0 0 0.0 0.0 573.4 576.0
tokio-dpdk 350,000 350,000 231,845 33.76% 0 17,187 0 0 0.0 0.0 1433.6 960.8
tokio-dpdk 700,000 700,000 231,659 66.91% 0 13,072 0 0 0.0 0.0 2867.2 960.0
plain-rust 70,000 70,000 69,000 1.43% — — — — 227.0 0.0 286.7 282.6
plain-rust 140,000 140,000 138,978 0.73% — — — — 0.0 0.0 573.4 569.2
plain-rust 350,000 350,000 348,815 0.34% — — — — 241.0 0.0 1433.6 1428.8
plain-rust 700,000 700,000 398,868 43.02% — — — — 0.0 0.0 2867.2 1633.8

1400B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% — — — — 122.0 842.0 784.0 784.0
native-dpdk 140,000 140,000 140,000 0.00% — — — — 145.0 516.0 1568.0 1568.0
native-dpdk 350,000 350,000 349,998 0.00% — — — — 133.0 2110.0 3920.0 3920.0
native-dpdk 700,000 475,188 460,415 3.11% — — — — 2528.0 5687.0 5322.1 5156.6
rust-dpdk 70,000 70,000 69,000 1.43% 0 43,182 0 0 0.0 0.0 784.0 772.8
rust-dpdk 140,000 140,000 139,000 0.71% 0 35,305 0 0 0.0 0.0 1568.0 1556.8
rust-dpdk 350,000 350,000 349,000 0.29% 0 31,470 0 0 0.0 0.0 3920.0 3908.8
rust-dpdk 700,000 475,764 466,142 2.02% 0 32,170 0 0 0.0 0.0 5328.6 5220.8
tokio-dpdk 70,000 70,000 69,000 1.43% 0 40,396 0 0 0.0 0.0 784.0 776.1
tokio-dpdk 140,000 140,000 139,000 0.71% 0 31,242 0 0 0.0 0.0 1568.0 1563.5
tokio-dpdk 350,000 350,000 150,532 56.99% 0 7,480 0 0 0.0 0.0 3920.0 1693.2
tokio-dpdk 700,000 476,267 160,320 66.34% 0 8,688 0 0 0.0 0.0 5334.2 1803.3
plain-rust 70,000 70,000 69,000 1.43% — — — — 216.0 0.0 784.0 772.8
plain-rust 140,000 140,000 138,990 0.72% — — — — 0.0 0.0 1568.0 1556.7
plain-rust 350,000 350,000 348,770 0.35% — — — — 0.0 0.0 3920.0 3906.2
plain-rust 700,000 475,846 447,288 6.00% — — — — 0.0 0.0 5329.5 5009.6

8500B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% — — — — 138.0 439.0 4760.0 4760.0
native-dpdk 140,000 78,293 78,266 0.03% — — — — 13847.0 16501.0 5323.9 5322.1
native-dpdk 350,000 78,285 77,960 0.41% — — — — 14172.0 14808.0 5323.4 5301.3
rust-dpdk 70,000 70,000 69,000 1.43% 0 32,009 0 0 0.0 0.0 4760.0 4692.0
rust-dpdk 140,000 78,340 77,724 0.79% 0 21,329 0 0 0.0 0.0 5327.1 5285.2
rust-dpdk 350,000 78,336 77,900 0.56% 0 6,676 0 0 0.0 0.0 5326.9 5297.2
tokio-dpdk 70,000 70,000 55,327 20.96% 0 4,331 0 0 0.0 0.0 4760.0 3764.9
tokio-dpdk 140,000 78,350 58,906 24.82% 0 14,217 0 0 0.0 0.0 5327.8 4008.4
tokio-dpdk 350,000 78,329 59,005 24.67% 0 5,001 0 0 0.0 0.0 5326.4 4015.2
plain-rust 70,000 70,000 38,549 44.93% — — — — 0.0 0.0 4760.0 2621.3
plain-rust 140,000 78,352 77,780 0.73% — — — — 0.0 0.0 5328.0 5289.1
plain-rust 350,000 78,286 77,905 0.49% — — — — 0.0 0.0 5323.4 5297.6

NIC Drops Instrumentation Self-Check

Compares (NIC-FINAL − NIC-BASELINE) one-shot snapshots against the sum of per-tick [PERF] deltas over the reporter's lifetime. These MUST match exactly — a mismatch means per-tick delta bookkeeping is losing data.

Config Status imissed (expected / actual / Δ) ierrors (expected / actual / Δ) rx_nombuf (expected / actual / Δ)
native-dpdk no instrumentation — — —
rust-dpdk OK 0 / 0 / 0 403,501 / 403,501 / 0 0 / 0 / 0
tokio-dpdk OK 0 / 0 / 0 267,830 / 267,830 / 0 0 / 0 / 0
plain-rust no instrumentation — — —

expected = FINAL − BASELINE (raw NIC counter delta across reporter lifetime). actual = sum of per-tick [PERF] delta fields. Any Δ ≠ 0 is a bug in the tick loop's bookkeeping.

plain-rust Kernel NIC Drops (ethtool -S delta)

ethtool snapshots not available — baseline or final file missing in $LOGS_DIR/dut-plain-rust-ethtool-*.txt.

Full results artifact

- Append Run #42 results (no regression) from GH Actions run 27476064107
- Mark roadmap item #15 complete (PR #79, already merged)
- Mark roadmap item #16 complete (PR #80)
@gspivey
gspivey merged commit 96fc713 into development Jun 13, 2026
5 checks passed
@github-actions

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: 1e86a402

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.3M 11.0M 1.0x 91
RX recv_from 64B 3.4M 4.6M 0.7x 216
TX send_to 1400B 1.6M 1.6M 1.0x 615
RX recv_from 1400B 1.1M 1.2M 0.9x 867

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 8.5M 8.4M 1.0x 118
RX recv_from (IPv6) 64B 3.9M 5.5M 0.7x 182
TX send_to (IPv6) 1400B 2.6M 2.6M 1.0x 379
RX recv_from (IPv6) 1400B 1.1M 1.1M 1.0x 874

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.3M 8.5M 1.32x
RX recv_from (sync) 64B 3.4M 3.9M 0.85x
TX send_to (sync) 1400B 1.6M 2.6M 0.62x
RX recv_from (sync) 1400B 1.1M 1.1M 0.97x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.5% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: 1e86a402

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.6M 11.3M 1.0x 88
RX recv_from 64B 3.7M 4.8M 0.8x 208
TX send_to 1400B 1.8M 1.8M 1.0x 550
RX recv_from 1400B 1.1M 1.2M 0.9x 823

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 9.0M 8.9M 1.0x 112
RX recv_from (IPv6) 64B 4.1M 5.7M 0.7x 176
TX send_to (IPv6) 1400B 2.9M 2.9M 1.0x 345
RX recv_from (IPv6) 1400B 1.1M 1.2M 0.9x 814

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.6M 9.0M 1.29x
RX recv_from (sync) 64B 3.7M 4.1M 0.89x
TX send_to (sync) 1400B 1.8M 2.9M 0.63x
RX recv_from (sync) 1400B 1.1M 1.1M 0.97x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.29x (OK)

OK: IPv6 is 29.1% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-04d6a176be9f27df6 (DPDK ENI: 10.0.1.102)
  • Receiver: i-0b021c4323dac6cd5 (DPDK ENI: 10.0.1.197)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0b9ec5a879d349d8b (DPDK ENI: 10.0.1.39)
  • Receiver: i-0e61d71015f7c4d69 (DPDK ENI: 10.0.1.119)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

1 similar comment
@github-actions

Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions

Copy link
Copy Markdown

✅ Integration Tests Passed (Run 27477428925)

Branch: 80/merge | Commit: 1e86a402

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.119:9000 (MAC: 02:9d:30:1f:c5:05)
echo listening on 10.0.1.119:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.39:9000 (MAC: 02:5f:f1:77:24:4f)
echo listening on 10.0.1.39:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.119:9000: 'arp-probe #1'
Test complete
[2026-06-13T20:13:20Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T20:13:20Z] INFO: Test: udp_send_receive
[2026-06-13T20:13:21Z] INFO: UDP send/receive succeeded
[2026-06-13T20:13:21Z] INFO: Test: echo_roundtrip
[2026-06-13T20:13:22Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T20:13:22Z] INFO: Test: payload_integrity
[2026-06-13T20:13:22Z] INFO: Response received, checking payload match...
[2026-06-13T20:13:22Z] INFO: Payload integrity verified (found in response)
[2026-06-13T20:13:22Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-13T20:13:22Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-06-13T20:17:48Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-06-13T20:17:48Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-06-13T20:17:48Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-06-13T20:17:48Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-06-13T20:16:49Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-06-13T20:16:49Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-06-13T20:16:49Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-06-13T20:16:49Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.119:9000 (MAC: 02:9d:30:1f:c5:05)
echo listening on 10.0.1.119:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.39:9000 (MAC: 02:5f:f1:77:24:4f)
echo listening on 10.0.1.39:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-06-13T20:08:47Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.119:9000
Bind address: 10.0.1.39:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.39:32768 (MAC: 02:5f:f1:77:24:4f)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.119:9000: 'arp-probe #1'
Test complete
[2026-06-13T20:08:48Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T20:08:48Z] INFO: Test: udp_send_receive
[2026-06-13T20:08:50Z] INFO: UDP send/receive succeeded
[2026-06-13T20:08:50Z] INFO: Test: echo_roundtrip
[2026-06-13T20:08:51Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T20:08:51Z] INFO: Test: payload_integrity
[2026-06-13T20:08:52Z] INFO: Response received, checking payload match...
[2026-06-13T20:08:52Z] INFO: Payload integrity verified (found in response)
[2026-06-13T20:08:52Z] INFO: Test: jumbo_diagnostics
[2026-06-13T20:08:52Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-06-13T20:08:52Z] INFO: Interface MTU:
  9001
  65536
[2026-06-13T20:08:52Z] INFO:   ens5: MTU=9001
[2026-06-13T20:08:52Z] INFO:   lo: MTU=65536
[2026-06-13T20:08:52Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-06-13T20:08:52Z] INFO: DPDK port config (from echo server log):
[2026-06-13T20:08:52Z] INFO:   (no MTU info in echo log)
[2026-06-13T20:08:52Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-06-13T20:08:52Z] INFO: Test: jumbo_echo_8000
[2026-06-13T20:08:54Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.119:9000
Bind address: 10.0.1.39:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.39:32768 (MAC: 02:5f:f1:77:24:4f)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.119:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.119:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.119:9000 (expected 8000, OK)
Test complete
[2026-06-13T20:08:54Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-06-13T20:08:54Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-06-13T20:08:54Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-06-13T20:13:20Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.119:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.119:9000: 'arp-probe #1'
Test complete
[2026-06-13T20:13:20Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T20:13:20Z] INFO: Test: udp_send_receive
[2026-06-13T20:13:21Z] INFO: UDP send/receive succeeded
[2026-06-13T20:13:21Z] INFO: Test: echo_roundtrip
[2026-06-13T20:13:22Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T20:13:22Z] INFO: Test: payload_integrity
[2026-06-13T20:13:22Z] INFO: Response received, checking payload match...
[2026-06-13T20:13:22Z] INFO: Payload integrity verified (found in response)
[2026-06-13T20:13:22Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-13T20:13:22Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-06-13T20:17:48Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-06-13T20:17:48Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-06-13T20:17:48Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-06-13T20:17:48Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-06-13T20:16:47Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-06-13T20:16:47Z] INFO: Local IP: 10.0.1.39, Peer IP: 10.0.1.119, Port: 9000
[2026-06-13T20:16:47Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.119       0x1         0x2         02:9d:30:1f:c5:05     *        ens5
10.0.1.39        0x1         0x2         02:5f:f1:77:24:4f     *        ens5
10.0.1.137       0x1         0x2         02:49:19:0d:4c:ad     *        ens5
10.0.1.1         0x1         0x2         02:93:42:cc:9a:a7     *        ens5
[2026-06-13T20:16:47Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-06-13T20:16:47Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-06-13T20:16:47Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1916464 Jun 13 20:05 /opt/dpdk-stdlib/target/release/test-client
[2026-06-13T20:16:47Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.119 --port 9000 --bind-ip 10.0.1.39 --count 10 --delay 200
[2026-06-13T20:16:49Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.119:9000
Bind address: 10.0.1.39:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.39:32768 (MAC: 02:5f:f1:77:24:4f)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.119:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-06-13T20:16:49Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-06-13T20:16:49Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-06-13T20:16:49Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-06-13T20:16:49Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0e61d71015f7c4d69 (receiver)
Command ID: c7bf3449-aa80-465d-938b-4db16e4d0994

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0e61d71015f7c4d69 (receiver)
Command ID: 5388d087-60ee-4076-8838-95e94b3b3eb7

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0e61d71015f7c4d69 (receiver)
Command ID: 3da8a724-f559-4bbc-b3f8-a3cf15eff13f

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0e61d71015f7c4d69 (receiver)
Command ID: 387bddda-372d-4ef5-865e-c0cd10045efc

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.39) ===
Status: InProgress
Instance: i-0b9ec5a879d349d8b (sender)
Command ID: 74b13fa8-baac-4f61-a749-17f31a0108a6

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0b9ec5a879d349d8b (sender)
Command ID: 4f8050ce-b74f-48ee-bcd2-08b86ff7ee27

=== STDOUT ===


=== STDERR ===


Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:2f:67:37:69:8b brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0d95aee7bd9093e04
    altname device-number-0.0
    inet 10.0.1.202/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2263sec preferred_lft 2263sec
    inet6 fe80::2f:67ff:fe37:698b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ad:57:35:85:6b brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-04634f33bb6ae4239
    altname device-number-0.0
    inet 10.0.1.217/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2284sec preferred_lft 2284sec
    inet6 fe80::ad:57ff:fe35:856b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-06-13T20:08:08Z
hostname: ip-10-0-1-202.ec2.internal
kernel: 6.18.33-63.124.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:2f:67:37:69:8b brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0d95aee7bd9093e04
    altname device-number-0.0
    inet 10.0.1.202/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3146sec preferred_lft 3146sec
    inet6 fe80::2f:67ff:fe37:698b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.114 dev ens5 lladdr 02:95:95:ed:26:c9 STALE 
10.0.1.137 dev ens5 lladdr 02:49:19:0d:4c:ad REACHABLE 
10.0.1.1 dev ens5 lladdr 02:93:42:cc:9a:a7 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.202 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.202 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.202 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.202 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:2f:67:37:69:8b/ 02:9d:30:1f:c5:05/ 

--- ENI: 02:2f:67:37:69:8b/ ---
  device-number: 0
  local-ipv4s: 10.0.1.202
  subnet-id: subnet-05b23de0de0f4564e
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:9d:30:1f:c5:05/ ---
  device-number: 1
  local-ipv4s: 10.0.1.119
  subnet-id: subnet-05b23de0de0f4564e
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:93:42:cc:9a:a7 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.202 ens5
Unicast reply from 10.0.1.1 [02:93:42:CC:9A:A7]  0.533ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jun 13 20:07 .
drwxr-xr-x. 14 root root     3100 Jun 13 20:07 ..
crw-------.  1 root root 243,   0 Jun 13 20:07 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jun 13 20:00 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.052749] printk: legacy console [ttyS0] enabled
[    0.053779] x2apic enabled
[    0.057996] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.058105] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.067851] audit: type=2000 audit(1781380828.438:1): state=initialized audit_enabled=0 res=1
[    0.087440] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.157526] ACPI: Interpreter enabled
[    0.157680] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.176072] pci 0000:00:05.0: enabling Extended Tags
[    0.228811] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.236948] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.244907] IPI shorthand broadcast: enabled
[    3.466122] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    3.504078] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    3.570974] VFIO - User Level meta-driver version: 0.3
[    4.003893] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    4.014910] ena 0000:00:05.0: ENA device version: 0.10
[    4.015654] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    4.114678] ena 0000:00:05.0: ENA Large LLQ is disabled
[    4.126822] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:2f:67:37:69:8b
[    4.182395] ena 0000:00:05.0 ens5: renamed from eth0
[  412.037094] pci 0000:00:06.0: enabling Extended Tags
[  412.041144] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  412.052887] ena 0000:00:06.0: ENA device version: 0.10
[  412.053642] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  412.152868] ena 0000:00:06.0: ENA Large LLQ is disabled
[  412.164849] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:9d:30:1f:c5:05
[  412.172241] ena 0000:00:06.0 ens6: renamed from eth0
[  447.306925] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  447.308362] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-06-13T20:08:01Z
hostname: ip-10-0-1-217.ec2.internal
kernel: 6.18.33-63.124.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ad:57:35:85:6b brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-04634f33bb6ae4239
    altname device-number-0.0
    inet 10.0.1.217/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3151sec preferred_lft 3151sec
    inet6 fe80::ad:57ff:fe35:856b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.137 dev ens5 lladdr 02:49:19:0d:4c:ad REACHABLE 
10.0.1.1 dev ens5 lladdr 02:93:42:cc:9a:a7 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.217 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.217 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.217 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.217 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:5f:f1:77:24:4f/ 02:ad:57:35:85:6b/ 

--- ENI: 02:5f:f1:77:24:4f/ ---
  device-number: 1
  local-ipv4s: 10.0.1.39
  subnet-id: subnet-05b23de0de0f4564e
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:ad:57:35:85:6b/ ---
  device-number: 0
  local-ipv4s: 10.0.1.217
  subnet-id: subnet-05b23de0de0f4564e
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:93:42:cc:9a:a7 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.217 ens5
Unicast reply from 10.0.1.1 [02:93:42:CC:9A:A7]  0.536ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jun 13 20:07 .
drwxr-xr-x. 14 root root     3100 Jun 13 20:06 ..
crw-------.  1 root root 243,   0 Jun 13 20:07 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jun 13 20:00 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.052043] printk: legacy console [ttyS0] enabled
[    0.053061] x2apic enabled
[    0.057264] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.057371] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.067118] audit: type=2000 audit(1781380827.004:1): state=initialized audit_enabled=0 res=1
[    0.086712] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.147860] ACPI: Interpreter enabled
[    0.147860] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.166677] pci 0000:00:05.0: enabling Extended Tags
[    0.239305] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.257362] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.260223] IPI shorthand broadcast: enabled
[    2.965178] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.996576] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    3.076021] VFIO - User Level meta-driver version: 0.3
[    3.479741] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    3.505911] ena 0000:00:05.0: ENA device version: 0.10
[    3.506636] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    3.614653] ena 0000:00:05.0: ENA Large LLQ is disabled
[    3.626728] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:ad:57:35:85:6b
[    3.706140] ena 0000:00:05.0 ens5: renamed from eth0
[  391.863005] pci 0000:00:06.0: enabling Extended Tags
[  391.867140] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  391.880072] ena 0000:00:06.0: ENA device version: 0.10
[  391.880855] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  391.982056] ena 0000:00:06.0: ENA Large LLQ is disabled
[  391.994717] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:5f:f1:77:24:4f
[  392.002286] ena 0000:00:06.0 ens6: renamed from eth0
[  442.077379] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  442.078870] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.067251] pid_max: default: 32768 minimum: 301
[    0.179741] iommu: Default domain type: Translated
[    0.179741] NetLabel:  unlabeled traffic allowed by default
[    0.201235] PCI: CLS 0 bytes, default 64
[    0.260032] nvme nvme0: 2/0/0 default/read/poll queues
[    0.464297] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.464414] systemd[1]: No hostname configured, using default hostname.
[    0.557392] systemd[1]: Queued start job for default target initrd.target.
[    2.781348] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  483.615741] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8018)
[  756.324467] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8551)
[ 1205.822216] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9450)

sender-dmesg-crashes.log

[    0.066527] pid_max: default: 32768 minimum: 301
[    0.170488] iommu: Default domain type: Translated
[    0.170511] NetLabel:  unlabeled traffic allowed by default
[    0.206516] PCI: CLS 0 bytes, default 64
[    0.274995] nvme nvme0: 2/0/0 default/read/poll queues
[    0.418537] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.418658] systemd[1]: No hostname configured, using default hostname.
[    0.494614] systemd[1]: Queued start job for default target initrd.target.
[    2.311330] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 1024.024014] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9290)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  483.614171] vfio-pci 0000:00:06.0: reset done
[  483.615741] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8018)
[  483.617179] vfio-pci 0000:00:06.0: resetting
[  483.834039] vfio-pci 0000:00:06.0: reset done
[  563.370854] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  564.382201] ena 0000:00:06.0: ENA device version: 0.10
[  564.382922] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  564.482177] ena 0000:00:06.0: ENA Large LLQ is disabled
[  564.494101] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:9d:30:1f:c5:05
[  564.504973] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[  584.135957] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  584.137176] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  756.107933] vfio-pci 0000:00:06.0: resetting
[  756.322900] vfio-pci 0000:00:06.0: reset done
[  756.324467] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8551)
[  756.325889] vfio-pci 0000:00:06.0: resetting
[  756.542784] vfio-pci 0000:00:06.0: reset done
[  830.425505] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  831.438473] ena 0000:00:06.0: ENA device version: 0.10
[  831.439238] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  831.540402] ena 0000:00:06.0: ENA Large LLQ is disabled
[  831.552820] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:9d:30:1f:c5:05
[  831.560785] ena 0000:00:06.0 ens6: renamed from eth0
[ 1120.992033] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1120.993473] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1205.593709] vfio-pci 0000:00:06.0: resetting
[ 1205.820658] vfio-pci 0000:00:06.0: reset done
[ 1205.822216] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9450)
[ 1205.823650] vfio-pci 0000:00:06.0: resetting
[ 1206.040570] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[ 1024.025449] vfio-pci 0000:00:06.0: resetting
[ 1024.242324] vfio-pci 0000:00:06.0: reset done
[ 1085.459071] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1086.470587] ena 0000:00:06.0: ENA device version: 0.10
[ 1086.471353] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1086.570572] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1086.582462] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:5f:f1:77:24:4f
[ 1086.593907] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1115.785856] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1115.787286] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1222.913433] vfio-pci 0000:00:06.0: resetting
[ 1223.121702] vfio-pci 0000:00:06.0: reset done
[ 1223.123246] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9732)
[ 1223.124838] vfio-pci 0000:00:06.0: resetting
[ 1223.341425] vfio-pci 0000:00:06.0: reset done
[ 1223.629685] vfio-pci 0000:00:06.0: resetting
[ 1223.841559] vfio-pci 0000:00:06.0: reset done
[ 1223.843116] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9752)
[ 1223.844719] vfio-pci 0000:00:06.0: resetting
[ 1224.061489] vfio-pci 0000:00:06.0: reset done
[ 1225.340951] vfio-pci 0000:00:06.0: resetting
[ 1225.561574] vfio-pci 0000:00:06.0: reset done
[ 1225.563133] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9775)
[ 1225.564711] vfio-pci 0000:00:06.0: resetting
[ 1225.781546] vfio-pci 0000:00:06.0: reset done
[ 1227.061033] vfio-pci 0000:00:06.0: resetting
[ 1227.271570] vfio-pci 0000:00:06.0: reset done
[ 1227.273134] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9798)
[ 1227.274721] vfio-pci 0000:00:06.0: resetting
[ 1227.491488] vfio-pci 0000:00:06.0: reset done

@github-actions

Copy link
Copy Markdown

✅ Integration Tests Passed — Graviton (run)

Branch: 80/merge | Commit: 1e86a402

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.197:9000 (MAC: 02:fb:00:41:17:89)
echo listening on 10.0.1.197:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.102:9000 (MAC: 02:4c:c5:56:22:1f)
echo listening on 10.0.1.102:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.197:9000: 'arp-probe #1'
Test complete
[2026-06-13T20:11:23Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-13T20:11:23Z] INFO: Test: udp_send_receive
[2026-06-13T20:11:24Z] INFO: UDP send/receive succeeded
[2026-06-13T20:11:24Z] INFO: Test: echo_roundtrip
[2026-06-13T20:11:25Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-13T20:11:25Z] INFO: Test: payload_integrity
[2026-06-13T20:11:25Z] INFO: Response received, checking payload match...
[2026-06-13T20:11:25Z] INFO: Payload integrity verified (found in response)
[2026-06-13T20:11:25Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-13T20:11:25Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant