Skip to content

docs: mark ROADMAP #35 complete (PR #117) - #119

Merged
gspivey merged 2 commits into
developmentfrom
agent/mark-roadmap-35-complete
Aug 17, 2026
Merged

gspivey merged 2 commits into
developmentfrom
agent/mark-roadmap-35-complete

Conversation

@gspivey

@gspivey gspivey commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Roadmap Item

ROADMAP #35: dpdk-stdlib-tcp: EC2 tier-2 and tier-3 integration test scripts

Spec: .kiro/specs/tcp-support/ tasks 15.2, 15.3

Summary

PR #117 implemented and merged the tier-2 and tier-3 TCP integration test scripts but did not update ROADMAP.md or tasks.md. This PR finalizes the bookkeeping:

Tests

No code changes — docs-only. All 1041 existing tests pass locally (cargo build && cargo test).

ROADMAP item #35 (EC2 tier-2 and tier-3 integration test scripts) was
implemented and merged in PR #117 but the ROADMAP.md and tasks.md were
not updated. This commit finalizes:

- ROADMAP.md: mark item #35 as [x] Complete · PR: #117
- tasks.md: check off tasks 15.2 and 15.3
@github-actions

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: 3f929686

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.4M 10.9M 1.0x 91
RX recv_from 64B 3.4M 4.4M 0.8x 226
TX send_to 1400B 1.6M 1.6M 1.0x 611
RX recv_from 1400B 1.0M 1.1M 0.9x 924

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 8.7M 8.2M 1.1x 121
RX recv_from (IPv6) 64B 3.9M 5.4M 0.7x 186
TX send_to (IPv6) 1400B 2.7M 2.7M 1.0x 376
RX recv_from (IPv6) 1400B 1.0M 1.1M 0.9x 937

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.4M 8.7M 1.32x
RX recv_from (sync) 64B 3.4M 3.9M 0.87x
TX send_to (sync) 1400B 1.6M 2.7M 0.61x
RX recv_from (sync) 1400B 1.0M 1.0M 1.00x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.2% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: 3f929686

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 12.0M 11.6M 1.0x 86
RX recv_from 64B 3.7M 4.7M 0.8x 214
TX send_to 1400B 1.9M 1.9M 1.0x 526
RX recv_from 1400B 1.1M 1.2M 0.9x 811

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 9.1M 9.0M 1.0x 111
RX recv_from (IPv6) 64B 4.1M 5.6M 0.7x 178
TX send_to (IPv6) 1400B 2.9M 2.9M 1.0x 341
RX recv_from (IPv6) 1400B 1.1M 1.2M 0.9x 805

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 12.0M 9.1M 1.31x
RX recv_from (sync) 64B 3.7M 4.1M 0.91x
TX send_to (sync) 1400B 1.9M 2.9M 0.64x
RX recv_from (sync) 1400B 1.1M 1.1M 0.99x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.31x (OK)

OK: IPv6 is 31.2% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-01f4e3ccc0086cdca (DPDK ENI: 10.0.1.131)
  • Receiver: i-0eb39614777338c6c (DPDK ENI: 10.0.1.124)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

Integration Test Failure — Graviton (Run 32034020617)

Branch: 119/merge | Commit: 3f92968691f66e70a61794fd6a215f5e3ac0c198

failure-summary.json

{
  "failed_step": "deploy_infrastructure",
  "error": "Infrastructure deployment failed",
  "exit_code": 2,
  "timestamp": "2026-08-17T13:26:53.467023Z",
  "sender_instance_id": "i-05f3d446438e3902f",
  "receiver_instance_id": "i-035a3b36f8d46174b",
  "commit": "3f92968691f66e70a61794fd6a215f5e3ac0c198",
  "run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/32034020617"
}```

### receiver-console-output.log (62978 bytes, last 80 lines)

[ 3930.414038] schedule_preempt_disabled+0x18/0x24
[ 3930.414655] __mutex_lock.constprop.0+0x2cc/0x91c
[ 3930.415301] __mutex_lock_slowpath+0x1c/0x28
[ 3930.415877] mutex_lock+0x60/0x78
[ 3930.416324] uevent_show+0x90/0x140
[ 3930.416789] dev_attr_show+0x28/0x70
[ 3930.417280] sysfs_kf_seq_show+0x90/0x140
[ 3930.417814] kernfs_seq_show+0x34/0x40
[ 3930.418329] seq_read_iter+0x1dc/0x4ec
[ 3930.418818] kernfs_fop_read_iter+0x40/0x4c
[ 3930.419393] vfs_read+0x210/0x2c0
[ 3930.419843] ksys_read+0x74/0x104
[ 3930.420284] __arm64_sys_read+0x24/0x3c
[ 3930.420802] invoke_syscall+0x4c/0x100
[ 3930.421315] el0_svc_common.constprop.0+0x164/0x188
[ 3930.421962] do_el0_svc+0x3c/0x98
[ 3930.422417] el0_svc+0x34/0x108
[ 3930.422831] el0t_64_sync_handler+0xbc/0x140
[ 4015.516191] vfio-pci 0000:00:06.0: Relaying device request to user (#110)
[ 4053.275294] INFO: task systemd-journal:841 blocked for more than 1105 seconds.
[ 4053.276185] Tainted: G U 6.1.177-224.371.amzn2023.aarch64 #1
[ 4053.277202] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4053.278262] task:systemd-journal state:D stack:0 pid:841 ppid:1 flags:0x00800a04
[ 4053.279396] Call trace:
[ 4053.279713] __switch_to+0xe0/0x140
[ 4053.280192] __schedule+0x268/0x700
[ 4053.280655] schedule+0x58/0xf4
[ 4053.281088] schedule_preempt_disabled+0x18/0x24
[ 4053.281713] __mutex_lock.constprop.0+0x2cc/0x91c
[ 4053.282338] __mutex_lock_slowpath+0x1c/0x28
[ 4053.282924] mutex_lock+0x60/0x78
[ 4053.283359] uevent_show+0x90/0x140
[ 4053.283825] dev_attr_show+0x28/0x70
[ 4053.284314] sysfs_kf_seq_show+0x90/0x140
[ 4053.284864] kernfs_seq_show+0x34/0x40
[ 4053.285380] seq_read_iter+0x1dc/0x4ec
[ 4053.285866] kernfs_fop_read_iter+0x40/0x4c
[ 4053.286429] vfs_read+0x210/0x2c0
[ 4053.286881] ksys_read+0x74/0x104
[ 4053.287331] __arm64_sys_read+0x24/0x3c
[ 4053.287841] invoke_syscall+0x4c/0x100
[ 4053.288352] el0_svc_common.constprop.0+0x164/0x188
[ 4053.289006] do_el0_svc+0x3c/0x98
[ 4053.289453] el0_svc+0x34/0x108
[ 4053.289871] el0t_64_sync_handler+0xbc/0x140
[ 4117.923721] vfio-pci 0000:00:06.0: Relaying device request to user (#120)
[ 4176.152333] INFO: task systemd-journal:841 blocked for more than 1228 seconds.
[ 4176.153227] Tainted: G U 6.1.177-224.371.amzn2023.aarch64 #1
[ 4176.154254] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4176.155308] task:systemd-journal state:D stack:0 pid:841 ppid:1 flags:0x00800a04
[ 4176.156431] Call trace:
[ 4176.156763] __switch_to+0xe0/0x140
[ 4176.157233] __schedule+0x268/0x700
[ 4176.157700] schedule+0x58/0xf4
[ 4176.158129] schedule_preempt_disabled+0x18/0x24
[ 4176.158750] __mutex_lock.constprop.0+0x2cc/0x91c
[ 4176.159382] __mutex_lock_slowpath+0x1c/0x28
[ 4176.159961] mutex_lock+0x60/0x78
[ 4176.160400] uevent_show+0x90/0x140
[ 4176.160870] dev_attr_show+0x28/0x70
[ 4176.161360] sysfs_kf_seq_show+0x90/0x140
[ 4176.161898] kernfs_seq_show+0x34/0x40
[ 4176.162409] seq_read_iter+0x1dc/0x4ec
[ 4176.162911] kernfs_fop_read_iter+0x40/0x4c
[ 4176.163484] vfs_read+0x210/0x2c0
[ 4176.163924] ksys_read+0x74/0x104
[ 4176.164377] __arm64_sys_read+0x24/0x3c
[ 4176.164891] invoke_syscall+0x4c/0x100
[ 4176.165399] el0_svc_common.constprop.0+0x164/0x188
[ 4176.166047] do_el0_svc+0x3c/0x98
[ 4176.166504] el0_svc+0x34/0x108
[ 4176.166919] el0t_64_sync_handler+0xbc/0x140
[ 4220.311263] vfio-pci 0000:00:06.0: Relaying device request to user (#130)
[ 4322.718789] vfio-pci 0000:00:06.0: Relaying device request to user (#140)
[ 4425.106309] vfio-pci 0000:00:06.0: Relaying device request to user (#150)
[ 4527.503834] vfio-pci 0000:00:06.0: Relaying device request to user (#160)
[ 4629.901360] vfio-pci 0000:00:06.0: Relaying device request to user (#170)
[ 4732.298873] vfio-pci 0000:00:06.0: Relaying device request to user (#180)
[ 4834.696393] vfio-pci 0000:00:06.0: Relaying device request to user (#190)
[ 4870.485544] vfio-pci 0000:00:06.0: Removing from iommu group 0


### sender-console-output.log (54151 bytes, last 80 lines)

[ 993.727757] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 993.728838] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 993.740841] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:14:b2:1d:07:8d
[ 993.787918] ena 0000:00:06.0 ens6: renamed from eth0
[ 993.830490] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 993.832355] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 993.912364] zram_generator::config[19866]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1021.762449] zram_generator::config[20614]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1022.682254] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1022.683068] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1140.031386] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22437)
[ 1140.751423] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22456)
[ 1142.481430] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22480)
[ 1144.221437] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22503)
[ 1187.612399] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1188.620231] ena 0000:00:06.0: ENA device version: 0.10
[ 1188.620812] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1188.720081] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 1188.721161] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 1188.732754] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:14:b2:1d:07:8d
[ 1188.760243] ena 0000:00:06.0 ens6: renamed from eth0
[ 1188.802881] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1188.804780] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1188.884409] zram_generator::config[23328]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1229.821743] zram_generator::config[24103]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1230.704984] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1230.705757] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1289.583230] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:25191)
[ 1290.323173] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:25210)
[ 1291.063229] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:25229)
[ 1390.454490] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:26809)
[ 1450.695235] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:27908)
[ 1584.796765] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:30061)
[ 1645.037482] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:30904)
[ 1705.278187] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:31991)
[ 1834.239792] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:34175)
[ 1894.230164] netem: version 1.3
[ 1895.490526] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:35034)
[ 2057.142526] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:37227)
[ 2147.383486] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:38869)
[ 2267.624934] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:40725)
[ 2375.226232] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:42408)
[ 2569.348550] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:45591)
[ 2629.589310] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:46684)
[ 2689.829982] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:47521)
[ 2748.010716] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:48615)
[ 2889.349151] vfio-pci 0000:00:06.0: Unable to change power state from D3cold to D0, device inaccessible
[ 2889.350322] vfio-pci 0000:00:06.0: Unable to change power state from D3cold to D0, device inaccessible
[ 2889.351631] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 2907.463297] kauditd_printk_skb: 118 callbacks suppressed
[ 2907.463300] audit: type=1305 audit(1786918453.361:2193): op=set audit_pid=0 old=1330 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 2907.466514] audit: type=1131 audit(1786918453.361:2194): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.469261] audit: type=1131 audit(1786918453.361:2195): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.525812] audit: type=1131 audit(1786918453.421:2196): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.528715] audit: type=1131 audit(1786918453.421:2197): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.531708] audit: type=1131 audit(1786918453.421:2198): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.534789] audit: type=1130 audit(1786918453.421:2199): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.537743] audit: type=1131 audit(1786918453.421:2200): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2907.540732] audit: type=1334 audit(1786918453.421:2201): prog-id=445 op=UNLOAD
[ 2907.541691] audit: type=1334 audit(1786918453.421:2202): prog-id=444 op=UNLOAD
[ 2907.700618] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 2907.714122] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 2907.717032] systemd-journald[840]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 2907.724151] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 2907.726906] systemd-shutdown[1]: Unmounting file systems.
[ 2907.728076] (sd-remount)[51066]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota'.
[ 2907.729812] XFS: attr2 mount option is deprecated.
[ 2907.750053] systemd-shutdown[1]: All filesystems unmounted.
[ 2907.750791] systemd-shutdown[1]: Deactivating swaps.
[ 2907.751426] systemd-shutdown[1]: All swaps deactivated.
[ 2907.752085] systemd-shutdown[1]: Detaching loop devices.
[ 2907.753176] systemd-shutdown[1]: All loop devices detached.
[ 2907.753823] systemd-shutdown[1]: Stopping MD devices.
[ 2907.754532] systemd-shutdown[1]: All MD devices stopped.
[ 2907.755215] systemd-shutdown[1]: Detaching DM devices.
[ 2907.755935] systemd-shutdown[1]: All DM devices detached.
[ 2907.756643] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[ 2907.758851] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 2907.759640] systemd-shutdown[1]: Powering off.
[ 2908.031105] reboot: Power down


@github-actions

Copy link
Copy Markdown

❌ Integration Tests Failed — Graviton (run)

Branch: 119/merge | Commit: 3f929686

Test Results

Application Logs (last 20 lines)

@github-actions

Copy link
Copy Markdown

Integration Tests — ⚠️ 3 infra flakes · 0 real failures

3f92968 · run · Integration Tests · 16 tests: 13 ✅ · 0 ❌ real · 3 ⚠️ infra

Tier Test Result Time Reason
tier1-dpdk-echo arp_resolution ✅ pass 1.011s
tier1-dpdk-echo udp_send_receive ✅ pass 1.753s
tier1-dpdk-echo echo_roundtrip ✅ pass 1.551s
tier1-dpdk-echo payload_integrity ✅ pass 0.758s
tier1-dpdk-echo jumbo_diagnostics ✅ pass 0.041s
tier1-dpdk-echo jumbo_echo_8000 ✅ pass 1.780s
tier1-tcp-echo-async execution ⚠️ infra 0.000s ENI bind failed on receiver instance
tier1-tcp-echo bidir_echo ✅ pass 0.763s
tier1-tcp-echo bidir_multi ✅ pass 0.760s
tier2-kernel-interop arp_resolution ✅ pass 0.074s
tier2-kernel-interop udp_send_receive ✅ pass 1.075s
tier2-kernel-interop echo_roundtrip ✅ pass 0.869s
tier2-kernel-interop payload_integrity ✅ pass 0.068s
tier2-tcp-echo execution ⚠️ infra 0.000s ENI bind failed on receiver instance
tier3-iperf-interop our_app_sends ✅ pass 2.571s
tier4-jumbo-echo execution ⚠️ infra 0.000s ENI bind failed on receiver instance

⚠️ Infra flakes (3) — SSM/ENI setup, not code (task #10)

  • tier1-tcp-echo-async / execution — ENI bind failed on receiver instance
  • tier2-tcp-echo / execution — ENI bind failed on receiver instance
  • tier4-jumbo-echo / execution — ENI bind failed on receiver instance

@github-actions

Copy link
Copy Markdown

❌ Integration Tests Failed (Run 32034020600)

Branch: 119/merge | Commit: 3f929686

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier1-tcp-echo-async.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier1-tcp-echo.xml: 2 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier2-tcp-echo.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped
  • tier4-jumbo-echo.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    

Application Logs (last 20 lines)

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.131:9000 (MAC: 02:b7:42:f2:36:9f)
echo listening on 10.0.1.131:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.124:9000: 'arp-probe #1'
Test complete
[2026-08-17T13:25:55Z] INFO: ARP resolution succeeded (got response from peer)
[2026-08-17T13:25:55Z] INFO: Test: udp_send_receive
[2026-08-17T13:25:56Z] INFO: UDP send/receive succeeded
[2026-08-17T13:25:56Z] INFO: Test: echo_roundtrip
[2026-08-17T13:25:57Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-08-17T13:25:57Z] INFO: Test: payload_integrity
[2026-08-17T13:25:57Z] INFO: Response received, checking payload match...
[2026-08-17T13:25:57Z] INFO: Payload integrity verified (found in response)
[2026-08-17T13:25:57Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-08-17T13:25:57Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-08-17T13:32:20Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-08-17T13:32:20Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-08-17T13:32:20Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-08-17T13:32:20Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.131:9000 (MAC: 02:b7:42:f2:36:9f)
echo listening on 10.0.1.131:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-08-17T13:21:58Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.124:9000
Bind address: 10.0.1.131:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.131:32768 (MAC: 02:b7:42:f2:36:9f)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.124:9000: 'arp-probe #1'
Test complete
[2026-08-17T13:21:59Z] INFO: ARP resolution succeeded (got response from peer)
[2026-08-17T13:21:59Z] INFO: Test: udp_send_receive
[2026-08-17T13:22:01Z] INFO: UDP send/receive succeeded
[2026-08-17T13:22:01Z] INFO: Test: echo_roundtrip
[2026-08-17T13:22:03Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-08-17T13:22:03Z] INFO: Test: payload_integrity
[2026-08-17T13:22:03Z] INFO: Response received, checking payload match...
[2026-08-17T13:22:03Z] INFO: Payload integrity verified (found in response)
[2026-08-17T13:22:04Z] INFO: Test: jumbo_diagnostics
[2026-08-17T13:22:04Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-08-17T13:22:04Z] INFO: Interface MTU:
  1500
  9001
  65536
[2026-08-17T13:22:04Z] INFO:   docker0: MTU=1500
[2026-08-17T13:22:04Z] INFO:   ens5: MTU=9001
[2026-08-17T13:22:04Z] INFO:   lo: MTU=65536
[2026-08-17T13:22:04Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-08-17T13:22:04Z] INFO: DPDK port config (from echo server log):
[2026-08-17T13:22:04Z] INFO:   (no MTU info in echo log)
[2026-08-17T13:22:04Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-08-17T13:22:04Z] INFO: Test: jumbo_echo_8000
[2026-08-17T13:22:05Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.124:9000
Bind address: 10.0.1.131:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.131:32768 (MAC: 02:b7:42:f2:36:9f)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.124:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.124:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.124:9000 (expected 8000, OK)
Test complete
[2026-08-17T13:22:05Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-08-17T13:22:05Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-08-17T13:22:05Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-08-17T13:25:55Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.124:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.124:9000: 'arp-probe #1'
Test complete
[2026-08-17T13:25:55Z] INFO: ARP resolution succeeded (got response from peer)
[2026-08-17T13:25:55Z] INFO: Test: udp_send_receive
[2026-08-17T13:25:56Z] INFO: UDP send/receive succeeded
[2026-08-17T13:25:56Z] INFO: Test: echo_roundtrip
[2026-08-17T13:25:57Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-08-17T13:25:57Z] INFO: Test: payload_integrity
[2026-08-17T13:25:57Z] INFO: Response received, checking payload match...
[2026-08-17T13:25:57Z] INFO: Payload integrity verified (found in response)
[2026-08-17T13:25:57Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-08-17T13:25:57Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

sender-test-client-iperf.log

[2026-08-17T13:32:17Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-08-17T13:32:17Z] INFO: Local IP: 10.0.1.131, Peer IP: 10.0.1.124, Port: 9000
[2026-08-17T13:32:17Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.129       0x1         0x2         02:4b:f6:88:ae:63     *        ens5
10.0.1.70        0x1         0x2         02:d5:7e:cc:b4:03     *        ens5
10.0.1.131       0x1         0x2         02:b7:42:f2:36:9f     *        ens5
10.0.1.1         0x1         0x2         02:b7:4f:42:fe:91     *        ens5
10.0.1.124       0x1         0x2         02:27:25:70:a6:a7     *        ens5
[2026-08-17T13:32:17Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-08-17T13:32:17Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-08-17T13:32:17Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 2009272 Aug 16 18:29 /opt/dpdk-stdlib/target/release/test-client
[2026-08-17T13:32:17Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.124 --port 9000 --bind-ip 10.0.1.131 --count 10 --delay 200
[2026-08-17T13:32:20Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.124:9000
Bind address: 10.0.1.131:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.131:32768 (MAC: 02:b7:42:f2:36:9f)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.124:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-08-17T13:32:20Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-08-17T13:32:20Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-08-17T13:32:20Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-08-17T13:32:20Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 34a31c40-4ed5-4a3f-87ce-88cdcddc54e2

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 4011ce34-f19f-4852-99b0-f764eebaf240

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 66c455ec-dde2-41c4-931f-4c9e719f347c

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action bind) ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 5ede902c-617f-483e-aa20-2ab20042dff6

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 8b4f484f-140f-4e2d-bb27-943faa85b95b

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: pkill -f 'target/release/echo' 2>/dev/null || true; pkill -f 'target/release/test-client' 2>/dev/null || true; pkill -f 'target/release/tokio-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tokio-tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-test-client' 2>/dev/null || true; pkill -f 'target/release/tcp-kernel-client' 2>/dev/null || true; pkill -f 'target/release/plain-tcp-echo' 2>/dev/null || true; pkill -f 'python3.*udp_echo' 2>/dev/null || true; sleep 2; rm -rf /var/run/dpdk/ 2>/dev/null || true; cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.124) ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 62272c9a-ba94-4cb4-b2e0-7d8cb630b245

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action bind) ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 8f45db98-2303-4600-813d-db9b61325cd1

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: pkill -f 'target/release/echo' 2>/dev/null || true; pkill -f 'target/release/test-client' 2>/dev/null || true; pkill -f 'target/release/tokio-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tokio-tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-test-client' 2>/dev/null || true; pkill -f 'target/release/tcp-kernel-client' 2>/dev/null || true; pkill -f 'target/release/plain-tcp-echo' 2>/dev/null || true; pkill -f 'python3.*udp_echo' 2>/dev/null || true; sleep 2; rm -rf /var/run/dpdk/ 2>/dev/null || true; cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.124) ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 12866445-e690-4d08-8c66-c066bf92e7e3

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action bind) ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 6ab63c1c-0ec7-4127-af9a-8343bbbb7d7a

=== STDOUT ===


=== STDERR ===


=== Polling timeout 60s (SSM timeout: 30s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action status) ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: b576bb11-9e54-4977-a963-c5b0bb13caaa

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 60s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 711bf611-6545-4b4f-83b9-73ea357b053d

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 60s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: ef05d251-ba45-48e3-84ea-0f91fbc44822

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 60s ===
Status: InProgress
Instance: i-0eb39614777338c6c (receiver)
Command ID: 09451421-71d7-4d50-9b0e-c266d8812520

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-01f4e3ccc0086cdca (sender)
Command ID: 5e583cb3-c350-4446-b5ae-b736e7a5331b

=== STDOUT ===


=== STDERR ===


Network & PCI State

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:d1:8a:53:56:05 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0527d0e5fc6675ef5
    altname device-number-0.0
    inet 10.0.1.243/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3281sec preferred_lft 3281sec
    inet6 fe80::d1:8aff:fe53:5605/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:9a:99:0c:ed brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
9: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:b7:42:f2:36:9f brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname eni-0f503110c384a1eb9
    altname device-number-1.0
    inet 10.0.1.131/24 metric 513 brd 10.0.1.255 scope global dynamic ens6
       valid_lft 3120sec preferred_lft 3120sec
    inet6 fe80::b7:42ff:fef2:369f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-08-17T13:21:20Z
hostname: ip-10-0-1-56.ec2.internal
kernel: 6.1.177-224.371.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:c2:07:3b:2f:5d brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0164074f9a403b7a5
    altname device-number-0.0
    inet 10.0.1.56/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3350sec preferred_lft 3350sec
    inet6 fe80::c2:7ff:fe3b:2f5d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:1c:b9:12:1e brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:b7:4f:42:fe:91 REACHABLE 
10.0.1.129 dev ens5 lladdr 02:4b:f6:88:ae:63 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.56 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.56 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.56 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.56 metric 512 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:27:25:70:a6:a7/ 02:c2:07:3b:2f:5d/ 

--- ENI: 02:27:25:70:a6:a7/ ---
  device-number: 1
  local-ipv4s: 10.0.1.124
  subnet-id: subnet-0fd08bbfbf6c0a69b
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:c2:07:3b:2f:5d/ ---
  device-number: 0
  local-ipv4s: 10.0.1.56
  subnet-id: subnet-0fd08bbfbf6c0a69b
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:b7:4f:42:fe:91 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.56 ens5
Unicast reply from 10.0.1.1 [02:B7:4F:42:FE:91]  0.540ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Aug 17 13:19 .
drwxr-xr-x. 14 root root     3100 Aug 17 13:17 ..
crw-------.  1 root root 243,   0 Aug 17 13:19 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Aug 17 13:17 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.059582] x2apic enabled
[    0.066495] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.073024] audit: type=2000 audit(1786972619.338:1): state=initialized audit_enabled=0 res=1
[    0.075436] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.098900] ACPI: Interpreter enabled
[    0.098900] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.123882] pci 0000:00:05.0: enabling Extended Tags
[    0.132336] pci 0000:00:06.0: enabling Extended Tags
[    0.164392] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.194873] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.277507] IPI shorthand broadcast: enabled
[    0.277570] AES CTR mode by8 optimization enabled
[    2.314818] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.354508] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    2.415413] VFIO - User Level meta-driver version: 0.3
[    3.272148] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.2g
[    3.283339] ena 0000:00:05.0: ENA device version: 0.10
[    3.284034] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    3.394720] ena 0000:00:05.0: ENA Large LLQ is disabled
[    3.407068] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0604000, mac addr 02:c2:07:3b:2f:5d
[    3.421405] ena 0000:00:06.0: ENA device version: 0.10
[    3.422189] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[    3.534302] ena 0000:00:06.0: ENA Large LLQ is disabled
[    3.546815] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:27:25:70:a6:a7
[    3.655986] ena 0000:00:05.0 ens5: renamed from eth0
[    3.724880] ena 0000:00:06.0 ens6: renamed from eth1
[    4.959562] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[    4.967260] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels
[  147.950186] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  147.951677] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-08-17T13:21:12Z
hostname: ip-10-0-1-243.ec2.internal
kernel: 6.1.177-224.371.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:d1:8a:53:56:05 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0527d0e5fc6675ef5
    altname device-number-0.0
    inet 10.0.1.243/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3369sec preferred_lft 3369sec
    inet6 fe80::d1:8aff:fe53:5605/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:9a:99:0c:ed brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.129 dev ens5 lladdr 02:4b:f6:88:ae:63 REACHABLE 
10.0.1.70 dev ens5 lladdr 02:d5:7e:cc:b4:03 REACHABLE 
10.0.1.1 dev ens5 lladdr 02:b7:4f:42:fe:91 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.243 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.243 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.243 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.243 metric 512 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:b7:42:f2:36:9f/ 02:d1:8a:53:56:05/ 

--- ENI: 02:b7:42:f2:36:9f/ ---
  device-number: 1
  local-ipv4s: 10.0.1.131
  subnet-id: subnet-0fd08bbfbf6c0a69b
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:d1:8a:53:56:05/ ---
  device-number: 0
  local-ipv4s: 10.0.1.243
  subnet-id: subnet-0fd08bbfbf6c0a69b
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:b7:4f:42:fe:91 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.243 ens5
Unicast reply from 10.0.1.1 [02:B7:4F:42:FE:91]  0.537ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Aug 17 13:17 .
drwxr-xr-x. 14 root root     3100 Aug 17 13:17 ..
crw-------.  1 root root 243,   0 Aug 17 13:17 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Aug 17 13:17 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.059362] x2apic enabled
[    0.066195] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.072775] audit: type=2000 audit(1786972630.462:1): state=initialized audit_enabled=0 res=1
[    0.074979] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.098376] ACPI: Interpreter enabled
[    0.098376] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.123622] pci 0000:00:05.0: enabling Extended Tags
[    0.132117] pci 0000:00:06.0: enabling Extended Tags
[    0.163793] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.194439] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.282777] IPI shorthand broadcast: enabled
[    0.282843] AES CTR mode by8 optimization enabled
[    2.544803] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.585899] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    2.613753] VFIO - User Level meta-driver version: 0.3
[    3.634891] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.2g
[    3.644872] ena 0000:00:05.0: ENA device version: 0.10
[    3.645558] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    3.764554] ena 0000:00:05.0: ENA Large LLQ is disabled
[    3.777402] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0604000, mac addr 02:d1:8a:53:56:05
[    3.791818] ena 0000:00:06.0: ENA device version: 0.10
[    3.792548] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[    3.896564] ena 0000:00:06.0: ENA Large LLQ is disabled
[    3.907690] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[    3.954712] ena 0000:00:06.0 ens6: renamed from eth1
[    4.074728] ena 0000:00:05.0 ens5: renamed from eth0
[    5.271749] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[    5.283697] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels
[   36.108885] vfio-pci 0000:00:06.0: Adding to iommu group 0
[   36.110323] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

sender-dmesg-crashes.log

[    0.072775] pid_max: default: 32768 minimum: 301
[    0.134513] iommu: Default domain type: Translated 
[    0.134996] NetLabel:  unlabeled traffic allowed by default
[    0.142576] PCI: CLS 0 bytes, default 64
[    0.216199] nvme nvme0: 2/0/0 default/read/poll queues
[    0.447933] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.448029] systemd[1]: No hostname configured, using default hostname.
[    0.558358] systemd[1]: Queued start job for default target initrd.target.
[    1.760471] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    8.300235] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[  951.253447] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:19069)
[ 1274.972119] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24840)
[ 1275.731983] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24860)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  365.374354] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[  384.902377] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  384.903668] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  520.238519] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:11794)
[  595.170337] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  596.183644] ena 0000:00:06.0: ENA device version: 0.10
[  596.184402] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  596.284671] ena 0000:00:06.0: ENA Large LLQ is disabled
[  596.297321] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:27:25:70:a6:a7
[  596.307170] ena 0000:00:06.0 eth0: Local page cache is disabled for less than 16 channels
[ 1060.891440] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1060.892872] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1220.359970] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1221.373450] ena 0000:00:06.0: ENA device version: 0.10
[ 1221.374210] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1221.473878] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1221.486514] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:27:25:70:a6:a7
[ 1221.563935] ena 0000:00:06.0 ens6: renamed from eth0
[ 1221.605984] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1247.840025] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1247.841323] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1270.195588] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-echo:41178)
[ 1345.441727] vfio-pci 0000:00:06.0: Relaying device request to user (#0)
[ 1448.200104] vfio-pci 0000:00:06.0: Relaying device request to user (#10)
[ 1550.598428] vfio-pci 0000:00:06.0: Relaying device request to user (#20)
[ 1652.996795] vfio-pci 0000:00:06.0: Relaying device request to user (#30)
[ 1755.395039] vfio-pci 0000:00:06.0: Relaying device request to user (#40)
[ 1857.793260] vfio-pci 0000:00:06.0: Relaying device request to user (#50)
[ 1960.191490] vfio-pci 0000:00:06.0: Relaying device request to user (#60)
[ 2062.599716] vfio-pci 0000:00:06.0: Relaying device request to user (#70)

sender-console-output.log (PCI/driver events only)

[ 1014.032572] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1043.104903] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1043.106251] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1199.549745] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1200.559892] ena 0000:00:06.0: ENA device version: 0.10
[ 1200.560664] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1200.659872] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1200.672254] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1200.719940] ena 0000:00:06.0 ens6: renamed from eth0
[ 1200.771699] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1230.134145] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1230.135417] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1274.972119] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24840)
[ 1275.731983] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24860)
[ 1322.735249] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1323.748073] ena 0000:00:06.0: ENA device version: 0.10
[ 1323.748826] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1323.849358] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1323.862627] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1323.909422] ena 0000:00:06.0 ens6: renamed from eth0
[ 1323.971427] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1488.543412] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1488.544724] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1645.510393] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1646.523537] ena 0000:00:06.0: ENA device version: 0.10
[ 1646.524295] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1646.624843] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1646.639058] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1646.688118] ena 0000:00:06.0 ens6: renamed from eth0
[ 1646.739960] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Deploy

Deploying PerfTestStack...
Configs: plain-rust,rust-dpdk,tokio-dpdk,native-dpdk
Packet sizes: 64,512,1400,8500

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0b60d4d05d9d014d0 (DPDK ENI: 10.0.1.105)
  • Receiver: i-0e3691dc5f829bbfb (DPDK ENI: 10.0.1.25)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Instances Ready

  • TRex: i-009f1ca09f48d870c (10.0.1.146)
  • DUT: i-0be5094a302370a35 (10.0.1.226)
  • Instance type: c6in.xlarge

@github-actions

Copy link
Copy Markdown

[Perf] Stage: TRex Config

Starting TRex configuration (MAC discovery + NIC binding)...

@github-actions

Copy link
Copy Markdown

[Perf] Stage: TRex Config OK

  • TX: 0000:00:06.0 MAC: 02:99:48:52:f4:6b
  • RX: 0000:00:07.0 MAC: 02:57:40:d5:3a:33
  • Gateway MAC: 02:19:59:0a:56:d7
    Starting TRex server...

@github-actions

Copy link
Copy Markdown

[Perf] Stage: TRex Started

TRex server running. Beginning benchmarks...

@github-actions

Copy link
Copy Markdown

[Perf] DUT Ready

DUT instance i-0be5094a302370a35 SSM working, build complete.

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (1/4)

Running plain-rust benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: plain-rust preflight

TRex API OK: 2 ports
  Port 0: 02:99:48:52:f4:6b
  Port 1: 02:57:40:d5:3a:33
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: plain-rust result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:99:48:52:f4:6b

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 68,994 pps, Drop: 1.4371%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,979 pps, Drop: 0.7296%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 230,975 pps, Drop: 34.0072%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 482,833 pps, Drop: 31.0238%, Lat avg: 0.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,966 pps, Drop: 0.7383%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 224,633 pps, Drop: 35.8192%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 476,528 pps, Drop: 31.9246%, Lat avg: 0.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 240.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,922 pps, Drop: 0.7703%, Lat avg: 151.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 216,650 pps, Drop: 38.1%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 467,580 pps, Drop: 33.2029%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 37,033 pps, Drop: 47.0951%, Lat avg: 175.0 us
  Target: 140,000 pps ... TX: 125,288 pps, RX: 124,261 pps, Drop: 0.82%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 125,288 pps, RX: 124,092 pps, Drop: 0.9544%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/plain-rust.json
EXIT_CODE=0

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (2/4)

Running rust-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: rust-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:99:48:52:f4:6b
  Port 1: 02:57:40:d5:3a:33
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: rust-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:99:48:52:f4:6b

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 68,990 pps, Drop: 1.4437%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,920 pps, Drop: 0.7711%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,773 pps, Drop: 0.3506%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 698,392 pps, Drop: 0.2297%, Lat avg: 276.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 68,994 pps, Drop: 1.4372%, Lat avg: 196.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,958 pps, Drop: 0.7441%, Lat avg: 91.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,882 pps, Drop: 0.3195%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 696,842 pps, Drop: 0.4512%, Lat avg: 283.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 68,990 pps, Drop: 1.4426%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,946 pps, Drop: 0.7532%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,853 pps, Drop: 0.3278%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 561,216 pps, Drop: 19.8262%, Lat avg: 911.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4284%, Lat avg: 542.0 us
  Target: 140,000 pps ... TX: 125,317 pps, RX: 124,151 pps, Drop: 0.9299%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 125,299 pps, RX: 121,785 pps, Drop: 2.8052%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/rust-dpdk.json
EXIT_CODE=0

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (3/4)

Running tokio-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

[Perf] Benchmark Diag: tokio-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:99:48:52:f4:6b
  Port 1: 02:57:40:d5:3a:33
PREFLIGHT_OK

@github-actions

Copy link
Copy Markdown

[Perf] Stage: Benchmark (4/4)

Running native-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions

Copy link
Copy Markdown

Integration Tests — ⚠️ 2 infra flakes · 0 real failures

3f92968 · run · Integration Tests (Graviton) · 15 tests: 13 ✅ · 0 ❌ real · 2 ⚠️ infra

Tier Test Result Time Reason
tier1-dpdk-echo arp_resolution ✅ pass 0.984s
tier1-dpdk-echo udp_send_receive ✅ pass 1.723s
tier1-dpdk-echo echo_roundtrip ✅ pass 1.533s
tier1-dpdk-echo payload_integrity ✅ pass 0.725s
tier1-dpdk-echo jumbo_diagnostics ✅ pass 0.022s
tier1-dpdk-echo jumbo_echo_8000 ✅ pass 1.731s
tier1-tcp-echo-async execution ⚠️ infra 0.000s ENI bind failed on receiver instance
tier1-tcp-echo bidir_echo ✅ pass 0.728s
tier1-tcp-echo bidir_multi ✅ pass 0.742s
tier2-kernel-interop arp_resolution ✅ pass 0.034s
tier2-kernel-interop udp_send_receive ✅ pass 1.036s
tier2-kernel-interop echo_roundtrip ✅ pass 0.840s
tier2-kernel-interop payload_integrity ✅ pass 0.037s
tier2-tcp-echo execution ⚠️ infra 0.000s ENI bind failed on receiver instance
tier3-iperf-interop our_app_sends ✅ pass 2.540s

⚠️ Infra flakes (2) — SSM/ENI setup, not code (task #10)

  • tier1-tcp-echo-async / execution — ENI bind failed on receiver instance
  • tier2-tcp-echo / execution — ENI bind failed on receiver instance

Performance tests passed on GH Actions run 32038128064.
Docs-only change — no data-path code modified. No regression observed.
rust-dpdk improved at 64B/700K (0.2% drop vs 3.5% in Run #50).
@github-actions

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: efe95ec8

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 14.6M 14.1M 1.0x 71
RX recv_from 64B 4.4M 5.6M 0.8x 177
TX send_to 1400B 2.1M 2.1M 1.0x 475
RX recv_from 1400B 1.4M 1.5M 0.9x 677

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 11.1M 10.7M 1.0x 93
RX recv_from (IPv6) 64B 5.0M 6.7M 0.7x 148
TX send_to (IPv6) 1400B 3.4M 3.4M 1.0x 295
RX recv_from (IPv6) 1400B 1.4M 1.5M 0.9x 668

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 14.6M 11.1M 1.32x
RX recv_from (sync) 64B 4.4M 5.0M 0.88x
TX send_to (sync) 1400B 2.1M 3.4M 0.62x
RX recv_from (sync) 1400B 1.4M 1.4M 0.98x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 31.8% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: efe95ec8

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 14.7M 14.0M 1.0x 71
RX recv_from 64B 4.3M 5.7M 0.8x 175
TX send_to 1400B 2.1M 2.1M 1.0x 482
RX recv_from 1400B 1.3M 1.4M 0.9x 715

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 10.9M 10.6M 1.0x 94
RX recv_from (IPv6) 64B 5.0M 6.9M 0.7x 145
TX send_to (IPv6) 1400B 3.3M 3.4M 1.0x 295
RX recv_from (IPv6) 1400B 1.3M 1.4M 0.9x 696

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 14.7M 10.9M 1.35x
RX recv_from (sync) 64B 4.3M 5.0M 0.86x
TX send_to (sync) 1400B 2.1M 3.3M 0.63x
RX recv_from (sync) 1400B 1.3M 1.3M 0.98x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.35x (OK)

OK: IPv6 is 35.1% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

Integration Test Failure (Run 32041431301)

Branch: 119/merge | Commit: efe95ec82011581fec83c675b3481cffe62aeb0e

failure-summary.json

{
  "failed_step": "deploy_infrastructure",
  "error": "Infrastructure deployment failed",
  "exit_code": 2,
  "timestamp": "2026-08-17T15:22:07.448375Z",
  "sender_instance_id": "i-01f4e3ccc0086cdca",
  "receiver_instance_id": "i-0eb39614777338c6c",
  "commit": "efe95ec82011581fec83c675b3481cffe62aeb0e",
  "run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/32041431301"
}```

### receiver-console-output.log (30532 bytes, last 80 lines)

[ 1964.057201] __schedule+0x1bf/0x540
[ 1964.057722] schedule+0x5a/0xd0
[ 1964.058203] schedule_preempt_disabled+0x11/0x20
[ 1964.058895] __mutex_lock.constprop.0+0x372/0x6c0
[ 1964.059576] ? seq_read_iter+0x3ac/0x4b0
[ 1964.060145] ? __kmem_cache_alloc_node+0x179/0x280
[ 1964.060847] driver_override_show+0x1e/0x50
[ 1964.061457] dev_attr_show+0x19/0x40
[ 1964.062010] sysfs_kf_seq_show+0x99/0x100
[ 1964.062601] seq_read_iter+0x120/0x4b0
[ 1964.063194] ? selinux_file_permission+0x108/0x150
[ 1964.063885] vfs_read+0x207/0x2e0
[ 1964.064381] ksys_read+0x5f/0xe0
[ 1964.064909] do_syscall_64+0x35/0x90
[ 1964.065450] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 1964.066202] RIP: 0033:0x7f0f720ff7f2
[ 1964.066735] RSP: 002b:00007ffc1fbe7238 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1964.067811] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f0f720ff7f2
[ 1964.068845] RDX: 0000000000020000 RSI: 00007f0f72328000 RDI: 0000000000000003
[ 1964.069895] RBP: 00007f0f72328000 R08: 00007f0f72327010 R09: 0000000000000000
[ 1964.070927] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000022000
[ 1964.071971] R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000020000
[ 1964.073012]
[ 1964.073363] INFO: task bash:47379 blocked for more than 245 seconds.
[ 1964.074303] Tainted: G U OE 6.1.177-224.371.amzn2023.x86_64 #1
[ 1964.075386] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1964.076524] task:bash state:D stack:0 pid:47379 ppid:47364 flags:0x00000002
[ 1964.077725] Call Trace:
[ 1964.078118]
[ 1964.078451] __schedule+0x1bf/0x540
[ 1964.078982] schedule+0x5a/0xd0
[ 1964.079476] schedule_preempt_disabled+0x11/0x20
[ 1964.080161] __mutex_lock.constprop.0+0x372/0x6c0
[ 1964.080861] ? driver_set_override+0x78/0x150
[ 1964.081511] driver_set_override+0x90/0x150
[ 1964.082125] driver_override_store+0x15/0x30
[ 1964.082766] kernfs_fop_write_iter+0x13a/0x1d0
[ 1964.083438] vfs_write+0x2a6/0x3b0
[ 1964.083962] ksys_write+0x5f/0xe0
[ 1964.084464] do_syscall_64+0x35/0x90
[ 1964.085005] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 1964.085755] RIP: 0033:0x7f41c32ff897
[ 1964.086299] RSP: 002b:00007ffd04dbdd08 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 1964.087390] RAX: ffffffffffffffda RBX: 00007f41c33fd780 RCX: 00007f41c32ff897
[ 1964.088433] RDX: 0000000000000001 RSI: 000056339819bfa0 RDI: 0000000000000001
[ 1964.089457] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f41c33b2d20
[ 1964.090488] R10: 00007f41c33b2c20 R11: 0000000000000246 R12: 0000000000000001
[ 1964.091530] R13: 000056339819bfa0 R14: 0000000000000001 R15: 00007f41c33f89c0
[ 1964.092541]
[ 2062.599716] vfio-pci 0000:00:06.0: Relaying device request to user (#70)
[ 2164.997953] vfio-pci 0000:00:06.0: Relaying device request to user (#80)
[ 2267.386174] vfio-pci 0000:00:06.0: Relaying device request to user (#90)
[ 2369.784401] vfio-pci 0000:00:06.0: Relaying device request to user (#100)
[ 2472.182639] vfio-pci 0000:00:06.0: Relaying device request to user (#110)
[ 2574.580894] vfio-pci 0000:00:06.0: Relaying device request to user (#120)
[ 2676.979170] vfio-pci 0000:00:06.0: Relaying device request to user (#130)
[ 2779.377381] vfio-pci 0000:00:06.0: Relaying device request to user (#140)
[ 2881.775665] vfio-pci 0000:00:06.0: Relaying device request to user (#150)
[ 2984.173829] vfio-pci 0000:00:06.0: Relaying device request to user (#160)
[ 3086.572050] vfio-pci 0000:00:06.0: Relaying device request to user (#170)
[ 3188.970256] vfio-pci 0000:00:06.0: Relaying device request to user (#180)
[ 3291.368459] vfio-pci 0000:00:06.0: Relaying device request to user (#190)
[ 3393.766677] vfio-pci 0000:00:06.0: Relaying device request to user (#200)
[ 3496.164887] vfio-pci 0000:00:06.0: Relaying device request to user (#210)
[ 3598.563088] vfio-pci 0000:00:06.0: Relaying device request to user (#220)
[ 3700.961296] vfio-pci 0000:00:06.0: Relaying device request to user (#230)
[ 3803.359492] vfio-pci 0000:00:06.0: Relaying device request to user (#240)
[ 3905.757706] vfio-pci 0000:00:06.0: Relaying device request to user (#250)
[ 4008.155906] vfio-pci 0000:00:06.0: Relaying device request to user (#260)
[ 4110.554122] vfio-pci 0000:00:06.0: Relaying device request to user (#270)
[ 4212.952335] vfio-pci 0000:00:06.0: Relaying device request to user (#280)
2026/08/17 14:28:00Z: SSM Agent unable to acquire credentials: unexpected error getting instance profile role credentials or calling UpdateInstanceInformation. Skipping default host management fallback: retrieved credentials failed to report to ssm. Error: RequestError: send request failed
caused by: Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.220.37.7:443: i/o timeout
[ 4315.350574] vfio-pci 0000:00:06.0: Relaying device request to user (#290)
[ 4417.748848] vfio-pci 0000:00:06.0: Relaying device request to user (#300)
[ 4520.147046] vfio-pci 0000:00:06.0: Relaying device request to user (#310)
[ 4622.545257] vfio-pci 0000:00:06.0: Relaying device request to user (#320)
[ 4724.943518] vfio-pci 0000:00:06.0: Relaying device request to user (#330)
[ 4827.341754] vfio-pci 0000:00:06.0: Relaying device request to user (#340)
[ 4868.859428] vfio-pci 0000:00:06.0: Removing from iommu group 0


### sender-console-output.log (17437 bytes, last 80 lines)

[ 1012.752530] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1013.765740] ena 0000:00:06.0: ENA device version: 0.10
[ 1013.766500] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1013.867682] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1013.881109] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1013.940759] ena 0000:00:06.0 ens6: renamed from eth0
[ 1014.032572] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1014.034264] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1014.127102] zram_generator::config[20184]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 1042.167766] zram_generator::config[20927]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 1043.104903] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1043.106251] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1199.549745] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1200.559892] ena 0000:00:06.0: ENA device version: 0.10
[ 1200.560664] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1200.659872] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1200.672254] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1200.719940] ena 0000:00:06.0 ens6: renamed from eth0
[ 1200.771699] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1200.773504] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1200.901423] zram_generator::config[23277]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 1229.144118] zram_generator::config[24025]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 1230.134145] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1230.135417] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1274.972119] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24840)
[ 1275.731983] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24860)
[ 1322.735249] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1323.748073] ena 0000:00:06.0: ENA device version: 0.10
[ 1323.748826] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1323.849358] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1323.862627] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1323.909422] ena 0000:00:06.0 ens6: renamed from eth0
[ 1323.971427] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1323.973209] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1324.115830] zram_generator::config[25691]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 1487.587316] zram_generator::config[28626]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 1488.543412] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1488.544724] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1645.510393] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1646.523537] ena 0000:00:06.0: ENA device version: 0.10
[ 1646.524295] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1646.624843] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1646.639058] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1646.688118] ena 0000:00:06.0 ens6: renamed from eth0
[ 1646.739960] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1646.741898] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1646.942519] zram_generator::config[30960]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 2311.906856] zram_generator::config[42159]: zram0: system has too much memory (5027MB), limit is 800MB, ignoring.
[ 2335.799355] kauditd_printk_skb: 111 callbacks suppressed
[ 2335.799357] audit: type=1305 audit(1786974966.228:2143): op=set audit_pid=0 old=1450 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 2335.825203] audit: type=1131 audit(1786974966.258:2144): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2335.828214] audit: type=1131 audit(1786974966.258:2145): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2336.135719] audit: type=1131 audit(1786974966.568:2146): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2336.138661] audit: type=1131 audit(1786974966.568:2147): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2336.141654] audit: type=1131 audit(1786974966.568:2148): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2336.145077] audit: type=1130 audit(1786974966.568:2149): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2336.147986] audit: type=1131 audit(1786974966.568:2150): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2336.154588] audit: type=1334 audit(1786974966.578:2151): prog-id=906 op=UNLOAD
[ 2336.155580] audit: type=1334 audit(1786974966.578:2152): prog-id=905 op=UNLOAD
[ 2336.486995] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 2336.553051] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 2336.556114] systemd-journald[884]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 2336.563786] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 2336.566469] systemd-shutdown[1]: Unmounting file systems.
[ 2336.567753] (sd-remount)[42487]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota'.
[ 2336.569663] XFS: attr2 mount option is deprecated.
[ 2336.589088] systemd-shutdown[1]: All filesystems unmounted.
[ 2336.589903] systemd-shutdown[1]: Deactivating swaps.
[ 2336.590647] systemd-shutdown[1]: All swaps deactivated.
[ 2336.591381] systemd-shutdown[1]: Detaching loop devices.
[ 2336.592549] systemd-shutdown[1]: All loop devices detached.
[ 2336.593332] systemd-shutdown[1]: Stopping MD devices.
[ 2336.594103] systemd-shutdown[1]: All MD devices stopped.
[ 2336.594868] systemd-shutdown[1]: Detaching DM devices.
[ 2336.595620] systemd-shutdown[1]: All DM devices detached.
[ 2336.596387] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[ 2336.598464] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 2336.599530] systemd-shutdown[1]: Powering off.
[ 2336.845664] ACPI: PM: Preparing to enter system sleep state S5
[ 2336.846609] reboot: Power down


### All instance-logs files

total 68
drwxr-xr-x 2 runner runner 4096 Aug 17 15:22 .
drwxr-xr-x 21 runner runner 4096 Aug 17 15:21 ..
-rw-r--r-- 1 runner runner 392 Aug 17 15:22 failure-summary.json
-rw-r--r-- 1 runner runner 30532 Aug 17 15:21 receiver-console-output.log
-rw-r--r-- 1 runner runner 334 Aug 17 15:22 receiver-ssm-failure.log
-rw-r--r-- 1 runner runner 17437 Aug 17 15:21 sender-console-output.log

@github-actions

Copy link
Copy Markdown

❌ Integration Tests Failed (Run 32041431301)

Branch: 119/merge | Commit: efe95ec8

Test Results

Application Logs (last 20 lines)

Full Application Logs (last 200 lines each)
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== SSM Command Failed ===
Status: Failed
Instance: i-0eb39614777338c6c (receiver)
Command ID: f78c8164-55d5-4cb1-923a-a81225dba06f

=== STDOUT ===


=== STDERR ===


=== SSM Command Failed ===
Status: Failed
Instance: i-0eb39614777338c6c (receiver)
Command ID: 1515436c-f20b-43f4-91dc-003030edcb03

=== STDOUT ===


=== STDERR ===


Network & PCI State
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[ 1960.191490] vfio-pci 0000:00:06.0: Relaying device request to user (#60)
[ 2062.599716] vfio-pci 0000:00:06.0: Relaying device request to user (#70)
[ 2164.997953] vfio-pci 0000:00:06.0: Relaying device request to user (#80)
[ 2267.386174] vfio-pci 0000:00:06.0: Relaying device request to user (#90)
[ 2369.784401] vfio-pci 0000:00:06.0: Relaying device request to user (#100)
[ 2472.182639] vfio-pci 0000:00:06.0: Relaying device request to user (#110)
[ 2574.580894] vfio-pci 0000:00:06.0: Relaying device request to user (#120)
[ 2676.979170] vfio-pci 0000:00:06.0: Relaying device request to user (#130)
[ 2779.377381] vfio-pci 0000:00:06.0: Relaying device request to user (#140)
[ 2881.775665] vfio-pci 0000:00:06.0: Relaying device request to user (#150)
[ 2984.173829] vfio-pci 0000:00:06.0: Relaying device request to user (#160)
[ 3086.572050] vfio-pci 0000:00:06.0: Relaying device request to user (#170)
[ 3188.970256] vfio-pci 0000:00:06.0: Relaying device request to user (#180)
[ 3291.368459] vfio-pci 0000:00:06.0: Relaying device request to user (#190)
[ 3393.766677] vfio-pci 0000:00:06.0: Relaying device request to user (#200)
[ 3496.164887] vfio-pci 0000:00:06.0: Relaying device request to user (#210)
[ 3598.563088] vfio-pci 0000:00:06.0: Relaying device request to user (#220)
[ 3700.961296] vfio-pci 0000:00:06.0: Relaying device request to user (#230)
[ 3803.359492] vfio-pci 0000:00:06.0: Relaying device request to user (#240)
[ 3905.757706] vfio-pci 0000:00:06.0: Relaying device request to user (#250)
[ 4008.155906] vfio-pci 0000:00:06.0: Relaying device request to user (#260)
[ 4110.554122] vfio-pci 0000:00:06.0: Relaying device request to user (#270)
[ 4212.952335] vfio-pci 0000:00:06.0: Relaying device request to user (#280)
[ 4315.350574] vfio-pci 0000:00:06.0: Relaying device request to user (#290)
[ 4417.748848] vfio-pci 0000:00:06.0: Relaying device request to user (#300)
[ 4520.147046] vfio-pci 0000:00:06.0: Relaying device request to user (#310)
[ 4622.545257] vfio-pci 0000:00:06.0: Relaying device request to user (#320)
[ 4724.943518] vfio-pci 0000:00:06.0: Relaying device request to user (#330)
[ 4827.341754] vfio-pci 0000:00:06.0: Relaying device request to user (#340)
[ 4868.859428] vfio-pci 0000:00:06.0: Removing from iommu group 0

sender-console-output.log (PCI/driver events only)

[ 1014.032572] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1043.104903] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1043.106251] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1199.549745] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1200.559892] ena 0000:00:06.0: ENA device version: 0.10
[ 1200.560664] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1200.659872] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1200.672254] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1200.719940] ena 0000:00:06.0 ens6: renamed from eth0
[ 1200.771699] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1230.134145] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1230.135417] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1274.972119] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24840)
[ 1275.731983] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:24860)
[ 1322.735249] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1323.748073] ena 0000:00:06.0: ENA device version: 0.10
[ 1323.748826] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1323.849358] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1323.862627] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1323.909422] ena 0000:00:06.0 ens6: renamed from eth0
[ 1323.971427] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1488.543412] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1488.544724] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1645.510393] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1646.523537] ena 0000:00:06.0: ENA device version: 0.10
[ 1646.524295] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1646.624843] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1646.639058] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0600000, mac addr 02:b7:42:f2:36:9f
[ 1646.688118] ena 0000:00:06.0 ens6: renamed from eth0
[ 1646.739960] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels

@gspivey
gspivey merged commit 863cc34 into development Aug 17, 2026
5 of 7 checks passed
@github-actions

Copy link
Copy Markdown

Integration Test Failure — Graviton (Run 32041431312)

Branch: 119/merge | Commit: efe95ec82011581fec83c675b3481cffe62aeb0e

failure-summary.json

{
  "failed_step": "deploy_infrastructure",
  "error": "Infrastructure deployment failed",
  "exit_code": 2,
  "timestamp": "2026-08-17T15:21:36.236019Z",
  "sender_instance_id": "i-0b60d4d05d9d014d0",
  "receiver_instance_id": "i-0e3691dc5f829bbfb",
  "commit": "efe95ec82011581fec83c675b3481cffe62aeb0e",
  "run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/32041431312"
}```

### receiver-console-output.log (62834 bytes, last 80 lines)

[ 2332.890551] sysfs_kf_write+0x4c/0x60
[ 2332.891049] kernfs_fop_write_iter+0x140/0x1d8
[ 2332.891638] vfs_write+0x2a4/0x3a0
[ 2332.892104] ksys_write+0x74/0x104
[ 2332.892559] __arm64_sys_write+0x24/0x3c
[ 2332.893091] invoke_syscall+0x4c/0x100
[ 2332.893599] el0_svc_common.constprop.0+0x54/0x188
[ 2332.894238] do_el0_svc+0x3c/0x98
[ 2332.894691] el0_svc+0x34/0x108
[ 2332.895117] el0t_64_sync_handler+0xbc/0x140
[ 2400.051412] vfio-pci 0000:00:06.0: Relaying device request to user (#60)
[ 2455.732202] INFO: task systemd-journal:840 blocked for more than 614 seconds.
[ 2455.733085] Tainted: G U 6.1.177-224.371.amzn2023.aarch64 #1
[ 2455.734107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2455.735154] task:systemd-journal state:D stack:0 pid:840 ppid:1 flags:0x00800a04
[ 2455.736290] Call trace:
[ 2455.736618] __switch_to+0xe0/0x140
[ 2455.737086] __schedule+0x268/0x700
[ 2455.737560] schedule+0x58/0xf4
[ 2455.737986] schedule_preempt_disabled+0x18/0x24
[ 2455.738604] __mutex_lock.constprop.0+0x2cc/0x91c
[ 2455.739243] __mutex_lock_slowpath+0x1c/0x28
[ 2455.739827] mutex_lock+0x60/0x78
[ 2455.740270] uevent_show+0x90/0x140
[ 2455.740734] dev_attr_show+0x28/0x70
[ 2455.741218] sysfs_kf_seq_show+0x90/0x140
[ 2455.741773] kernfs_seq_show+0x34/0x40
[ 2455.742274] seq_read_iter+0x1dc/0x4ec
[ 2455.742765] kernfs_fop_read_iter+0x40/0x4c
[ 2455.743339] vfs_read+0x210/0x2c0
[ 2455.743785] ksys_read+0x74/0x104
[ 2455.744234] __arm64_sys_read+0x24/0x3c
[ 2455.744756] invoke_syscall+0x4c/0x100
[ 2455.745258] el0_svc_common.constprop.0+0x164/0x188
[ 2455.745914] do_el0_svc+0x3c/0x98
[ 2455.746366] el0_svc+0x34/0x108
[ 2455.746778] el0t_64_sync_handler+0xbc/0x140
[ 2455.747380] INFO: task cat:46497 blocked for more than 491 seconds.
[ 2455.748215] Tainted: G U 6.1.177-224.371.amzn2023.aarch64 #1
[ 2455.749236] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2455.750295] task:cat state:D stack:0 pid:46497 ppid:46496 flags:0x00000004
[ 2455.751432] Call trace:
[ 2455.751754] __switch_to+0xe0/0x140
[ 2455.752227] __schedule+0x268/0x700
[ 2455.752696] schedule+0x58/0xf4
[ 2455.753114] schedule_preempt_disabled+0x18/0x24
[ 2455.753740] __mutex_lock.constprop.0+0x2cc/0x91c
[ 2455.754372] __mutex_lock_slowpath+0x1c/0x28
[ 2455.754958] mutex_lock+0x60/0x78
[ 2455.755406] driver_override_show+0x30/0x74
[ 2455.755971] dev_attr_show+0x28/0x70
[ 2455.756444] sysfs_kf_seq_show+0x90/0x140
[ 2455.756984] kernfs_seq_show+0x34/0x40
[ 2455.757500] seq_read_iter+0x1dc/0x4ec
[ 2455.758003] kernfs_fop_read_iter+0x40/0x4c
[ 2455.758574] vfs_read+0x210/0x2c0
[ 2455.759015] ksys_read+0x74/0x104
[ 2455.759455] __arm64_sys_read+0x24/0x3c
[ 2455.759974] invoke_syscall+0x4c/0x100
[ 2455.760479] el0_svc_common.constprop.0+0x54/0x188
[ 2455.761140] do_el0_svc+0x3c/0x98
[ 2455.761579] el0_svc+0x34/0x108
[ 2455.761994] el0t_64_sync_handler+0xbc/0x140
[ 2502.452839] vfio-pci 0000:00:06.0: Relaying device request to user (#70)
[ 2604.854258] vfio-pci 0000:00:06.0: Relaying device request to user (#80)
[ 2707.255695] vfio-pci 0000:00:06.0: Relaying device request to user (#90)
[ 2809.657108] vfio-pci 0000:00:06.0: Relaying device request to user (#100)
[ 2912.058532] vfio-pci 0000:00:06.0: Relaying device request to user (#110)
[ 3014.459958] vfio-pci 0000:00:06.0: Relaying device request to user (#120)
[ 3116.861384] vfio-pci 0000:00:06.0: Relaying device request to user (#130)
[ 3219.262814] vfio-pci 0000:00:06.0: Relaying device request to user (#140)
[ 3321.664231] vfio-pci 0000:00:06.0: Relaying device request to user (#150)
[ 3424.065655] vfio-pci 0000:00:06.0: Relaying device request to user (#160)
[ 3526.467082] vfio-pci 0000:00:06.0: Relaying device request to user (#170)
[ 3628.868508] vfio-pci 0000:00:06.0: Relaying device request to user (#180)
[ 3731.269934] vfio-pci 0000:00:06.0: Relaying device request to user (#190)
[ 3833.671358] vfio-pci 0000:00:06.0: Relaying device request to user (#200)
2026/08/17 15:17:52Z: SSM Agent unable to acquire credentials: unexpected error getting instance profile role credentials or calling UpdateInstanceInformation. Skipping default host management fallback: retrieved credentials failed to report to ssm. Error: RequestError: send request failed
caused by: Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.220.37.186:443: i/o timeout
[ 3936.072792] vfio-pci 0000:00:06.0: Relaying device request to user (#210)


### sender-console-output.log (54507 bytes, last 80 lines)

[ 1467.656505] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1467.658368] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1467.788525] zram_generator::config[26754]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1495.328532] zram_generator::config[27428]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1496.287704] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1496.288467] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1606.096879] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:29015)
[ 1606.846907] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:29102)
[ 1608.596899] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:29306)
[ 1610.326899] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:29329)
[ 1658.042950] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1659.055547] ena 0000:00:06.0: ENA device version: 0.10
[ 1659.056133] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1659.155411] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 1659.156471] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 1659.167494] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:4a:91:74:21:49
[ 1659.205605] ena 0000:00:06.0 ens6: renamed from eth0
[ 1659.274233] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1659.276111] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1659.403354] zram_generator::config[30209]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1686.687460] zram_generator::config[30651]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1687.639495] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1687.640310] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1730.717929] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:31473)
[ 1731.447837] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:31493)
[ 1776.391064] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1777.403832] ena 0000:00:06.0: ENA device version: 0.10
[ 1777.404430] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1777.503875] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 1777.504942] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 1777.516626] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:4a:91:74:21:49
[ 1777.566549] ena 0000:00:06.0 ens6: renamed from eth0
[ 1777.588917] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1777.590751] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1777.752620] zram_generator::config[32374]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1935.103849] zram_generator::config[35249]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 1936.032903] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1936.033836] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 2086.376445] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 2087.389073] ena 0000:00:06.0: ENA device version: 0.10
[ 2087.389659] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 2087.488925] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 2087.490015] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 2087.501605] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:4a:91:74:21:49
[ 2087.529106] ena 0000:00:06.0 ens6: renamed from eth0
[ 2087.571525] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 2087.573365] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 2087.660093] zram_generator::config[37692]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 2722.213485] zram_generator::config[48729]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[ 2740.615768] kauditd_printk_skb: 109 callbacks suppressed
[ 2740.615770] audit: type=1305 audit(1786978777.903:2217): op=set audit_pid=0 old=1393 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 2740.619413] audit: type=1131 audit(1786978777.903:2218): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.622159] audit: type=1131 audit(1786978777.903:2219): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.641748] audit: type=1131 audit(1786978777.923:2220): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.644678] audit: type=1131 audit(1786978777.923:2221): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.647610] audit: type=1131 audit(1786978777.923:2222): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.650701] audit: type=1130 audit(1786978777.923:2223): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.653665] audit: type=1131 audit(1786978777.923:2224): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 2740.656635] audit: type=1334 audit(1786978777.933:2225): prog-id=937 op=UNLOAD
[ 2740.657603] audit: type=1334 audit(1786978777.933:2226): prog-id=936 op=UNLOAD
[ 2740.780990] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 2740.793464] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 2740.796558] systemd-journald[841]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 2740.835896] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 2740.838764] systemd-shutdown[1]: Unmounting file systems.
[ 2740.839961] (sd-remount)[48834]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota'.
[ 2740.841708] XFS: attr2 mount option is deprecated.
[ 2740.859374] systemd-shutdown[1]: All filesystems unmounted.
[ 2740.860031] systemd-shutdown[1]: Deactivating swaps.
[ 2740.860734] systemd-shutdown[1]: All swaps deactivated.
[ 2740.861392] systemd-shutdown[1]: Detaching loop devices.
[ 2740.862482] systemd-shutdown[1]: All loop devices detached.
[ 2740.863114] systemd-shutdown[1]: Stopping MD devices.
[ 2740.863838] systemd-shutdown[1]: All MD devices stopped.
[ 2740.865232] systemd-shutdown[1]: Detaching DM devices.
[ 2740.865905] systemd-shutdown[1]: All DM devices detached.
[ 2740.866554] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[ 2740.868738] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 2740.869841] systemd-shutdown[1]: Powering off.
[ 2741.104677] reboot: Power down


@github-actions

Copy link
Copy Markdown

❌ Integration Tests Failed — Graviton (run)

Branch: 119/merge | Commit: efe95ec8

Test Results

Application Logs (last 20 lines)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant