Repository navigation
feat: RX backpressure accounting and socket drop counters - #33
Merged
Merged
Conversation
Adds `SO_RCVBUF`-equivalent byte-based accounting to the socket-level
receive queue, with atomic drop counters applications can read without
acquiring the queue lock. This closes the gap the README called out as
"the most important gap for production use" — apps now have visibility
into receive-side backpressure instead of silently losing packets when
the internal VecDeque fills up.
- ReceiveQueue tracks `current_bytes` against `max_bytes` in addition to
the existing packet-count cap; push returns Err(payload) so callers
can record the dropped size.
- UdpSocket gains `set_recv_buffer_size` / `recv_buffer_size` /
`recv_buffer_bytes` / `recv_drops` / `reset_recv_drops`, plus a
`RecvDropStats { packets, bytes }` public type.
- All four push call sites (multicore connected-filter, RTC
connected-filter, RTC burst overflow, ARP resolution loop) now record
drops via a single `record_rx_drop(len)` helper which also bumps a
new `rx_drops_buffer_full` perf counter.
- Default buffer is 256 KiB (close to Linux `net.core.rmem_default`).
- 18 new unit tests cover byte-limit enforcement, packet-count cap,
pop/reclaim, setter validation, perf counter integration, and the
socket-level getters. Full suite: 280 tests passing.
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
[Perf] Stage: DeployDeploying |
✅ Integration Tests Passed (Run 24272160915)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[Perf] Stage: Instances Ready
|
[Perf] Stage: TRex ConfigStarting TRex configuration (MAC discovery + NIC binding)... |
[Perf] Stage: TRex Config OK
|
[Perf] Stage: TRex StartedTRex server running. Beginning benchmarks... |
[Perf] DUT ReadyDUT instance |
[Perf] Stage: Benchmark (1/3)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (2/3)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (3/3)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Diag: testpmd logtestpmd output (last 30 lines) |
[Perf] Stage: Results[03:11:01] INFO Generating markdown summary... Performance Test Results — unknownCommit: 64B packets
512B packets
1400B packets
8500B packets
|
64B/700K rust-dpdk holds 698k pps (0.27% drop) vs plain-rust's 615k (12.1% drop). 512B/700K rust-dpdk delivers 696k pps (0.57% drop) vs plain-rust's 459k (34.4% drop). No regression from the new RX buffer accounting — byte-counting cost is below the measurement noise floor at all tested rates.
[CI] Stage: DeployInfrastructure ready.
|
The new socket-level recv-buffer drops (rx_drops_buffer_full) were already being incremented and folded into the aggregate rx_drops in PerfReporter, but (a) they were indistinguishable from worker ring-full drops in the log line, and (b) the rust-dpdk perf-test launcher never passed --perf-interval, so PerfReporter never ran. Changes: - RateSnapshot: split rx_drops into rx_ring_drops + rx_buf_drops while keeping the aggregate rx_drops field for back-compat. - PerfReporter [PERF] line: print rx_ring_drops and rx_buf_drops as their own key=value pairs alongside the aggregate. - run-perf-tests.sh: pass --perf-interval 10 to the rust-dpdk echo launcher so [PERF] lines actually appear in dut-rust-dpdk-app.log (which the perf workflow already attaches to PR comments). This lets us compare our internal drop counters against TRex's drop % side-by-side in the perf-test report.
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 24273354938)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[CI] Stage: DeployInfrastructure ready.
|
… to matrix Threads perf reporting and recv-drop snapshots through the AsyncUdpSocket trait so async users can opt into the same instrumentation that the sync dpdk-udp UdpSocket exposes. Adds a new tokio-dpdk perf-test config and an "App Drops" column in the per-step results table so the gap between TX/RX can be attributed to either app-level backpressure or AWS network loss. Trait + backends: - AsyncUdpSocket: new async fn enable_perf_reporting and recv_drops with no-op defaults so the Tokio backend works unchanged. - DpdkUdpSocket: implements both via spawn_blocking (cannot use blocking_lock inside an async runtime). - New RecvDropsSnapshot type so callers do not need to depend on dpdk-udp. tokio-echo: - New --perf-interval flag (default 0/disabled), parallels apps/echo. When set, calls socket.enable_perf_reporting on startup. - perf-test-stack.ts now builds tokio-echo with --features dpdk so the binary actually uses the DPDK backend in benchmarks. Per-step App Drops: - perf.rs [PERF] log line gains a leading ts_unix=<epoch.fff> field so samples can be bucketed into TRex per-step time ranges. - run_benchmark.py records ts_start_unix/ts_end_unix on every result step. - run-perf-tests.sh fetches the full set of [PERF] lines per config into $LOGS_DIR/dut-<config>-perf.log, then aggregate_results parses them and injects app_drops/app_ring_drops/app_buf_drops into each step. The markdown summary adds an "App Drops" column. tokio-dpdk config: - New start_dut_tokio_dpdk function (DPDK bind, hugepages, --perf-interval 10). - Wired into the config dispatch, log mapping, dut_stop_all_apps pkill list. - Default CONFIGS in run-perf-tests.sh and the perf-tests.yml workflow input now include tokio-dpdk between rust-dpdk and native-dpdk.
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 24273703471)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[Perf] Stage: TRex Config OK
|
[Perf] Stage: TRex StartedTRex server running. Beginning benchmarks... |
[Perf] DUT ReadyDUT instance |
[Perf] Stage: Benchmark (1/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (2/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (3/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (4/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Diag: testpmd logtestpmd output (last 30 lines) |
[Perf] Stage: Results[17:33:44] INFO Generating markdown summary... Performance Test Results — unknownCommit: 64B packets
512B packets
1400B packets
8500B packets
NIC Drops Instrumentation Self-CheckCompares
plain-rust Kernel NIC Drops (ethtool -S delta)ethtool snapshots not available — baseline or final file missing in |
Captures the Run #11 results after bcd83ca fixed the [NIC-FINAL] emission path. This is the run where the NIC Drops Instrumentation Self-Check table shows OK for both rust-dpdk and tokio-dpdk with zero drift, and where rust-dpdk's 403,492 ierrors cross-validates against upstream testpmd's 403,450 (delta 42 packets, 0.000037%). https://claude.ai/code/session_01AEEjCeNiuSA8MWU7BPy15G
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 24288806390)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
gspivey
pushed a commit
that referenced
this pull request
Jun 19, 2026
gspivey
pushed a commit
that referenced
this pull request
Aug 14, 2026
gspivey
added a commit
that referenced
this pull request
Aug 14, 2026
## Summary Implements **ROADMAP item #33** — `dpdk-stdlib-tcp`: Performance CI workflow. Creates `.github/workflows/perf-tests-tcp.yml`, a manually-triggered (`workflow_dispatch`) TCP performance test workflow matching the existing UDP `perf-tests.yml` pattern but specialized for TCP/ASTF benchmarks. ## Spec Reference - Spec: `.kiro/specs/tcp-support/` · task `15.5` - Requirements satisfied: 17.1, 17.2, 17.3, 17.4, 17.5, 17.6, 17.7 ## Features - **workflow_dispatch trigger** with TCP-specific configurable inputs: - `payload_sizes`: TCP payload sizes (default: 64,512,1400,65536) - `cps_rates`: target connections-per-second (default: 100,500,1000,5000) - `duration`: seconds per rate step (default: 30) - `configs`: DUT configurations (default: plain-rust-tcp,rust-dpdk-tcp,tokio-dpdk-tcp) - `teardown`: stack destroy toggle - **Concurrency group** `perf-tests-tcp` (cancel-in-progress: false) — separate from UDP perf group - **validate-cdk** pre-flight job (synthesize PerfTestStack) - **perf-tests-tcp** job: - AWS OIDC authentication - SSM AMI lookup (DPDK + TRex) - PR number resolution for comment posting - Invokes `run-perf-tests.sh` with `--tcp-payload-sizes`, `--cps-rates`, `--configs` flags - Failure diagnostics posted as structured PR comment - `tcp-perf-results` artifact with **90-day retention** - `tcp-perf-instance-logs` artifact with 30-day retention - Safety-net CDK teardown on failure ## Tests - `cargo build` — passes (workflow YAML doesn't affect Rust compilation) - `cargo test` — all 1040+ tests pass, 0 failures - The workflow reuses the existing `run-perf-tests.sh` infrastructure which already supports TCP configs via `run_tcp_benchmark_for_config` (TRex ASTF mode) ## Tradeoffs - Uses a separate concurrency group (`perf-tests-tcp`) from UDP (`perf-tests`) so TCP and UDP perf tests can run independently without blocking each other. Both share the same CDK stack so they won't actually run simultaneously, but this gives clearer queue semantics. --------- Co-authored-by: Agent Router <agent@agent-router.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements RX backpressure and drop counters — the roadmap item the README called out as "the most important gap for production use". Applications now have first-class visibility into receive-side packet loss instead of silently losing datagrams when the internal socket buffer fills.
This is the next well-scoped feature from the Planned roadmap, picking up right after TX/RX checksum work (already in "Done" from #32).
What changed
ReceiveQueuenow tracks both packet count and queued payload bytes.push()returnsResult<(), Vec<u8>>— on reject it hands the payload back so the caller can record the drop size.pop()decrementscurrent_bytes.UdpSocketgains the SO_RCVBUF-style API:recv_buffer_size()/set_recv_buffer_size(bytes)— mirrors POSIX getsockopt/setsockopt SO_RCVBUFrecv_buffer_bytes()— live watermarkrecv_drops() -> RecvDropStats { packets, bytes }— lock-free atomic readreset_recv_drops()— for windowed measurementrecv_queue.push()call sites (multicore connected-filter, RTC connected-filter, RTC burst overflow, ARP resolution loop) now record drops via a singlerecord_rx_drop(len)helper.rx_drops_buffer_fullis rolled into the existingrx_dropsrate on the periodic reporter, so socket-level drops show up alongside worker-ring drops with no extra plumbing.net.core.rmem_default); constantsDEFAULT_RECV_BUFFER_BYTES/DEFAULT_RECV_BUFFER_PACKETSare public so apps can compare against them.Scope / non-goals
sk_rcvbuf/sk_rmem_alloc. Hardware RX ring sizing is still configured separately viaTopologyConfig.Test plan
cargo buildclean (only pre-existingunused_unsafewarnings indpdk/)cargo test— 280 tests passing (was 262). 18 new tests cover:pop()reclaims capacity for subsequent pushesset_max_bytes()updates live limitset_recv_buffer_size(0)rejected withInvalidInputrecord_rx_dropalso bumps therx_drops_buffer_fullperf countercounter_snapshot_ratesupdated to aggregate buffer-full drops intorx_dropshttps://claude.ai/code/session_01AEEjCeNiuSA8MWU7BPy15G