Repository navigation
bench: a cell measured while the conntrack table was full is discarded - #289
Merged
Merged
Conversation
With nf_conntrack loaded (Docker's NAT rules load it) every connection, loopback included, keeps a table entry for 120 s after it closes. Once the table is full the kernel drops new SYNs silently: clients sit in SYN-SENT, retransmit, and the run reads as a slow or idle server. Ad-hoc connection-churn tests on this box filled the 262144-entry table within seconds whenever each run used new ports or addresses. bench_conntrack_full (lib.sh) reports the table at 95% or more, and any.sh discards the cell the way it already discards one with socket errors or an unsaturated reactor. README lists the new reason next to the SO_REUSEPORT one. Checked against fake count/max files: 94/100 not flagged, 95/100 flagged, no conntrack not flagged; this machine (1137/262144) not flagged.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
any.shnow discards a cell measured while the kernel's connection-tracking table was full.With
nf_conntrackloaded (Docker's NAT rules load it, as on the bench box), every connection, loopback included, keeps a table entry for 120 s after it closes. When the table is full the kernel drops new SYNs without reporting anything: clients sit in SYN-SENT, retransmit after a second, and the run reads as a slow or idle server. While measuring short-lived connections for #91, ad-hoc churn runs filled the 262,144-entry table within seconds whenever each run used new ports or addresses. Runs then came back at 0 conn/s, or half-utilised.nstatshowed every SYN retransmit failing (TCPSynRetrans==TCPRetransFail) with no listen drops.bench_conntrack_fullinlib.shreports the table once it is at 95% or more, andany.shdiscards the cell the same way it discards one with socket errors or an unsaturated reactor. The README lists the new reason next to the SO_REUSEPORT one, with the workaround for ad-hoc churn tests: reuse one port and a few addresses, so the entries recycle.Tests
Checked against fake count/max files: 94/100 is not flagged, 95/100 is flagged and prints
95/100, and a machine without conntrack is not flagged. This box (1137/262144) is not flagged. I did not fill the real table to demonstrate it, because that drops new connections machine-wide.The keep-alive drivers
any.shuses don't open connections fast enough to hit this. The check guards the shape that does.