collector/filesystem: clamp negative free/avail block counts to zero - #3763
Open
pujitha24 wants to merge 1 commit into
Open
collector/filesystem: clamp negative free/avail block counts to zero#3763pujitha24 wants to merge 1 commit into
pujitha24 wants to merge 1 commit into
Conversation
Motivation: Issue prometheus#1672 reports node_filesystem_avail_bytes (and, less often, node_filesystem_free_bytes) occasionally reporting a value vastly larger than node_filesystem_size_bytes, while the size metric stays correct. The reporter's df output at the time showed a total size of 879510155264 bytes with an absurdly larger avail value. This matches a known, recurring class of kernel bug: statfs(2)'s f_bavail is computed by the kernel as f_bfree minus reserved blocks (e.g. ext3/ext4 reserves ~5% of blocks for root). When reserved blocks exceed free blocks - which happens as a filesystem fills up - that subtraction goes negative, but f_bavail/f_bfree are unsigned 64-bit kernel fields, so the negative result wraps via two's complement to a value near 2^64. This exact wraparound pattern has been documented independently for ext3/ext4 reserved-block exhaustion, CephFS, and AFS quota handling. golang.org/x/sys/unix.Statfs_t.Bavail/.Bfree are typed uint64 on linux/amd64, so node_exporter faithfully converts the wrapped kernel value into an enormous, misleading byte count - it is not miscomputing anything itself, it is relaying a bogus value as-is. Because the original report was never root-caused by the reporter or maintainer (see the issue thread), this change addresses the known, independently-documented wraparound class rather than a confirmed repro of this specific report; it is offered as a likely fix. Approach: Add a blocksToBytes helper in collector/filesystem_linux.go that treats a block count whose sign bit is set (i.e. int64(blocks) < 0) as a wrapped negative value and returns 0 instead of an enormous byte count. Apply it when computing free (buf.Bfree) and avail (buf.Bavail). buf.Blocks (size) is left unchanged since total block count is not derived from a subtraction that can go negative in the kernel. Zero is not merely a safe fallback here: when reserved blocks exceed free blocks, zero blocks truly are available to unprivileged users, so clamping avail to zero is the semantically correct value, not an approximation. Validation: - go build ./... and GOOS=linux GOARCH=amd64 go build ./... both pass. - GOOS=linux GOARCH=amd64 go vet ./collector/... passes. - GOOS=linux GOARCH=amd64 go test -c ./collector/ compiles cleanly (this file is Linux-only and the dev host is Darwin/arm64 with no Linux runtime available, so the compiled test binary could not be executed directly). - Added TestBlocksToBytes covering a normal value, zero, the smallest and largest block counts with/without the sign bit set, and math.MaxUint64 (wrapped -1). The underlying arithmetic was also manually verified against the original report's numbers: blocksToBytes(858896636, 1024) == 879510155264, matching the exact size value from the issue's df output. Fixes prometheus#1672 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
Issue #1672 reports node_filesystem_avail_bytes (and, less often,
node_filesystem_free_bytes) occasionally reporting a value vastly
larger than node_filesystem_size_bytes, while the size metric stays
correct. The reporter's df output at the time showed a total size of
879510155264 bytes with an absurdly larger avail value.
This matches a known, recurring class of kernel bug: statfs(2)'s
f_bavail is computed by the kernel as f_bfree minus reserved blocks
(e.g. ext3/ext4 reserves ~5% of blocks for root). When reserved blocks
exceed free blocks - which happens as a filesystem fills up - that
subtraction goes negative, but f_bavail/f_bfree are unsigned 64-bit
kernel fields, so the negative result wraps via two's complement to a
value near 2^64. This exact wraparound pattern has been documented
independently for ext3/ext4 reserved-block exhaustion, CephFS, and
AFS quota handling. golang.org/x/sys/unix.Statfs_t.Bavail/.Bfree are
typed uint64 on linux/amd64, so node_exporter faithfully converts the
wrapped kernel value into an enormous, misleading byte count - it is
not miscomputing anything itself, it is relaying a bogus value as-is.
Because the original report was never root-caused by the reporter or
maintainer (see the issue thread), this change addresses the known,
independently-documented wraparound class rather than a confirmed
repro of this specific report; it is offered as a likely fix.
Approach:
Add a blocksToBytes helper in collector/filesystem_linux.go that
treats a block count whose sign bit is set (i.e. int64(blocks) < 0)
as a wrapped negative value and returns 0 instead of an enormous byte
count. Apply it when computing free (buf.Bfree) and avail
(buf.Bavail). buf.Blocks (size) is left unchanged since total block
count is not derived from a subtraction that can go negative in the
kernel.
Zero is not merely a safe fallback here: when reserved blocks exceed
free blocks, zero blocks truly are available to unprivileged users,
so clamping avail to zero is the semantically correct value, not an
approximation.
Validation:
(this file is Linux-only and the dev host is Darwin/arm64 with no
Linux runtime available, so the compiled test binary could not be
executed directly).
and largest block counts with/without the sign bit set, and
math.MaxUint64 (wrapped -1). The underlying arithmetic was also
manually verified against the original report's numbers:
blocksToBytes(858896636, 1024) == 879510155264, matching the exact
size value from the issue's df output.
Fixes #1672
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com