Skip to content

collector/filesystem: clamp negative free/avail block counts to zero - #3763

Open
pujitha24 wants to merge 1 commit into
prometheus:masterfrom
pujitha24:auto/issue-1672
Open

collector/filesystem: clamp negative free/avail block counts to zero#3763
pujitha24 wants to merge 1 commit into
prometheus:masterfrom
pujitha24:auto/issue-1672

Conversation

@pujitha24

Copy link
Copy Markdown

Motivation:
Issue #1672 reports node_filesystem_avail_bytes (and, less often,
node_filesystem_free_bytes) occasionally reporting a value vastly
larger than node_filesystem_size_bytes, while the size metric stays
correct. The reporter's df output at the time showed a total size of
879510155264 bytes with an absurdly larger avail value.

This matches a known, recurring class of kernel bug: statfs(2)'s
f_bavail is computed by the kernel as f_bfree minus reserved blocks
(e.g. ext3/ext4 reserves ~5% of blocks for root). When reserved blocks
exceed free blocks - which happens as a filesystem fills up - that
subtraction goes negative, but f_bavail/f_bfree are unsigned 64-bit
kernel fields, so the negative result wraps via two's complement to a
value near 2^64. This exact wraparound pattern has been documented
independently for ext3/ext4 reserved-block exhaustion, CephFS, and
AFS quota handling. golang.org/x/sys/unix.Statfs_t.Bavail/.Bfree are
typed uint64 on linux/amd64, so node_exporter faithfully converts the
wrapped kernel value into an enormous, misleading byte count - it is
not miscomputing anything itself, it is relaying a bogus value as-is.

Because the original report was never root-caused by the reporter or
maintainer (see the issue thread), this change addresses the known,
independently-documented wraparound class rather than a confirmed
repro of this specific report; it is offered as a likely fix.

Approach:
Add a blocksToBytes helper in collector/filesystem_linux.go that
treats a block count whose sign bit is set (i.e. int64(blocks) < 0)
as a wrapped negative value and returns 0 instead of an enormous byte
count. Apply it when computing free (buf.Bfree) and avail
(buf.Bavail). buf.Blocks (size) is left unchanged since total block
count is not derived from a subtraction that can go negative in the
kernel.

Zero is not merely a safe fallback here: when reserved blocks exceed
free blocks, zero blocks truly are available to unprivileged users,
so clamping avail to zero is the semantically correct value, not an
approximation.

Validation:

  • go build ./... and GOOS=linux GOARCH=amd64 go build ./... both pass.
  • GOOS=linux GOARCH=amd64 go vet ./collector/... passes.
  • GOOS=linux GOARCH=amd64 go test -c ./collector/ compiles cleanly
    (this file is Linux-only and the dev host is Darwin/arm64 with no
    Linux runtime available, so the compiled test binary could not be
    executed directly).
  • Added TestBlocksToBytes covering a normal value, zero, the smallest
    and largest block counts with/without the sign bit set, and
    math.MaxUint64 (wrapped -1). The underlying arithmetic was also
    manually verified against the original report's numbers:
    blocksToBytes(858896636, 1024) == 879510155264, matching the exact
    size value from the issue's df output.

Fixes #1672

Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com

Motivation:
Issue prometheus#1672 reports node_filesystem_avail_bytes (and, less often,
node_filesystem_free_bytes) occasionally reporting a value vastly
larger than node_filesystem_size_bytes, while the size metric stays
correct. The reporter's df output at the time showed a total size of
879510155264 bytes with an absurdly larger avail value.

This matches a known, recurring class of kernel bug: statfs(2)'s
f_bavail is computed by the kernel as f_bfree minus reserved blocks
(e.g. ext3/ext4 reserves ~5% of blocks for root). When reserved blocks
exceed free blocks - which happens as a filesystem fills up - that
subtraction goes negative, but f_bavail/f_bfree are unsigned 64-bit
kernel fields, so the negative result wraps via two's complement to a
value near 2^64. This exact wraparound pattern has been documented
independently for ext3/ext4 reserved-block exhaustion, CephFS, and
AFS quota handling. golang.org/x/sys/unix.Statfs_t.Bavail/.Bfree are
typed uint64 on linux/amd64, so node_exporter faithfully converts the
wrapped kernel value into an enormous, misleading byte count - it is
not miscomputing anything itself, it is relaying a bogus value as-is.

Because the original report was never root-caused by the reporter or
maintainer (see the issue thread), this change addresses the known,
independently-documented wraparound class rather than a confirmed
repro of this specific report; it is offered as a likely fix.

Approach:
Add a blocksToBytes helper in collector/filesystem_linux.go that
treats a block count whose sign bit is set (i.e. int64(blocks) < 0)
as a wrapped negative value and returns 0 instead of an enormous byte
count. Apply it when computing free (buf.Bfree) and avail
(buf.Bavail). buf.Blocks (size) is left unchanged since total block
count is not derived from a subtraction that can go negative in the
kernel.

Zero is not merely a safe fallback here: when reserved blocks exceed
free blocks, zero blocks truly are available to unprivileged users,
so clamping avail to zero is the semantically correct value, not an
approximation.

Validation:
- go build ./... and GOOS=linux GOARCH=amd64 go build ./... both pass.
- GOOS=linux GOARCH=amd64 go vet ./collector/... passes.
- GOOS=linux GOARCH=amd64 go test -c ./collector/ compiles cleanly
  (this file is Linux-only and the dev host is Darwin/arm64 with no
  Linux runtime available, so the compiled test binary could not be
  executed directly).
- Added TestBlocksToBytes covering a normal value, zero, the smallest
  and largest block counts with/without the sign bit set, and
  math.MaxUint64 (wrapped -1). The underlying arithmetic was also
  manually verified against the original report's numbers:
  blocksToBytes(858896636, 1024) == 879510155264, matching the exact
  size value from the issue's df output.

Fixes prometheus#1672

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 30, 2026 02:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

node_filesystem_{free,avail}_bytes reporting values larger than node_filesystem_size_bytes

2 participants