Skip to content

Display the configured percentile in the console reports - #490

Merged
PragTob merged 6 commits into
bencheeorg:mainfrom
dajiaohuang:fix/489-configured-percentile
Oct 5, 2026
Merged

PragTob merged 6 commits into
bencheeorg:mainfrom
dajiaohuang:fix/489-configured-percentile

Conversation

@dajiaohuang

@dajiaohuang dajiaohuang commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Console reports previously required the 99th percentile and raised a MatchError when a supported percentiles configuration omitted it. All three formatters now report the highest calculated percentile and label the column accordingly. The default remains 99; an empty percentile configuration uses the calculated 75th percentile. Reports without percentile results display N/A, while missing expected ranks still raise instead of being silently hidden.

Fixes #489.

Validation after the review revisions: real Benchee.run regressions cover [50, 90] and [] across all three report headers. The two helper regressions fail against the previous implementation. On Ubuntu with Elixir 1.14/Erlang, the full suite passed: 138 doctests, 352 tests, 0 failures, 4 excluded and 1 skipped. Formatting, test compilation with warnings treated as errors and strict Credo also passed. This does not establish the upstream Elixir 1.20/OTP 29 matrix; no current-head CI checks are reported.

The console formatters matched `percentiles: %{99 => value}` and
hardcoded the "99th %" column, so a `percentiles` configuration
without 99 raised a MatchError and lost all results.

Report the highest calculated percentile instead, which keeps 99 as
the default while making the column follow the configuration.
The fourth argument of format_scenario carries the percentile now, so the
branches matching true could never be reached - dialyzer flagged both.

@PragTob PragTob left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for your contribution! 💚

I have some questions around the main implementation which I left inline of code where I don't quite understand why it is the way it is. Would be glad to figure that out 👀

Otherwise looks good and thanks for contributing! I'm shocked this was broken actually. If you're comfortable, it'd be great to add a test to benchee_test.exs - which is where the more higher level tests live. Just configuring a different percentile and doing broad assumptions that it all works out (the file has plenty of examples). I just wanna make sure there is no other "I hard expect 99th percentile to be there" bug anywhere in the stack :)

Image

Comment thread lib/benchee/formatters/console/helpers.ex Outdated
Comment thread lib/benchee/formatters/console/helpers.ex Outdated
Comment thread lib/benchee/formatters/console/reductions.ex Outdated
Comment thread lib/benchee/formatters/console/memory.ex
@dajiaohuang

Copy link
Copy Markdown
Contributor Author

Addressed the implementation questions in the latest commit:

  • percentile_value/2 uses Map.fetch!, so a missing rank cannot silently display a different percentile's value.
  • displayed_percentile/1 returns nil when no ranks exist; the three reports show N/A instead of inventing a 99th-percentile value. Statistex still computes 25/50/75 for an empty configuration, and the integration test covers that actual behavior.
  • The reductions selection uses a pipeline.
  • Added Benchee.run/2 integration tests exercising runtime, memory, and reductions reports with [50, 90] and []. The benchmark varies allocations across isolated measurements so all three statistics tables are exercised.

The two helper regressions fail against the previous PR helper and pass with the update. Native Ubuntu/WSL Elixir 1.14 validation: full suite passes (138 doctests, 352 tests, 4 excluded, 1 skipped), affected-file formatting passes, test compilation with warnings-as-errors passes, and strict Credo reports no issues. Credo notes some checks are incompatible with this local Elixir version; the hosted version/platform matrix remains pending.

@PragTob

PragTob commented Oct 5, 2026

Copy link
Copy Markdown
Member

Thank you!

@PragTob
PragTob merged commit 1531f7b into bencheeorg:main Oct 5, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Console formatter crashes when percentiles does not include 99

2 participants