Skip to content

fix: precision loss when decoding binary NUMERIC values - #285

Open
aminghadersohi wants to merge 1 commit into
aws:masterfrom
aminghadersohi:exact-binary-numeric
Open

aminghadersohi wants to merge 1 commit into
aws:masterfrom
aminghadersohi:exact-binary-numeric

Conversation

@aminghadersohi

@aminghadersohi aminghadersohi commented Sep 26, 2026 •

Copy link
Copy Markdown

Description

With the binary protocol, numeric_in_binary decodes NUMERIC values as Decimal(raw_value).scaleb(-scale). scaleb rounds to the current decimal context, which is 28 significant digits by default. The fix passes a 39-digit context to scaleb. That is enough for any 8- or 16-byte NUMERIC value, so every digit is kept and the exponent is still -scale.

Motivation and Context

A NUMERIC column with more than 28 significant digits is silently rounded on read:

stored (NUMERIC(38,18)):  12345678901234567890.123456789012345678
returned:                 12345678901234567890.12345679

This was reproduced against Redshift Serverless (1.0.434008). The value is stored exactly; psycopg2 reads it back exactly. Writes are unaffected because numeric_out sends the full text form. There is no existing issue for this.

Testing

  • New test_numeric_in_binary_is_exact_beyond_28_digits in test/unit/datatype/test_data_in.py. It covers 38-digit, negative, smallest-scale, 39-digit (int128 max) and 8-byte values, and asserts exact equality and the exponent. The existing numeric tests use isclose(rel_tol=1e-6), which cannot detect this.
  • Without the fix, 3 of the 5 new cases fail (the values with more than 28 digits). With the fix, all 77 tests in test_data_in.py pass.
  • pytest test/unit: 2043 passed, 13 skipped. The 2 failures in test_browser_idc_auth_plugin.py also fail on the unmodified tree in that environment.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • Local run of ./build.sh succeeds (not run)
  • Code changes have been run against the repository's pre-commit hooks (not run)
  • Commit messages follow Conventional Commit Specification (the PR title does; the single commit message predates that and is not prefixed)
  • I have read the README document
  • I have added tests to cover my changes
  • I have run all unit tests using pytest test/unit and they are passing (all pass except the 2 pre-existing test_browser_idc_auth_plugin.py failures noted above)

CHANGELOG.md is not edited: it is maintained at release time.

License

  • By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

numeric_in_binary returned Decimal(raw_value).scaleb(-scale), which rounds
to the current decimal context (28 significant digits by default). NUMERIC
values with more digits were silently rounded, e.g. NUMERIC(38,18)
12345678901234567890.123456789012345678 was read as
12345678901234567890.12345679. Scale with a 39-digit context, enough for any
16-byte value, so every digit is kept.
@aminghadersohi aminghadersohi changed the title Fix precision loss when decoding binary NUMERIC values fix: precision loss when decoding binary NUMERIC values Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant