Skip to content

fix(model): bind decimals beyond 15 significant digits exactly - #4190

Merged
bpamiri merged 3 commits into
developfrom
peter/wide-decimal-exact-bind
Oct 5, 2026
Merged

bpamiri merged 3 commits into
developfrom
peter/wide-decimal-exact-bind

Conversation

@bpamiri

@bpamiri bpamiri commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4172

Problem

Lucee and BoxLang bind cf_sql_decimal / cf_sql_numeric params through a double. A DECIMAL / NUMERIC value with more than 15 significant digits was therefore stored, or compared in a WHERE, as a different number on PostgreSQL, SQL Server, MySQL and CockroachDB. Adobe binds these exactly when a scale is passed, which Wheels always does.

Change

1. Exact bind type (Base.$applyWideDecimalBind(), from $queryParams())

A decimal / numeric param switches to the adapter's exact form when:

  • every value is a plain decimal string (^-?[0-9]+(\.[0-9]+)?$: no exponent, + or grouping), and
  • at least one value has more than 15 significant digits.
Adapter Exact form
SQL Server, MySQL cf_sql_varchar, inside an exact cast (step 2)
PostgreSQL, CockroachDB cf_sql_other (PostgreSQL rejects varchar for numeric; CockroachDB's varchar WHERE fails)
SQLite, H2, Oracle unchanged

2. Exact cast on MySQL and SQL Server (Base.$castWideDecimalParams(), called from a $performQuery() override in both adapters)

Text alone isn't compared exactly on these two databases:

  • MySQL compares a multi-element IN list of text values with a DECIMAL column as doubles, so a neighbouring value matched. =, single-element IN and OR of = were already exact.
  • SQL Server converts the text to the column's decimal type, rounding a value with more fraction digits than the column onto the stored value, with both = and IN. SQL Server's = already behaved this way before this PR (through the double bind), so this is not a regression.

So a wide param, one value or an IN list, binds as CAST(? AS DECIMAL(p, s)), one param per value:

  • s is the value's own fraction digits, never the column's. A cast to the column's scale would round an over-precise value onto a stored value it doesn't equal.
  • p is 65 on MySQL (scale up to 30) and 38 on SQL Server (scale up to 38).
  • A value the database can't store in any DECIMAL equals no stored value. It is left out of an IN list while another value remains, which leaves IN and NOT IN unchanged. A lone such value, or a list of only such values, keeps the text bind: dropping them all would change NOT IN for NULL rows, and on SQL Server the cast would overflow.
  • The bound-parameter count is unchanged, and parameterize=false (decimal literals in the SQL text, already exact) is untouched.

Values with up to 15 significant digits, exponent forms and nulls keep their existing bind: no behaviour change below the threshold. A probe confirmed that today's cfqueryparam scale does not round ordinary values: amount = 3.004 and amount IN (1.5, 3.004) both match nothing against a DECIMAL(10,2) holding 3.00, on all 4 databases. So per-value scale keeps = and IN consistent.

Tests

vendor/wheels/tests/specs/model/wideDecimalBindSpec.cfc (with a DecAmount test model):

Unit (any database):

  • ordinary values keep cf_sql_decimal / cf_sql_numeric on all 4 adapters;
  • wide values (19.10 digits, a 16-digit integer, a list) map to each adapter's exact form;
  • exponent forms, nulls, and SQLite/H2/Oracle are unchanged;
  • the MySQL and SQL Server cast rewrite, for a list and a single value, at each value's own scale;
  • unstorable values are dropped while another remains, and a list of only those is kept;
  • ordinary values, parameterize=false and the other 5 adapters are left uncast;
  • digit counting.

Live (PostgreSQL, SQL Server, MySQL, CockroachDB): columns amount DECIMAL(10,2), wide DECIMAL(38,10) and whole DECIMAL(38,0).

  • ordinary values are stored and matched;
  • 1234567890123456789.1234567890 is stored exactly, and a 27-digit integer part is exact;
  • a raw-SQL exact insert is matched by = and IN;
  • on the scale-10 and scale-0 columns, with = and IN:
    • the exact value matches (1), including with trailing zeros;
    • a one-digit-off neighbour doesn't (0);
    • a value with more fraction digits than the column doesn't (0);
    • NOT IN and <> behave accordingly.

Runs

RED:

  • On develop (separate clean worktree, spec files only): Lucee 7 and BoxLang fail on all 4 databases. On BoxLang × PostgreSQL, a one-digit-off value matched.
  • On the previous head e54bced (text bind only): Lucee 7 × MySQL fails the neighbour-IN negative (0 → 1). Lucee 7 × SQL Server matches the extra-fraction value with = and IN (0 → 1).

GREEN on ed98313 (pass / fail+error):

Engine PostgreSQL SQL Server MySQL CockroachDB
Lucee 7 (model) 1331 / 0 1335 / 0 1352 / 0 1313 / 0
Adobe 2025 (model) 1328 / 0 1332 / 0 1343 / 0 1310 / 0
BoxLang (model) 1331 / 0 1335 / 0 1346 / 0 1313 / 0

Full suite, because the $performQuery() override sees every query on these two databases:

Engine SQL Server MySQL
Lucee 7 6514 / 0 6543 / 0
Adobe 2025 6509 / 0 6532 / 0

Full compat matrix: run 37118814632 on ed98313.

Related: #4162 / #4173 is the same engine behaviour for Oracle NUMBER(20..38, 0) integers.

🤖 Generated with Claude Code

Lucee and BoxLang bind cf_sql_decimal / cf_sql_numeric through a double,
so a value with more than 15 significant digits was stored, or matched in
a WHERE, as a different number on PostgreSQL, SQL Server, MySQL and
CockroachDB. Such a value now binds as cf_sql_varchar (SQL Server, MySQL)
or cf_sql_other (PostgreSQL, CockroachDB); values with up to 15
significant digits, exponent forms and nulls keep their existing bind.

Fixes #4172

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

rev1-r1 review @ e54bced (draft, 4.3): APPROVE on code; merge gated on compat-matrix run 37114648853 and on 4.2.0 shipping.

  • Base.$queryParams() now ends with $applyWideDecimalBind(). For a non-null cf_sql_decimal/cf_sql_numeric param it switches to the adapter's exact form only when every value is a plain decimal literal (optional -, digits, optional fraction; no exponent or grouping) and at least one has more than 15 significant digits. Values a double holds exactly keep today's bind.
  • Per adapter: PostgreSQL/CockroachDB use cf_sql_other (an untyped bind the server parses; varchar is rejected against numeric there). SQL Server and MySQL use cf_sql_varchar. Every other adapter returns "" and is unchanged, and the SQLite and Oracle $queryParams overrides call super first, so they inherit the no-op.
  • $significantDigitCount() ignores leading zeros and a fraction's trailing zeros, and the spec pins the edges (0.10→1, -000123.4500→5, 120→3).
  • The decisive spec is the negative WHERE case (L128): a neighbour differing only in the 29th significant digit must count 0. That's what proves the comparison is exact rather than going through a double.

For the matrix read I'll check L128 on every MySQL leg. MySQL documents that a string compared with a DECIMAL is compared as a double, unless the optimizer converts the constant to the column's type. If any MySQL leg counts 1 there, the MySQL insert is exact but the match isn't, and MySQL would need a different form (e.g. a CAST in the generated SQL).

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Wheels Test Results

216 999 tests   212 827 ✅  1h 5m 9s ⏱️
 16 576 suites    4 172 💤
     32 files          0 ❌

Results for commit 8a0617e.

♻️ This comment has been updated with latest results.

The text bind for a decimal beyond 15 significant digits was still inexact
on two databases. MySQL compares a multi-element IN list of text values
with a DECIMAL column as doubles, so a neighbouring value matched. SQL
Server converts the text to the column type, rounding a value with more
fraction digits than the column onto the stored value, with = and IN.

Both adapters now bind such a value inside CAST(? AS DECIMAL(p, s)), s
being the value own fraction digits (MySQL 65/30, SQL Server 38/38), via
Base.$castWideDecimalParams() from a $performQuery() override. An IN
value the database cannot store is dropped while another remains; a lone
unstorable value keeps the text bind. Specs pin =, IN, NOT IN and <> on
scale-0 and scale-10 columns on all four databases.

Refs #4172

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

rev1-r1 delta review @ ed98313 (from e54bced; draft, 4.3): APPROVE on code; merge gated on compat-matrix run 37118814632 and on 4.2.0 shipping.

  • The MySQL IN gap is closed, and the SQL Server scale rounding with it. In both adapters $performQuery (same signature as Base) runs $castWideDecimalParams first. Each high-precision decimal param that $applyWideDecimalBind would bind as text becomes CAST(? AS DECIMAL(p, s)), with s = the value's own fraction digits, never the column's, which would round it onto a stored value it doesn't equal. An IN list becomes (CAST(?…), CAST(?…)) with one single-value param per element, and the parameter count is unchanged.
  • Limits: MySQL is 65/30 and SQL Server 38/38. $storableDecimalValues checks fraction ≤ scale and integer + fraction ≤ precision, so every emitted cast is valid.
  • Unstorable values: a value no DECIMAL can hold equals no stored value, so it's dropped from an IN list while another value remains (exact for IN and NOT IN). A single such value, or a list of only such values, keeps its text bind, which preserves NULL NOT IN (…) three-valued logic rather than going through the empty-list shortcut.
  • parameterize=false (decimal literals, already exact) and every other adapter are untouched.
  • Adobe: the helper returns its parts rather than mutating a passed array (Adobe copies arrays on call), and the rewritten list is assigned back through the arguments struct, which is passed by reference. That's the right fix for the bug the author caught.
  • Specs: the cast shape per adapter; the drop-while-one-remains rule; ordinary/inline/other adapters uncast; the digit counts; and =/IN exactness, including neighbour negatives, against both scale-0 and scale-10 columns.

@bpamiri

bpamiri commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

Compat matrix 37118814632 on ed98313: fully green.

  • All 32 legs (5 engines × the databases each runs): 0 failures and 0 errors, read from each leg's JUnit report, not just the job status.
  • wideDecimalBindSpec: 12 cases present on every leg.
    • The 4 live specs ran (not skipped) on all 20 PostgreSQL, SQL Server, MySQL and CockroachDB legs, across Lucee 6, Lucee 7, Adobe 2023, Adobe 2025 and BoxLang.
    • They skip as designed on the 12 SQLite, H2 and Oracle legs.
    • So the neighbour-IN and extra-fraction negatives hold on every MySQL and SQL Server leg, Adobe and BoxLang included.

Approved by rev1-r1 at ed98313. Stays a draft until after the 4.2.0 cut.

@bpamiri
bpamiri marked this pull request as ready for review October 5, 2026 04:05
@bpamiri bpamiri added this to the v4.2.0 milestone Oct 5, 2026
@bpamiri

bpamiri commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Final: APPROVE @ 8a0617e (lifted into 4.2; matrix 37262664711, all 32 legs read)

Delta since my approval at ed98313

Matrix

  • 32/32 legs have 0 fail, 0 error, with 518 bundles each.
  • wideDecimalBindSpec passes 12/12 with 0 skipped on every MySQL, PostgreSQL, SQL Server and CockroachDB leg across all five engines. Those are the adapters the exact bind applies to (variables.applies).
  • On SQLite, H2 and Oracle it passes 8/12. The 4 skips are the adapter-specific cases, "High-precision binds are adapter-specific; not <adapter>", as designed and as at my earlier approval.
  • fix(model): compare date values in the query builder and dynamic finders as dates on SQLite #4281's QueryBuilderDateValueSpec, the develop change that shares Base.cfc, passes 8/8 on every leg.
  • Skips against develop (perf(router): index dynamic routes by method and first two path segments #4183's matrix 37261921912), on lucee7-mysql, lucee7-sqlserver, adobe2025-postgres, boxlang-sqlite and lucee7-oracle: the only additions are this PR's 4 adapter-specific cases on SQLite and Oracle.

No open findings.

@bpamiri
bpamiri merged commit 187a581 into develop Oct 5, 2026
62 checks passed
@bpamiri
bpamiri deleted the peter/wide-decimal-exact-bind branch October 5, 2026 05:15
bpamiri added a commit that referenced this pull request Oct 5, 2026
…cache and exact wide numbers (#4319)

Add 4.1 to 4.2 entries for #4166 (cacheControllerIntegration) and for
#4173/#4190 (wide numbers bound exactly), and end four fragments with a
period.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High-precision DECIMAL/NUMERIC values can be stored as a different number on PostgreSQL, SQL Server, MySQL and CockroachDB

1 participant