Repository navigation
Empty-partition placeholder histograms still trigger terror.Log via (*Time).Sub during plan-time cardinality estimation in v8.5.6 (#56480 follow-up) #68527
Description
Summary
Issue #56480 reported a terror.Log flood of [types:1292]Incorrect time value: '{0 0 0 0 0 0 0}' originating from statistics.convertDatumToScalar. PR #52615 (merged and cherry-picked into v8.5.x — present in v8.5.6, commit ae18096e02) addresses one call site, but the same root cause is still reachable from a different code path at plan time, through (*types.Time).Sub at pkg/types/time.go:705. The fix in #52615 did not cover this path.
We are seeing the same error spammed ~6 GB/hour per tidb-server instance on a v8.5.6 cluster, filling disks within ~30 hours. The trigger is a partitioned table where some partitions are permanently empty, combined with an OLTP query that runs partition:all and pulls OutOfRangeRowCount for every partition (including empty ones whose stats_histograms row is a placeholder with a zero-value types.Time bound).
This is a regression-in-effect of #56480 even though the literal PR #52615 was applied. The bug is one layer higher in the cardinality stack.
Environment
| Field | Value |
|---|---|
| TiDB release version | v8.5.6 |
| Edition | Community |
| Build time | 2026-04-14 |
| Git commit | ae18096e023780bb56bfce33698abec0d4640d0a |
| Cluster shape | 9 tidb-server + PD + TiKV + TiFlash, baremetal, CentOS |
tidb_enable_auto_analyze |
tested both ON and OFF — error rate unchanged |
| Confirmed PR #52615 present | yes (cherry-pick 1d698c78) |
Stack trace (verbatim from one occurrence, 1 of 82 830 in a single rotated 300 MB log file)
[ERROR] [terror.go:324] ["encountered error"] [error="[types:1292]Incorrect time value: '{0 0 0 0 0 0 0}'"]
[stack="github.com/pingcap/tidb/pkg/parser/terror.Log
pkg/parser/terror/terror.go:324
github.com/pingcap/tidb/pkg/types.(*Time).Sub
pkg/types/time.go:705
github.com/pingcap/tidb/pkg/statistics.convertDatumToScalar
pkg/statistics/scalar.go:81
github.com/pingcap/tidb/pkg/statistics.(*Histogram).OutOfRangeRowCount
pkg/statistics/histogram.go:1002
github.com/pingcap/tidb/pkg/planner/cardinality.GetColumnRowCount
pkg/planner/cardinality/row_count_column.go:306
github.com/pingcap/tidb/pkg/planner/cardinality.GetRowCountByColumnRanges
pkg/planner/cardinality/row_count_column.go:71
github.com/pingcap/tidb/pkg/planner/cardinality.expBackoffEstimation
pkg/planner/cardinality/row_count_index.go:559
github.com/pingcap/tidb/pkg/planner/cardinality.getIndexRowCountForStatsV2
pkg/planner/cardinality/row_count_index.go:297
github.com/pingcap/tidb/pkg/planner/cardinality.GetRowCountByIndexRanges
pkg/planner/cardinality/row_count_index.go:85
github.com/pingcap/tidb/pkg/planner/core.detachCondAndBuildRangeForPath
pkg/planner/core/stats.go:411
github.com/pingcap/tidb/pkg/planner/core.fillIndexPath
pkg/planner/core/stats.go:202
github.com/pingcap/tidb/pkg/planner/core.deriveStats4DataSource
pkg/planner/core/stats.go:142
... (DataSource.DeriveStats / RecursiveDeriveStats frames, then physicalOptimize -> DoOptimize -> executor.Compile -> session.ExecuteStmt -> server.handleQuery) ..."]
Compared to #56480 / PR #52615:
- tidb throws error when querying zero datetime with table statistics #56480 / statistics: reduce allocation of types.Context #52615:
convertDatumToScalarwas called from the histogram load / range-bucket walking path, and the fix suppressed the terror flag there. - This report:
convertDatumToScalaris called from(*Histogram).OutOfRangeRowCountathistogram.go:1002, which is reached fromcardinality.GetColumnRowCountduring plan-time cardinality estimation. The terror is logged through(*types.Time).Subattime.go:705, before PR statistics: reduce allocation of types.Context #52615's check point. So the existing fix does not apply.
Reproduction (sketch)
We have not yet reduced this to a runnable bug-bash test, but the conditions are precise enough that it should be straightforward to construct:
- Create a partitioned table whose partitioning function admits permanently-empty partitions, with a multi-column UNIQUE/secondary key whose last column is
DATETIME. Conceptually:CREATE TABLE t_part ( id BIGINT NOT NULL AUTO_INCREMENT, a VARBINARY(64) NOT NULL, b INT NOT NULL, ts DATETIME NOT NULL, payload JSON NULL, PRIMARY KEY (id, b) /*T![clustered_index] CLUSTERED */, UNIQUE KEY uk_a_b_ts (a, b, ts) ) PARTITION BY HASH (b) PARTITIONS 16;
- Populate only a subset of partitions with rows; leave the rest empty.
ANALYZE TABLE t_part;— partitions with no data get placeholder rows inmysql.stats_histograms(distinct_count=0,null_count=0, no buckets inmysql.stats_buckets). The histogram'sBounds.GetRow(0)for thetscolumn returns the Go zero valuetypes.Time{}.- Run a high-QPS OLTP template that does:
The optimizer chooses
SELECT id, payload FROM t_part WHERE a = ? AND b = ? ORDER BY ts DESC LIMIT 1;
IndexLookUpwithpartition:all,IndexRangeScan ... keep order:true, desc. For every partition (including empties), cardinality is estimated, and for the empty ones the(ts)column path drives the planner intoOutOfRangeRowCountagainst a zero-value bound. - One
terror.Logline (~3 KB stack) per query, per affected partition.
We see this at ~750 QPS on the production template, which produces the ~6 GB/hour log growth quoted above.
What we verified does NOT fix it
| Mitigation | Outcome |
|---|---|
ANALYZE TABLE <t> (full) |
Stats rebuilt cleanly for non-empty partitions, SHOW STATS_HEALTHY >= 99, stats_buckets populated correctly. Error rate unchanged. Empty partitions have no rows to sample -> placeholder re-created with same zero-time bound. |
ANALYZE TABLE <t> UPDATE HISTOGRAM ON <ts_col> |
Same as above. |
DROP STATS <t> + re-ANALYZE |
Same. |
Rolling restart of all tidb-server instances (cache invalidation) |
Brief dip during reconnect window, then full rate resumes once stats reload from disk. |
SET GLOBAL tidb_enable_auto_analyze = OFF |
No effect — the error is plan-time, not analyze-time. |
Optimizer hints USE_INDEX(<other_idx>) / IGNORE_INDEX(uk_a_b_ts) |
Silently rejected. The 3-column unique key satisfies ORDER BY ts DESC LIMIT 1 via keep order:true, desc and the optimizer considers it cost-strictly-better. |
| Upgrading to v8.5.6 (which includes PR #52615) | No effect — different call path, see stack above. |
We have no in-engine workaround. Today we keep nodes alive with an external cron that prunes rotated tidb-*.log / tidb_slow_query-*.log files older than 60 minutes every 10 minutes — purely to outpace the flood. On our side, the durable plan is to remove partitioning from the affected legacy table entirely (side-table + cutover, no in-place ALTER), which should eliminate the empty-partition condition that triggers the placeholder histogram. That work is independent of TiDB and will not help anyone else hitting the same shape.
Production impact
On a 9-node cluster with / = 199 GB and tidb_log_level = 'info':
- Rotation rate: ~6 GB/hour per tidb-server (one 300 MB
tidb-YYYY-...logevery ~3 minutes). - One rotated 300 MB file contains 82 830 copies of the same stack trace (verified by
grep -c 'types:1292'). [log.file] max-days=7 / max-size=300(TiDB defaults plus our overrides) cannot keep up — 6 GB/h x 7d ~= 1 TB > disk size.- Disks fill from 0 -> 100% in ~30 hours.
The blast radius is anyone running a partitioned table with permanently-empty partitions and a DATETIME column in any index that the planner traverses with partition:all. This is not exotic — any time-bucketed or hash-bucketed partitioning with sparse buckets and a DATETIME secondary-key column will hit it.
Suggested fix directions
We do not have a PR ready, but the obvious candidates seem to be (in increasing order of invasiveness):
- Mirror PR statistics: reduce allocation of types.Context #52615 at the second site. Suppress the terror flag for the
(*Time).Subcall atpkg/types/time.go:705when invoked fromconvertDatumToScalarviaOutOfRangeRowCount. The bound being a zerotypes.Timeis a known sentinel for "placeholder histogram", not a real data error — it shouldn't produce ERROR-level logs. - Short-circuit
OutOfRangeRowCounton placeholder histograms. Inpkg/statistics/histogram.go:1002, if the histogram has 0 buckets (i.e.stats_bucketsis empty for this(table_id, is_index, hist_id)), skip the bound-difference calculation entirely and return 0 (ordefaultEstimation) without dereferencingBounds. This avoids the bad call site rather than suppressing its symptom. - Skip stats lookup for partitions with
NDV = 0. Inpkg/planner/cardinality/row_count_index.go/row_count_column.go, when iterating partitions for apartition:allplan, detect partitions whosestats_histograms.distinct_count = 0and use a fixed pseudo-estimate instead of walking their histograms.
Direction (2) feels the most surgical and least likely to regress other estimates. We would be happy to test a patch on the affected workload.
Cross-references
- tidb throws error when querying zero datetime with table statistics #56480 — original report (different call site)
- statistics: reduce allocation of types.Context #52615 — fix for tidb throws error when querying zero datetime with table statistics #56480 (does not cover the path described here)
Notes for whoever picks this up
Anonymised supporting artefacts (full ~3 KB live stack, mysql.stats_histograms / stats_buckets per-partition snapshot demonstrating the placeholder-histogram condition, a slow-log entry for the offending template, and EXPLAIN ANALYZE showing partition:all + keep order:true, desc) are posted as a follow-up comment below.
Activity
ti-chi-bot commentedon May 20, 2026
Welcome @developer-Bushido! It looks like this is your first issue to pingcap/tidb 🎉
developer-Bushido commentedon May 20, 2026
Additional anonymised artefacts from the same
v8.5.6cluster as the original report.All identifiers have been substituted: schema/table/column/index names → generic placeholders; user/service names →
app_user; IPs →<ip>; long hex strings (hash literals, digests) →<hash>; resource group →<rg>; large encoded plan blobs in slow-log →<elided>. Timestamps and numeric performance counters are kept verbatim because they are not identifying.1. Full ~3 KB terror stack (verbatim, single occurrence)
Note the path through both
GetRowCountByIndexRanges(which is what PR #52615 partially addressed at the histogram-load site) andGetColumnRowCount->OutOfRangeRowCount(the still-unfixed site, called viaexpBackoffEstimation). The terror itself is logged through(*Time).Sub @ pkg/types/time.go:705.2.
mysql.stats_histograms/mysql.stats_bucketssnapshot per partitionapp_db.t_partisPARTITION BY HASH(<b_partcol>) PARTITIONS 12, and 9 of 12 buckets currently contain no rows. Run immediately after a freshANALYZE TABLE app_db.t_part:Result:
Reading: each of the 9 empty partitions has all 7 column histograms present in
mysql.stats_histograms(one per column,col_hists=7) but all of them havedistinct_count=0(col_hists_zero_ndv=7) and zero rows inmysql.stats_buckets(n_buckets_col=0,n_buckets_idx=0). These are placeholder rows thatANALYZEcannot fill because there is no data to sample. The placeholder'sBounds.GetRow(0)for anyDATETIMEcolumn returns the Go zero valuetypes.Time{}, which is exactly the value reported as'{0 0 0 0 0 0 0}'in the terror message.For comparison, the 3 populated partitions (p3, p4, p5) have proper buckets (1024 per column, 768 per index — TiDB defaults).
3. Slow-log entry for one offending query (one of ~750 QPS on this template)
Worth highlighting: this is
Result_rows: 1,Process_keys: 2, returning a single row, butNum_cop_tasks: 13andKV_total: 1.34s— 13 cop tasks for a one-row lookup because the plan doespartition:allacross all 12 partitions plus one extra round. Most of the wall-clock is spent fanning out to empty partitions and unwinding their (placeholder) stats during planning + cop_task dispatch.4.
EXPLAIN ANALYZE(synthetic 64-char literal, no row match — plan structure is what matters)Key observations for the diagnosis:
IndexLookUp_26: access object: partition:all— every read does cardinality estimation against all 12 partitions, including the 9 empty ones documented in artefact parser: add gitignore #2.cop_task: {num: 12, ..., iters:12, tasks:12}andrpc_info:{Cop:{num_rpc:12, ...}}— the partition fan-out is real at execution time too, not only at plan time.IndexRangeScan_23 ... keep order:true, descon(a, b, ts)— the index walk preserves order on thets(DATETIME) column to satisfyORDER BY ts DESC LIMIT 1. This is the cardinality path that drops intoOutOfRangeRowCountagainst the empty-partition placeholder histogram.Combined: artefact #2 shows the placeholder histograms exist on 9 partitions; artefact #4 shows the plan touches all 12; artefact #1 shows where in the planner the placeholder is read; artefact #3 shows the production query template that drives ~750 QPS of these together.
truffle-dev commentedon May 20, 2026
Some HEAD-sha pointers (
12bf86642f81cca7c7d62ea8dafe7b295d6febbb) on where the path lives and one citation correction:The PR that closed #56480 was actually #57300 ("statistics: avoid throw error when data contains invalid date value"), not #52615 (which is "statistics: reduce allocation of types.Context" and closes #52342). #57300 was a one-line patch on
statistics/scalar.gothat addedAllowInvalidDate: true, IgnoreZeroInDate: trueto thestmtctx.StatementContexthanded tovalueTime.Sub. After the planner-context refactor, that became the package-levelUTCWithAllowInvalidDateCtxatpkg/statistics/scalar.go:50, defined withtypes.DefaultStmtFlags|types.FlagIgnoreInvalidDateErr|types.FlagIgnoreZeroInDateErr. So the context still carries the suppression intent the original fix shipped.The reason the suppression is moot at HEAD is one screen up from the terror site.
(*Time).Subatpkg/types/time.go:701-710only consultsctx.Location()to forward intot.GoTime; it never readsctx.Flags(). So the twoterror.Log(errors.Trace(err))calls atpkg/types/time.go:705,707fire unconditionally on anyGoTimeerror, regardless of whether the caller setFlagIgnoreInvalidDateErr/FlagIgnoreZeroInDateErr. The intent of #57300 is defeated by a method that doesn't honor the flags it was handed. A minimal mirror would be to guard the twoterror.Logcalls withctx.Flags().IgnoreInvalidDate() / IgnoreZeroInDate()so the suppression contract works as the original fix shipped.Two secondary notes on the diagnosis:
The
hg.Len() == 0early-return atpkg/statistics/histogram.go:1110does exist (it's atpkg/statistics/histogram.go:946onrelease-8.5too), so the histogram reachingconvertDatumToScalarathistogram.go:1176(= the line 1002 site cited in v8.5.6) must haveLen() > 0. The slow log shows# Stats: t_part global:..., so the planner is reading the merged global histogram, not the per-partition placeholder. The global has buckets from the populated partitions; whatever the partition→global merge contributes from the 9 empty placeholders is what puts a zero-coreTime bound atBounds.GetRow(0). So direction (3) — "skip stats forNDV=0partitions" — would need to act during partition→global merging, not at planner stats-lookup time.The
Subbranch that hits the terror only fires whent.Type() == mysql.TypeTimestamp && t1.Type() == mysql.TypeTimestamp; the else branch iscalcTimeTimeDiffwith noterror.Logreachable. The repro sketch in the issue body usests DATETIME NOT NULL, but a DATETIME column would take the else branch and not produce this stack. The production column is likelyTIMESTAMP, worth confirming in the actual schema.lanfuchao commentedon Jun 15, 2026
Fix approach: skip
OutOfRangeRowCountfor empty-partition placeholder histograms at the call siteAfter tracing the full call path, we believe the most correct fix is not to suppress the
terror.Login(*types.Time).Sub(that log is legitimately useful as a signal for corrupt/invalid time values in real data), and not to add a zero-value guard inconvertDatumToScalar(that only treats the symptom).The root cause is that empty partitions should never reach
OutOfRangeRowCountin the first place. An empty partition has a placeholder histogram withLen() == 0(no buckets) — there is no statistical data, so out-of-range estimation is meaningless and should return 0 immediately without touchingBounds.Why the existing
hg.Len() == 0guard inOutOfRangeRowCountdoesn't helpThe placeholder histogram for an empty partition has
len(Buckets) == 0(soLen() == 0), butBounds.NumRows() > 0— theBoundschunk still contains a zero-valuetypes.Time{}entry. The current early-return inOutOfRangeRowCountcheckshg.Len() == 0correctly, but the real execution path reachesOutOfRangeRowCountviagetIndexRowCountForStatsV2afteroutOfRangeOnIndexreturnsfalsefor an empty histogram (becauseOutOfRangealso short-circuits onLen() == 0, so the outer guard doesn't fire either). The code then falls through to thec.Histogram.OutOfRangeRowCount/idx.Histogram.OutOfRangeRowCountcalls — and for the column-stats path, thec.Histogram.Len() > 0guard already exists; the index-stats path is the missing case.Proposed fix: guard
OutOfRangeRowCountcalls withLen() > 0ingetIndexRowCountForStatsV2In
pkg/planner/cardinality/row_count_index.go, inside the out-of-range block ofgetIndexRowCountForStatsV2:This is also consistent with the existing column path in
getColumnRowCount(row_count_column.go), wherec.OutOfRangealready returnsfalsefor empty histograms and the path is safe — the index path just needs the same treatment.Why this is correct semantically
An empty partition has zero rows. Its contribution to cardinality estimation for any range is 0. Calling
OutOfRangeRowCounton a placeholder histogram whose only content is a zero-value bound sentinel is both semantically wrong (there is nothing out of range when there are no rows) and the proximate cause of theterror.Logflood. Gating onLen() > 0is the minimal, surgical fix that:terror.Logbehavior (the log remains correct for real invalid-date data)Boundsfor placeholder histogramsHappy to test a patch against the production workload if that would help move this forward.
winoros commentedon Jun 24, 2026
Closing this as a duplicate of #69440.
This issue is a concrete production trigger of the same root cause:
TIMESTAMPbounds in statistics/cardinality estimation can reachstatistics.convertDatumToScalar -> types.Time.Sub -> GoTime -> terror.Log, so TiDB emitsERROR-level stacks during plan-time cardinality estimation while the optimizer continues.#69440 tracks the broader root cause and includes both:
TIMESTAMPenters a Stats V2 histogram bucket bound.Please continue the discussion in #69440.
winoros commentedon Jun 24, 2026
Closed as duplicate of #69440.