Repository navigation
Statistics of TIMESTAMP column is stored as datetime string without timezone information #52429
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
As you can see, the type of this column is TIMESTAMP, but we store its histogram with datetime string without timezone information. And the timezone is not UTC, it's decided by the session executing the ANALYZE command.
This is not correct and will introduce an estimation error.
Activity
winoros commentedon Apr 8, 2024
This is the root cause of #41985
winoros commentedon Apr 8, 2024
We've already stored the histogram of TIMESTAMP to string value. So it's not easy to store it at its original value for compatibility issues.
A possible fix is that we always store the datetime string with UTC timezone, and do conversion when doing row count estimation.
winoros commentedon Apr 9, 2024
TopN or index's histogram is correct. They're following the normal encoding&decoding procedure.
winoros commentedon Apr 9, 2024
It's a long-existing issue. I think i would not solve it before 8.1.0 is released. I'll fix it in the minor version.
ddl: cover the auto pre-split timestamp path
mjonss commentedon Sep 14, 2026
While reviewing #71008 (pre-split index keys encoded with the wrong session's context) I measured a
consequence of statistics not being stored in UTC that is worth recording here, because the fix for
this issue removes it as a side effect.
The round trip through a session zone has no inverse during a DST fall-back.
Stats-derived
TIMESTAMPvalues are decoded into some session zone and re-encoded from it. For TopNthat is
DecodeColumnTopNValue(encoded, ft, loc)followed bycodec.EncodeKey(loc, ...), whichlooks like it cancels out, and does for any fixed-offset zone. In a zone with DST it does not: during
the autumn fall-back one local wall clock covers two UTC instants, and re-encoding resolves the
ambiguity to the later one.
Measured with
Europe/Amsterdam, where clocks go back from 03:00 CEST to 02:00 CET on 2024-10-27:2024-10-27 00:30(first pass, CEST +02:00)02:302024-10-27 01:30 UTC2024-10-27 01:30(second pass, CET +01:00)02:302024-10-27 01:30 UTC2024-10-27 05:30(ordinary)06:302024-03-31 00:30/01:30(spring forward)01:30/03:30So two distinct stored values collapse onto one, and a value in the repeated hour is silently moved
forward by an hour. Nothing errors:
CoreTime.GoTime(pkg/types/core_time.go:170-185) validatesonly that the wall-clock fields survive, and for an ambiguous local time they do. Spring forward is
safe in this direction, since a UTC to local conversion never produces a nonexistent local time.
Why it belongs here rather than in its own issue. It exists only because a stats value is passed
through a non-UTC zone at all. Keeping statistics in UTC end to end means no ambiguous local time is
ever constructed, so this disappears without a separate fix. Filing it separately would just create
something that closes as a duplicate of this one.
Suggested addition to the fix for this issue: a test that pins the inverse, rather than trusting
that it holds. Something that takes a
TIMESTAMPvalue inside a fall-back repeated hour, for example2024-10-27 00:30 UTCwithEurope/Amsterdamin play, runs it through collection and read-back, andrequires the instant to be preserved. Under the current code that fails; under UTC end to end it
should pass, which makes it a direct verification of the fix rather than a restatement of it.
For reference, #71008's PR has a subtest pinning the current behavior
(
TestPlanAutoPreSplitIndexRegionsTimestampTimeZone, subtestDST fall-back collapses the repeated hour), so when this issue is fixed that assertion is expected to need updating, and that is thesignal the collapse is gone.
ddl, statistics: read statistics as UTC when pre-splitting index regions