[9.3] [MongoDB] Handle out-of-range BSON datetimes to prevent sync failures (#4148) - #4166
Merged
Merged
Conversation
…#4148) ## Summary Documents containing dates outside the Python `datetime` range (years 1–9999) cause pymongo to raise `InvalidBSON` while decoding, which aborts the entire MongoDB sync: ``` year 643385 is out of range (Consider Using CodecOptions(datetime_conversion=DATETIME_AUTO) ...) ``` This PR configures the MongoDB client with `DatetimeConversion.DATETIME_CLAMP`, so out-of-range values are clamped to `datetime.min` / `datetime.max` and remain valid dates. They then serialize to normal ISO date strings, exactly like in-range dates. ## Why DATETIME_CLAMP (and not DATETIME_AUTO) The earlier (closed) PR #3476 used `DATETIME_AUTO`, which keeps in-range values as dates but converts out-of-range values to `long` (epoch millis). That makes a single field hold **mixed** types (ISO string vs. long). Because Elasticsearch infers a field's type from the first document it sees, a later document of the other type is then rejected — turning the hard crash into an order-dependent ingestion failure. `DATETIME_CLAMP` avoids this entirely: every date serializes as a date string, so there is one consistent mapping, no crash, and no breaking change for existing users. The trade-off is that extreme dates lose their exact value (clamped to the datetime bounds) — an acceptable outcome for what are almost always sentinel/garbage dates. ## Context This revives and supersedes the work from the earlier (closed) PR #3476, which was never merged and predates the connector package refactor (it targeted the old single-file `connectors/sources/mongo.py`). This PR ports the fix to the current `connectors/sources/mongo/` package layout, switches the strategy to clamping, and adds unit test coverage. - Original PR: #3476 - Original issue: #3444 - Support case: elastic/sdh-search#1924 ## Changes - `get_client()`: set `datetime_conversion=DatetimeConversion.DATETIME_CLAMP`. - `serialize()`: no special date handling needed — clamped values are ordinary `datetime` objects and go through the existing `datetime -> isoformat()` branch. - Added unit tests: in-range and clamped-bound datetimes serialize to ISO strings, a client-config test asserting `DATETIME_CLAMP`, and an end-to-end test that encodes an out-of-range BSON date, decodes it with the client codec, and confirms it becomes a clamped `datetime` serialized as an ISO string (never a `DatetimeMS` or a number). ## Test plan - [x] `make clean install autoformat lint test PYTHON=python3.11` (all MongoDB tests pass; unrelated pre-existing flaky failures only) - [ ] Verify against a MongoDB collection containing out-of-range dates ## Release Note MongoDB connector: set default `datetime_conversion` to `DatetimeConversion.DATETIME_CLAMP` so out-of-range `datetime` values from MongoDB are clamped to valid dates instead of aborting the sync. See https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/dates-and-times/#handling-out-of-range-datetimes for additional information. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 62aa206)
Jan-Kazlouski-elastic
requested review from
Apmats,
artem-shelkovnikov,
erikcurrin-elastic and
seanstory
July 14, 2026 14:22
Jan-Kazlouski-elastic
enabled auto-merge (squash)
July 14, 2026 14:22
erikcurrin-elastic
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 9.3:
Manual backport: the original PR was labeled for
8.19,9.4,9.5,9.6but not9.3, so the automatic backport never ran for this branch. 9.3 is still receiving patch releases (9.3.8), so this bug fix belongs here too. Cherry-picked cleanly from62aa206(git cherry-pick -x).Made with Cursor