Repository navigation
fix(data-modeling): keep materialization on when temporal is unreachable - #116509
posthog[bot] wants to merge 1 commit into
Conversation
A failed Temporal connect in the v2 schedule lookup now raises SchedulingUnavailableError and leaves is_materialized unchanged. Other scheduling failures still disable materialization. The materialize action restores the previous state, and the endpoints enable unwinds its atomic block. Adds reenable_materialization to recover saved queries a past outage switched off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 8bbe4016-6ae8-4d06-bb8e-854675d2c9b0
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
🤖 CI report
|
| File | Comment lines | Added lines |
|---|---|---|
products/data_modeling/backend/logic/saved_query_materialization.py |
2 | 11 |
products/data_modeling/backend/models/datawarehouse_saved_query.py |
2 | 23 |
products/data_modeling/backend/schedule.py |
2 | 13 |
products/data_modeling/backend/management/commands/reenable_materialization.py |
1 | 66 |
products/endpoints/backend/logic/materialization.py |
1 | 4 |
This check does not block merging. It updates on every push and clears when the share drops.
![stamphog[bot]](https://avatars.githubusercontent.com/in/3000332?s=60&v=4){kind=small}
There was a problem hiding this comment.
Not approved — escalated to a human reviewer.
Re-add the stamphog label to request another review once you have addressed this.
The materialization-state changes enter risky territory without independent assurance; the merge-queue bot comment is not a review. Request a current-head review covering scheduling and recovery from @PostHog/team-data-modeling before re-requesting.
- Risky territory without independent assurance: Data models: products/data_modeling/backend/models/datawarehouse_saved_query.py changes the persisted materialization-state failure contract, preserving the flag when the initial Temporal lookup has a recognized availability failure.; Data models: products/data_modeling/backend/management/commands/reenable_materialization.py adds an operator command that writes saved-query materialization flags and schedules selected queries across teams.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 143L, 6F substantive, 243L/9F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1c-medium (243L, 9F, single-area, fix) |
| stamphog 2.5.0 | .stamphog/policy.yml @ e8cd392 · reviewed head e8cd392 |
![posthog[bot]](https://avatars.githubusercontent.com/in/1203049?s=60&v=4){kind=small}
| saved_query.is_materialized = True | ||
| saved_query.save(update_fields=["is_materialized"]) | ||
| try: | ||
| saved_query.schedule_materialization() |
There was a problem hiding this comment.
P2 · Promote recovered view nodes before reporting success
When a query's first materialization enable failed during the Temporal lookup, its node remains VIEW because the enable action never reached _mark_nodes_materialized. This command restores the flag and reports success, but schedule_materialization does not change the node type, so execute_dag skips it as ephemeral. Promote recovered nodes to MAT_VIEW after successful scheduling.
| saved_query.refresh_from_db(fields=["is_materialized"]) | ||
| if saved_query.is_materialized: | ||
| reenabled += 1 |
There was a problem hiding this comment.
P2 · Propagate reconciliation failures before counting recovery as successful
When recovering a failed first enable on an already-tiered DAG, the lookup can succeed while schedule reconciliation fails. _reconcile_dag_best_effort swallows that failure, so this flag check reports recovery although the tier never includes the query. Subsequent command runs skip it as already materialized. Use a reconciliation path that propagates errors before counting success.
This branch has not been deployed
1 participant
Problem
schedule_materializationfirst asks Temporal which DAGs run on tiers. When the connect fails (for example, a TLS handshake error), the broadexcept Exceptiontreats it as a permanent fault and setsis_materialized=False.Origin
a59815eChanges
UNAVAILABLE,DEADLINE_EXCEEDEDorRESOURCE_EXHAUSTEDfrom the v2 lookup, now raisesSchedulingUnavailableError. The error is captured, andis_materializedstays as it was.is_materializedand frequency, returns the same "Materialization failed" error as beforereenable_materializationcommand takes the saved query ids from the captured scheduling errors. It skips deleted and already materialized ones, and schedules the rest. It runs as a dry run unless you pass--apply. If a query fails to schedule again, the command reports it and leaves it off.Note
A brand-new managed view whose first schedule hits an outage stays
is_materialized=Trueuntil the next viewset sync schedules it. Before this change, the outage switched the view off permanently.How did you test this code?
test_endpoint_materialization.py,test_managed_viewset.py, plusruff,uv run mypy --cache-fine-grained .andhogli product:lint data_modeling.Test rationale:
test_temporal_outage_raises_and_keeps_materialization(TLS connect error,UNAVAILABLERPC) covers the reported regression. Both cases fail when the model change is reverted. It sits next totest_disables_materialization_when_v2_lookup_fails, which is unchanged and still proves that an unknown error disables materialization.test_saved_query_materialization.pyis now parameterized with an outage case. It catches the materialize action leavingis_materialized=Truewith no schedule after the new error.test_reenable_materialization.pycovers the dry run, and an apply run that must skip deleted queries and report a query with no node that cannot be scheduled.Release status
Docs update
None. No existing doc covers this scheduling contract.
🤖 Agent context
Autonomy: Fully autonomous
Agent: Claude Code, claude-opus-5-5
schedule_materialization. An outage that lasts an hour outlives any in-request retry. Raising lets each caller keep the correct state, and the managed viewset sync already retries on its next run./writing-tests,/writing-pr-descriptions.Created with PostHog Desktop from this inbox report.
🤖 Generated with Claude Code