Skip to content

fix: preserve string source provenance across execution - #27467

Open
ck89119 wants to merge 44 commits into
matrixorigin:mainfrom
ck89119:issue-27215-main
Open

fix: preserve string source provenance across execution#27467
ck89119 wants to merge 44 commits into
matrixorigin:mainfrom
ck89119:issue-27215-main

Conversation

@ck89119

@ck89119 ck89119 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #27215

What this PR does / why we need it:

  • adds an independent, MPool-owned string source provenance axis with row-level mixed metadata and lifecycle-safe propagation
  • preserves sources through expression evaluation, planner rewrites, VALUE_SCAN, vector transforms, grouping, aggregates, value windows, spill, and remote codecs
  • keeps semantic policies explicit: COALESCE selected-value; IF/CASE/IFNULL common-domain result source; UNION/DISTINCT/GROUP BY SQL equality with deterministic duplicate-source merge
  • precomputes same-preview GROUP key sources and reserves future-row sidecars before hash commit; allocation rejection remains on the pre-commit spill/retry path
  • makes MIN/MAX and MAX_BY winner/equal behavior source-aware across fixed/varlen/JSON/array values, including NULL MAX_BY winners, partial merge, and MIN/MAX extra contributors
  • uses MORPC v29 for source-aware transient batch and aggregate-state encoding with rolling-upgrade source-only downgrade
  • validates protobuf/process source values before narrowing and preserves the source-free fast path
  • merges current mo/main while retaining main's prepared-runtime-specialization revert; frontend differences add only SQLPrepare/COMStmt StringSource ownership

Testing

  • same-preview [Literal, UserVariable, Literal] group allocation rejection before hash publication and no-allocation-after-commit controls
  • group-key same/mixed source in both orders plus forced spill/reload
  • MAX_BY NULL first winner, replacement, same/mixed equal candidate, and partial merge
  • fixed/bytes MIN/MAX extra wins/ties/loses and fixed numeric/date/decimal winner/equal/partial tests
  • VALUE_SCAN legal/NULL/invalid source, constant/dynamic equivalence, and reset/reuse tests
  • aggregate/group-state codecs, malformed/reuse, group protocol, value-window, dispatch, and remote-result tests
  • frontend prepared-path tests confirming only SQLPrepare/COMStmt source additions after the main revert
  • public SQL BVT for projection, CTE, join, UNION, GROUP BY, DISTINCT, aggregate, materialization, flow control, and prepared reuse
  • diff-derived go list, go build, go vet, owning-package unit tests, BVT verification, gofmt, and git diff --check

Review round 8

  • merged mo/main at ba8954d543; resolved plan.proto/pipeline.proto additively and regenerated both .pb.go files with make generate-pb
  • replaced borrowed-input source mutation/restoration with hashmap.UnitLimit-bounded operator-owned preview scratch and selected-row source overrides
  • proved input source values and sidecar backing ownership remain unchanged for [Literal, Expression], including a shared GROUP BY / ANY_VALUE column
  • proved source-sidecar allocation rejection occurs before hash commit and no allocation occurs after commit, including cross-output-batch publication
  • fixed staticcheck S1009 and added deterministic writer-failure plus MIN/MAX overflow/null/const/empty-extra coverage; the new tests cover 12 blocks that were uncovered in the failed coverage artifact

Review round 9

  • keeps every touched group-key source sidecar alive from preflight through new-row append and existing-row updates, then performs one non-failing final normalization
  • tracks only hashmap.UnitLimit-bounded touched destination vectors; success and every error path finalize through a pre-commit-installed defer
  • covers two-batch a/Literal then a/Expression + b/Literal in internal commit and full operator paths; an allocation controller rejects only after Hash.GroupCount()==2 and observes no allocation
  • adds 25 covered blocks out of 28 newly modified executable blocks (89.29%); combined with the previous exact 741/988 result, this provides margin above the strict >75% gate
  • merged current mo/main at 53f6aa6f0b

Review round 10

  • classifies source/CommitPreview failures with a typed pre-publication error that preserves errors.Is through Unwrap; post-publication failures remain non-retryable
  • routes pre-publication commit failures through cancelGroupByPreflights and the existing spill/retry loop in Group, MergeGroup, and spill reload
  • adds full resident-group rejection tests for Group and MergeGroup: a/Literal then a/Expression, one rejection at resident group count 1, successful spill/retry, final source Expression and count 2
  • adds a direct CommitPreview phase-classification test
  • round-10 changed-block coverage is 21/32; combined with the previous successful CI baseline 754/999, projected coverage is 775/1031 = 75.17%, above the strict threshold
  • merged current mo/main at 6a8e7ecc53

Conflict resolution round 11

  • merged current mo/main at 22b91fe986
  • preserved main's MORPC v27 native ASOF capability and v28 owner-local lock capability; moved independent StringSource wire capability to MORPC v29 and made it latest
  • resolved source proto changes additively and regenerated pipeline.pb.go / plan.pb.go with make generate-pb; repeated generation is clean
  • added a v28 negative / v29 positive StringSource transport gate control

Review round 12

  • generic Vector PreExtend now admits mixed stringSources alongside every other row-parallel sidecar; successful pre-extension makes later SetLength allocation-free
  • the time-window T_any NULL partition materializer preserves selected SQLPrepare/COMStmt ownership for constant and selected NULLs
  • ANY_VALUE, fixed/bytes MIN/MAX, and MAX_BY now compute winning/equal source events in fill and merge preflight, including future group rows and MAX_BY equal candidates
  • direct fill/merge tests prove runtime source publication performs no allocation after preflight for all four aggregate families
  • full Group ANY_VALUE and MergeGroup MIN resident-state rejection tests prove allocation rejection occurs before hash publication and spill/retry succeeds
  • round-12 changed-block coverage is 84/115; combined with the previous successful CI baseline 778/1030, projected coverage is 862/1145 = 75.28%
  • merged current mo/main at ae12701a88; BVT remains 33/33

Review round 13

  • selected-batch preflight now retains a newly reserved mixed-source sidecar through length/payload publication and provenance propagation, then normalizes it at finalization
  • fixed MIN/MAX fill and merge preflight now mirrors runtime admission for every non-NULL candidate, including losing candidates and transient winners
  • added reject-next-allocation regressions for Expression-destination/Literal selected append and fixed MIN fill/merge loser/transient cases
  • owning/dependent package build, vet, UT, SCA, make, and string-source BVT (33/33) pass

Review round 14

  • aggregate source sidecars admitted by outer preflight are retained through the complete ANY_VALUE, fixed/bytes MIN/MAX, and MAX_BY BatchFill/BatchMerge publication and finalized on every exit
  • finalization visits only groups in the current work unit; each touched vector normalizes once, avoiding scans over historical aggregate state
  • reversed ANY_VALUE ordering to Literal then COMStmt under reject-next-allocation for direct fill/merge and full Group spill/retry; MergeGroup uses the same reverse-order resident/new-group regression
  • merged current mo/main at e866c535f5; build, vet, UT, SCA, make, and string-source BVT (33/33) pass

Review round 15

  • Group-key current and standby destinations now explicitly retain successful source-sidecar preflights, including existing-row admissions where finalLength equals Length
  • publication finalization remains the sole normalization boundary after selected append and existing-row updates complete
  • added reject-after-two-admissions coverage for a full 8192-row current group batch plus standby new group, and reject-after-one coverage for all-existing Literal-to-COMStmt reverse order
  • build, vet, UT, SCA, make, and string-source BVT (33/33) pass on current mo/main (e866c535f5)

CI/conflict update

  • merged current mo/main at 8dd1efc201
  • resolved the MORPC capability collision by preserving main's v29 FOUND_ROWS migration state and moving independent StringSource transport to MORPC v30
  • added v29-negative / v30-positive StringSource controls for dispatch, remote results, and process metadata
  • investigated failed Ubuntu UT job 97760234017: the sole failure was TestCloneCommitFailureRollbackKeepsSourceFiles during logservice/Hakeeper connection reset with concurrent Dragonboat heartbeat/propose timeouts; the exact test passes locally after the merge (21.77s)
  • owning/dependent build, vet, UT, SCA, make, and StringSource BVT (33/33) pass

Review follow-up: publication and NULL ownership

  • retained mixed StringSource sidecar reservations through all direct union publication paths (UnionOne, UnionMulti, Union, UnionBatch, and GetUnionAllFunction) and finalized them on every exit
  • retained and uniformly finalized every column reservation across Batch union/append preflight, preserving multi-column admission atomicity
  • preserved const-NULL StringSource across ColumnExpressionExecutor cache creation and reuse
  • made explicit LAG/LEAD NULL defaults copy the selected default row's source while keeping synthetic no-default NULL as Expression
  • unified buffered/streaming batch metadata failure cleanup so prepare kind, runtime domain, and StringSource all roll back
  • added deterministic post-preflight allocation rejection and decoder OOM regressions
  • merged current mo/main at c3974615c9; owning/dependent build, vet, UT, SCA, make, and StringSource BVT (33/33) pass

Hot-path review follow-up

  • restored an explicit O(1) source-free guard for Union/UnionBatch preflight and publication
  • summarized uniform scalar StringSource in O(1); only mixed row sidecars and explicit overrides scan selected rows
  • propagated uniform sources through scalar adoption or bulk range fill instead of Go-level row loops
  • avoided the duplicate mixed-sidecar summary scan when Batch already retained the destination reservation
  • retained all prior publication-order and mixed-source correctness guarantees, including zero-selection control
  • added no-metadata, uniform-source, and mixed-source benchmark controls
  • on the same Apple M1 Pro host, BenchmarkUnionBatchNoMetadata improved from the pre-fix 5.700–5.821 µs/op to a stable 159.0–187.5 ns/op range (0 B/op, 0 allocs/op); uniform scalar measured 164.6–221.3 ns/op
  • merged current mo/main at 5bc051abc4; owning/dependent UT, build, vet, SCA, make, and StringSource BVT (33/33) pass

Design-gate phase

A stable architecture record is now available at docs/design/CLAUDE_STRING_SOURCE_PROVENANCE_DESIGN.md, revision 148b1ad60e.

It links issue #27215 and this implementation PR and defines:

  • problem evidence, invariants, goals/non-goals, and measurable success criteria
  • status quo plus three credible alternatives and the selected scalar/on-demand-sidecar model
  • end-to-end owner and merge rules
  • allocation/publication lifecycle and capacity bounds
  • MORPC v29/v30 capability ownership, mixed-version downgrade, rollback, and protobuf compatibility
  • rollout, fallback, observability, security, validation, and explicit source-free/uniform performance budgets
  • design decisions, accepted tradeoffs, and implementation-conformance requirements

The document is intentionally marked Proposed. Implementation approval remains blocked until a maintainer provides a distinct, traceable design approval for exact revision 148b1ad60e; no approval is being inferred or self-recorded here.

Latest-main conflict resolution

  • merged current mo/main at 9a289d96d5
  • preserved main's MORPC v30 ownership for prepared numeric-prefix common-type casts
  • moved independent StringSource transport to MORPC v31, with v30-negative/v31-positive sender, receiver, process, and group gates
  • preserved both main's prepared runtime specialization/cached-compile fast path and COM_STMT StringSource assignment in the frontend conflict
  • updated the boundary inventory and proposed architecture record for the v30/v31 mixed-version, downgrade, and rollback contract
  • conflict-owning/dependent build, vet, UT, SCA, make, focused protocol controls, and StringSource BVT (33/33) pass

The current proposed design revision is now 72886d1618; distinct maintainer design approval remains pending for this exact revision.

# Conflicts:
#	pkg/pb/pipeline/pipeline.pb.go

@aunjgr aunjgr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review of exact head 3ab9dce4f816849fc9fd09993468ca1cfe52d3cc against merge base 22b91fe986b269d24a6b8816eafdd1fa6327c8e1.

[P1] Preflight source-preserving aggregate sidecars before publishing the group hash

The group-key source path correctly reserves its sidecars before CommitPreview, but the aggregate half does not. Group.buildOneBatch calls each aggregate's PreflightBatchFill at pkg/sql/colexec/group/exec2.go:499-506, commits the hash/key state at lines 513-520, and only afterward calls GroupGrow and BatchFill at lines 522-532. Errors from those post-commit calls are returned directly; they cannot enter the pre-publication spill/retry path.

The updated preserving aggregates still leave source allocations until that post-commit phase:

  • anyExec.PreflightBatchFill models the winner's varlen area and PrepareParamKind, but not StringSource (aggexec/capacity_preflight.go:768-825); BatchFill later calls SetRawBytesAtFromAndUnsetNull, which can allocate the first per-row source sidecar (aggexec/any2.go:37-52).
  • fixed MIN/MAX preflight's candidate/event model carries only value and prepare kind (capacity_preflight.go:1024-1155), while BatchFill first calls PreflightSetStringSourceAt at aggexec/minmax2.go:92-125.
  • byte MIN/MAX and MAX_BY have the same omission; equal-candidate source merging is especially missed because MAX_BY's preflight skips a candidate that does not replace the value winner, while runtime still merges its source.

A minimal reachable failure is an allocation-accounted GROUP with an existing aggregate state whose uniform source is Literal, followed by a unit that introduces a new group or an equal/winning COMStmt candidate. The preview preflights successfully and publishes the new hash group. GroupGrow/aggregate fill then needs the first mixed-source sidecar; if the bounded allocation controller rejects it, the statement returns an allocation error instead of spilling the resident prefix and retrying. The committed group state prevents safe retry. This contradicts the PR's claimed “source-sidecar allocation rejection before hash commit” closure.

Extend PreflightBatchFill/PreflightBatchMerge for every source-preserving aggregate to compute the exact winning/equal source events and reserve destination sidecars—including future group rows—before CommitPreview. After commit, GroupGrow, BatchFill, and BatchMerge must be non-allocating for this metadata. Add deterministic Group and MergeGroup regressions with the allocation controller rejecting the first source allocation after the preview; assert the rejection occurs pre-publication, spill/retry succeeds, and final winner/tie sources are correct. Include ANY_VALUE, fixed/byte MIN/MAX, and MAX_BY equal-candidate controls.

I also audited vector lifecycle/reset, planner literal/deep-copy/hash identity, v29 batch/process/aggregate transport downgrade, flow-control merge policies, and invalid-enum decode cleanup. I found no additional confirmed blocker.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep-reviewed exact head 3ab9dce4f816849fc9fd09993468ca1cfe52d3cc, including the provenance ownership/merge rules, vector lifecycle, aggregate/group publication order, transport boundaries, and prior review findings.

The overall independent source axis is reasonable, but its allocation and propagation contracts are not closed yet:

  1. [P1 lifecycle] Include the mixed-source sidecar in generic vector pre-extension.

    Vector.PreExtend delegates to extendWithBitmaps, which reserves data, requested bitmaps, prepareParamKinds, and runtime-domain metadata, but never stringSources. For a flat vector already carrying mixed row sources, PreExtend(64, mp) can therefore succeed while the source sidecar still has capacity only for the existing rows; the later SetLength calls preExtendStringSources and allocates/panics after the caller treated pre-extension as admission.

    This new row-parallel metadata must participate in the same generic capacity contract as every existing sidecar.

  2. [P1 correctness] Preserve NULL source ownership in the T_any time-window partition materializer.

    getPartitionSetFunction's special T_any closure calls SetConstNull and clears prepare-param metadata, but does not copy w.GetStringSourceAt(sel). A SQLPrepare/COMStmt NULL therefore becomes StringSourceExpression at this otherwise transparent boundary, contrary to the PR's explicit invariant that NULL retains its independent source. Copy/reset the selected source exactly and cover constant and selected NULL cases.

  3. [P1 atomicity/design] Preflight aggregate provenance before publishing GROUP state.

    Group.buildOneBatch preflights aggregate fills, commits the hash/key preview, grows groups, and then performs BatchFill. The new source-preserving aggregate preflights still model values and prepare kinds but omit the winning/equal source events. Runtime ANY_VALUE, fixed/bytes MIN/MAX, and MAX_BY can consequently allocate the first mixed stringSources sidecar only after the hash group is committed; rejection at that point cannot safely enter the existing spill/retry path.

    Extend both fill and merge preflights to model exact winner/tie source merges and reserve current/future destination rows before CommitPreview. After commit, GroupGrow, BatchFill, and BatchMerge must not allocate this metadata. Add allocation-rejection regressions for ANY_VALUE, fixed/bytes MIN/MAX, and MAX_BY equal-candidate behavior through Group and MergeGroup/spill retry.

These are manifestations of one design invariant: once provenance is independently row-owned state, every generic capacity/publication boundary must admit it atomically with the value, not as an operator-specific afterthought.

@aptend aptend left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review completed at exact head 2d52c8d, including the full diff, all prior reviews/inline threads, and the delta since 3ab9dce.

The two previously reported gaps were addressed, but two publication-boundary allocation gaps remain. Both are reproducible by allowing the outer preflight, rejecting the next allocation, and then running the supposedly preflighted mutation. These can make execution fail after logical/vector or group-hash publication, defeating the preflight/retry contract.

Validation on this head:

  • related Go package suite passed
  • focused provenance/preflight race tests passed for 10 repetitions
  • go vet passed
  • git diff --check passed

Please add allocation-rejection regressions for both cases described inline.

Comment thread pkg/container/vector/vector.go
Comment thread pkg/sql/colexec/aggexec/capacity_preflight.go

@aunjgr aunjgr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of exact head 2d52c8d8e28111be52755107c9e51f8100e6dbca against merge base ae12701a888fac102e9641cc7cbb8706527366d8.

[P1] Keep aggregate source reservations alive through the complete batch publication

applyStringSourceEvents now allocates the required destination sidecar before the group hash commit, but the reservation does not survive the runtime mutation sequence. Every SetStringSourceAtWithMP call immediately invokes normalizeStringSources (pkg/container/vector/vector.go:783-809), which releases the sidecar whenever the intermediate rows are uniform. The aggregate paths never call FinalizeStringSourcePreflight; that deferred-normalization lifetime currently exists only for group-key publication.

A concrete reverse-order case still allocates after commit: start with two NULL ANY_VALUE state rows carrying uniform scalar source Literal, then fill group 1 from Literal and group 2 from COMStmt. Preflight sees the second mixed event and allocates [Literal, Literal]. After CommitPreview, publishing group 1 writes Literal; normalization observes a uniform sidecar and frees it. Publishing group 2 then calls preflightStringSourceAt and must allocate the sidecar again. If the allocation controller rejects at that point, BatchFill returns after the hash groups have already been published, so spill/retry is no longer safe. The same ordering is reachable in fill/merge for MIN/MAX and MAX_BY.

The new direct test uses COMStmt before Literal, which keeps the sidecar mixed after the first write and therefore misses this interleaving. Please give aggregate preflight a batch-scoped deferred-normalization lifetime (including error/cancel finalization) so setters cannot discard admitted capacity until all correlated rows are published. Add fill and merge controls with the same-source row first and mixed-source row last, plus a Group/MergeGroup allocation controller that rejects any source allocation after commit.

The winner/equal-event calculations themselves match the runtime aggregate policies; I found no additional confirmed blocker in this delta.

@aptend aptend left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking aggregate publication still reallocates after the outer preflight for a valid source ordering; details are inline.

Comment thread pkg/sql/colexec/aggexec/string_source_capacity_preflight_test.go Outdated

@aptend aptend left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the complete change and all prior review history/threads at exact head 2ccb07b, with focused comparison from the previously reviewed head 2cd1569. The retained-preflight lifetime now covers reverse-order aggregate publication and current/standby group-key publication, closing the prior post-commit reallocation counterexamples. Verified affected package tests, 100x/50x focused stress runs, 10x race runs, go vet, and diff check. The Ubuntu UT failure is TestCloneCommitFailureRollbackKeepsSourceFiles failing amid service connection resets/HAKeeper and logservice timeouts, outside this diff. No blocking findings.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review of exact head d3d6a6d96651e54f0367a82dc538e70cafdd2ac7 against merge base c3974615c9419d2577aa37b2bfcd201b1515fff3.

[P1 performance] Restore the provenance-free bulk-copy fast path

summarizeStringSourceBatch and summarizeStringSourceSelection now scan every selected row unconditionally (pkg/container/vector/vector.go:2128-2158), even when the source has no StringSource metadata. Publication then scans the same rows again in propagatePrepareParamKindsBatch / propagatePrepareParamKindsAll (pkg/container/vector/vector.go:2530-2586). Consequently, an ordinary fixed-width UnionBatch that used to be one bulk copy now pays two Go-level row loops for every vector in normal source-free execution.

This is a measured material regression using the repository existing BenchmarkUnionBatchNoMetadata, with the same Apple M4 host, CGo wrapper, native artifacts, -count=7, and -benchtime=500ms:

  • base c3974615c9: median about 110 ns/op (107.0-112.4), 0 allocs
  • head d3d6a6d966: median about 3206 ns/op (3203-3212), 0 allocs

That is roughly a 29x slowdown in a foundational vector-copy hot path, and it affects the overwhelmingly common case where both vectors are ordinary Expression-owned values. It also contradicts the PR goal that the source-free path remain fast.

Please short-circuit provenance work when neither side has StringSource metadata, handle uniform scalar sources in O(1), and reserve row scans for actual mixed sidecars or explicit per-row overrides. Batch-level preflight should also avoid causing a second identical scan inside the vector union. Keep this existing benchmark as the regression gate and add controls for uniform metadata and mixed sidecars.

I also rechecked the full provenance ownership/merge model, prior publication-order findings, latest-main integration, decoder rollback, const-NULL cache reuse, and LAG/LEAD NULL defaults. The new focused correctness tests pass; this measured hot-path regression is the remaining confirmed blocker.

@aunjgr aunjgr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head d3d6a6d96651e54f0367a82dc538e70cafdd2ac7 against exact base c3974615c9419d2577aa37b2bfcd201b1515fff3. One measured merge-blocking hot-path regression remains.

The metadata-free UnionBatch path now scans all source rows in summarizeStringSourceBatch, then scans them again in propagatePrepareParamKindsBatch to append the default Expression source. This applies to ordinary vectors where neither side has any StringSource or prepare-parameter metadata—the overwhelmingly common path that was previously a bulk copy.

I reproduced BenchmarkUnionBatchNoMetadata for 1,024 INT64 rows on the same Ryzen 7950X3D host, five 300 ms samples per exact revision:

  • base: 80.49–83.54 ns/op
  • head: 4.551–4.841 µs/op
  • both: 0 B/op, 0 allocs/op

That is about a 58x median slowdown in a foundational vector operation. Preserve the source-free bulk-copy fast path with an O(1) metadata-presence guard, handle uniform scalar metadata without per-row scans, and scan only mixed sidecars or explicit row overrides. Keep this benchmark as the regression gate and add uniform/mixed controls so the optimization does not weaken provenance correctness.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review of exact head df0a2ce.

The prior P1 hot-path blocker is materially addressed in bb66117: metadata-free Union/UnionBatch now has an O(1) guard, uniform sources are summarized and propagated without row scans, mixed sidecars remain row-exact, and the zero-selection case is covered. I also rechecked that the previously approved preflight-retention and publication-order invariants were not weakened by this delta. Latest CI is green. Local benchmark execution was unavailable because this worktree does not contain cgo/libmo.dylib, so I treated the same-host benchmark ranges in the PR as supporting evidence rather than reproducing them.

[P1 design gate] Add and obtain approval for the mandatory architecture/design record before implementation approval.

Change scope: the PR adds an independent row-level provenance axis across vectors, expressions, planner rewrites, group/spill/aggregates, process state, remote dispatch, protobufs, and MORPC v30. It changes more than eight production owners, a wire compatibility contract, mixed-version behavior, allocation/publication lifetimes, and foundational vector-copy hot paths. This independently triggers the repository design-first gate by package count, cross-subsystem ownership, wire protocol, lifecycle, compatibility, and performance criteria.

CLAUDE_STRING_SOURCE_BOUNDARY_INVENTORY.md is useful as an implementation boundary checklist, but it is not an approved design artifact: it does not identify the owning issue and implementation PR/revision, has no traceable design approval phase, and does not close problem evidence/success criteria, status quo plus credible alternatives, rollout/fallback/observability, or an explicit capacity/performance budget. The PR body and implementation review rounds cannot substitute for the required reviewed design.

Please place the design in docs/rfcs/, docs/design/, or another stable versioned location, link issue #27215 and this implementation PR, close the required decisions above (especially MORPC v29/v30 mixed-version downgrade and rollback, source-sidecar ownership/bounds, and source-free hot-path acceptance thresholds), and obtain a distinct design approval. Then the implementation can be approved against that exact revision.

Change scope: whole provenance feature/refactor
Trigger: package count plus cross-subsystem, wire, lifecycle, compatibility, and hot-path complexity triggers
Design: missing approved artifact; current boundary inventory is unapproved/incomplete
Decision: REQUEST_CHANGES

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design review PASS for exact revision 148b1ad.

Change scope: the complete row-level StringSource provenance feature across vector, expression/planner, group/aggregate/spill, process/remote transport, protobuf, and MORPC.
Trigger: package-count plus cross-subsystem ownership, wire compatibility, publication lifecycle, capacity, and hot-path complexity triggers.
Design: docs/design/CLAUDE_STRING_SOURCE_PROVENANCE_DESIGN.md.
Blocking findings: none.
Decision log: accepted the independent provenance axis; scalar fast path with on-demand one-byte-per-row mixed sidecar; selected-value and conservative merge rules; pre-publication reservation with retained normalization lifetime; v29 downgrade/v30 capability gate; no catalog migration; source-free 0-allocation and <=2x benchmark gate.
Decision: PASS.
Implementation review remains a separate phase and is evaluated against this exact design revision.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation review PASS against the separately approved design revision 148b1ad.

The previously reviewed implementation closes the row-source propagation, aggregate/group pre-publication reservation, retained-normalization lifetime, decoder cleanup, mixed-version downgrade, and source-free fast-path blockers. The new commit is design-only and introduces no implementation deviation.

I independently reran BenchmarkUnionBatchNoMetadata on Apple M4 with five 500 ms samples: 118.4-123.5 ns/op, 0 B/op, 0 allocs/op. This is close to the previously measured pre-feature baseline and comfortably satisfies the approved <=2x threshold. No remaining blocking finding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working size/XXL Denotes a PR that changes 2000+ lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants