Skip to content

Collection Profile 0.3.0: say which parent keys DETAIL_COVERAGE required_keys must cover #631

Open

Description

@tnunamak

Section 5.6 of the Collection Profile (docs/spec/collection-profile.md, around lines 1353-1370) defines a DETAIL_COVERAGE report's required_keys as the connector's own complete key accounting for one detail stream and parent boundary. It does not tie that set to the parent records the runtime accepted in the same run.

So a connector can emit parent records and then report required_keys: [], or omit a key, and the parent cursor commits with that key's detail never accounted for. The runtime cannot reject this without stalling a legitimate pattern. A connector emits required_keys: [] beside accepted parents when their detail is already held, which is what the Amazon orders and order_items streams do.

Data Connect accepts this as a known limit in 0.2.0 (data-connect PR #636 pins the behaviour in a test).

Proposal for 0.3.0: state which parent keys must appear in required_keys. One option is every parent key accepted for that boundary in the run, unless the report marks the key's detail as already held. A runtime can then check coverage without relying on the connector's own set.

Assisted-by: AI

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions