Skip to content

Fix probabilistic interleave of empty sources without replacement - #8627

Open
HaokaiDing wants to merge 1 commit into
huggingface:mainfrom
HaokaiDing:fix/interleave-empty-sources-without-replacement
Open

HaokaiDing wants to merge 1 commit into
huggingface:mainfrom
HaokaiDing:fix/interleave-empty-sources-without-replacement

Conversation

@HaokaiDing

Copy link
Copy Markdown

Fixes #8626.

When probabilistic interleave_datasets uses all_exhausted_without_replacement, an empty source starts as not exhausted. Its first draw appends an invalid source offset, producing a duplicate row from the next dataset or an IndexError when the empty source is last.

Mark empty sources exhausted initially for this strategy, so the existing skip-exhausted-source logic handles them. The other stopping strategies are unchanged.

This is separate from #8399 (no-probabilities all_exhausted) and #8318 (vectorized probabilistic first_exhausted / all_exhausted); the latter retains the without-replacement loop with the all-False initialization.

Validation:

  • Before the fix: 9 empty-source cases fail across seeds 0, 42 and 1234; 5 non-empty/all-empty controls pass.
  • After the fix: HF_HUB_OFFLINE=1 .venv/bin/pytest tests/test_arrow_dataset.py -k 'probabilities_without_replacement_empty_source or probabilities_without_replacement_all_empty' -q: 14 passed.
  • Ruff lint, Ruff formatting and git diff --check pass.
  • The full test suite was not run.

Prepared with OpenAI Codex; tests were executed locally and the diff received an independent review by another Codex agent.

Copilot AI lite review requested due to automatic review settings September 15, 2026 06:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Probabilistic interleave without replacement samples empty datasets

2 participants