Skip to content

push_to_hub() for videos - #7971

Merged
lhoestq merged 2 commits into
mainfrom
push_to_hub-for-videos
Jan 29, 2026
Merged

lhoestq merged 2 commits into
mainfrom
push_to_hub-for-videos

Conversation

@lhoestq

@lhoestq lhoestq commented Jan 29, 2026 •

Copy link
Copy Markdown
Member

possible now that row group sizes are auto-determined based on the content size after #7589

Videos are uploaded as PLAIN in Parquet to make sure they can be seeked remotely and with random access to frames in #7976

In the future it could be cool to have the same behavior as when videos are separate files, i.e. lazily load them instead of downloading them completely in streaming mode.

Right now there is this discrepency:

  • load_dataset("username/my-folder-of-videos", streaming=True) -> videos are lazy loaded one by one when iterating, and only actually downloaded when accessing frames in torchcodec
  • load_dataset("username/my-video-dataset-in-parquet", streaming=True) -> videos are downloaded one by one when iterating, even if no frame is accessed in torchcodec

close #7493

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq
lhoestq merged commit 518bf32 into main Jan 29, 2026
9 of 15 checks passed
@lhoestq
lhoestq deleted the push_to_hub-for-videos branch January 29, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

push_to_hub does not upload videos

2 participants