Repository navigation
Post-processing videos (still frame removal and speedup) #223
Description
This can be standalone from #212, but is of course related. The idea would be to remove segments of the video where there is little to no change (which could be difficult when there are blinking lights for example) and run a speedup of e.g., 16x, to later be uploaded to a separate playlist (we can worry about automatic video uploads later, though not sure if that requires just an API key or needs token.pickle).
https://claude.ai/share/217878d2-d194-4441-8c91-9af937ba787b
Metadata
Metadata
Assignees
Labels
No labels
Activity
sgbaird commentedon May 1, 2025
Per our conversation, I think the decision was to include something in the title (maybe original video id? Or could be timestamp) that we can keep the same between the original and the processed videos, and add some kind of note in the title like:
[processed, 16x]. @Neil-YLThis can be used with the YouTube API, grabbing lists of the video names from from original and processed. From the YouTube API quota perspective, if multiple pages are returned, each page will count towards the quota.
Per @Jonathan-Woo 's suggestion, hosting this on a paid-tier HuggingFace space with a the lowest tier GPU (could also try the free, preemptable A100) and the minimum sleep time of 5 minutes of inactivity seems pretty reasonable and low-cost.
@Jonathan-Woo which algorithms/codebases were you looking at? I had found some in the transcript above (a few links were dead of course), might be worth a quick look.
sgbaird commentedon May 1, 2025
@Jonathan-Woo here's the example video you can use: https://www.youtube.com/live/Tbru5BiokmU?si=Sbmfu5dggSYLpnsp
Jonathan-Woo commentedon May 1, 2025
Yep, I looked at the transcript, thanks for that. I'm looking into motion and auto-editor right now. Both can provide timestamps. I think I'll test both and examine their performance. Leaning more towards motion as it seems to be more popular.
sgbaird commentedon May 6, 2025
Related: https://docs.frigate.video/ (I guess @yakavetsiv is incorporating or at least has plans to). Thanks @kelvinchow23 for the ping about it
EDIT: using with esp32 cameras, PoE, and local network
Ilya running everything on a local server, Zima Cube (frigate, MQTT broker, Prefect local)
Jonathan-Woo commentedon May 7, 2025
I've been using auto-editor so far and the results are pretty good on the video linked above. The following command cut these sections out:
https-www-youtube-com-live-Tbru5BiokmU-si-WtsDaMc0DSHxFbKH_ALTERED.mp4
The tricky part is to tune:
There is also support for yt-dlp directly so we can pass the stream link directly. Also, there is support for hardware encoding.
One issue is that, since we don't have a wall clock, it's difficult to tell when video is being sped up. So, I'm working on adding an indicator for when video is being sped up. Then, it will be easier to tune the settings.
sgbaird commentedon May 9, 2025
Really cool to see this! Had thought about adding an overlay directly via the pi zero 2w but not sure if it will handle that extra processing well, and might complicate other things (for one, we lose the raw footage). Ultimately leaned away from it since YouTube already has timestamps.
White text with slightly transparent black background has often worked well from a contrast and aesthetics point of view.
Any luck with adding the indicator?
Also would be great if you could share the sections that were kept, if not too much work.
sgbaird commentedon May 12, 2025
@Jonathan-Woo bump
Jonathan-Woo commentedon May 12, 2025
Still working on adding the indicator. I'm using ffmpeg to add an overlay similar to https://video.stackexchange.com/questions/12105/add-an-image-overlay-in-front-of-video-using-ffmpeg.
Overall, this my plan:
This is the kept video:
video_short_ALTERED.mov
sgbaird commentedon May 12, 2025
Awesome, thanks! It seems like it did a pretty good job with separating the stale vs. non-stale
Quick aside: Originally, I was imagining deleting the stale sections entirely and speeding up the non-stale sections
Jonathan-Woo commentedon May 12, 2025
Sure that's totally possible. We would just have to update the
--silent-speedargument.Jonathan-Woo commentedon May 14, 2025
Overlay has been added.
overlayed.mp4
I'm concerned about removing stale sections entirely because I'm concerned that this editor may have false positives.
I will share the final sped up version soon. In the meantime we should think about what to remove/speed up.
sgbaird commentedon May 14, 2025
Nice! Good point about false positives. I'm worried a bit about devices that have a lot of stale time (e.g., 8 hrs of stale video @ 16x = 30 min), which might get cumbersome for promo vids and robotic training. Though, it could cause problems if people are doing post-mortem analysis (i.e., watching the sped-up video) and miss something that would otherwise catch their eye.
I could be missing something else too. I should probably document and consolidate the various intended uses somewhere (these have made it on a whiteboard occasionally).
Aside: is the intention to put the speedup factor in the overlay? (e.g.,
16x).Jonathan-Woo commentedon May 15, 2025
This is the edited version with the overlay indicating 16x. It still flickers a bit so the sensitivity governed by the motion will likely need to be tuned more.
Other parts of the pipeline left is the communication with youtube in terms of downloading, uploading, organizing into playlists, etc. Let's sync tomorrow?
edited.mp4
[-]MWE for post-processing videos (still frame removal and speedup)[/-][+]MWE for post-processing videos (still frame removal and speedup) on HF space[/+]14 remaining items
sgbaird commentedon Jun 20, 2025
cc @zweaung1014
sgbaird commentedon Jun 26, 2025
@zweaung1014 I sent you an invite to Hugging Face, once you've accepted the invite I'll give you access to https://huggingface.co/spaces/AccelerationConsortium/Video-Processing. Probably, we'll make it public (ensuring that there aren't any secrets in the history). From there, could you work on replacing the yt-dlp downloading with your playwright implementation?
zweaung1014 commentedon Jun 26, 2025
@sgbaird Turns out, my playwright implementation might also be running into cookie issues. The download sometimes works and sometimes doesn't. It's pretty inconsistent. When i manually download from an incognito browser, it doesn't download. But when I do it from a normal one, it downloads. Since the playwright script opens its own session, it also behaves like an incognito browser. Hence the cookie problem.
I put it in my repo: https://github.com/zweaung1014/yt_download_2FA.git
verify_pyotp.pyis what runs the scriptThe secret code is stored as a "Repository Secret"; not in the code.
zweaung1014 commentedon Jun 26, 2025
The way the script works is, it goes to the link first, and takes the video title. Then, it signs in with 2FA and goes to the YouTube Studio page. On this page, it finds the video title it stored earlier, and clicks it.
I was having a problem at first because I was telling it to click the checkbox, go to "More Actions", and click "Download". But this throws frequent errors due to timing. Clicking the video title first is a much more consistent process.
The last step is to click the Options menu and click "Download".
sgbaird commentedon Jun 26, 2025
Could you use Python to change the string URL to the correct one, rather than clicking to get there?
sgbaird commentedon Jun 26, 2025
It was seeming to work ok in the copilot actions. Not sure if there was something different or it was implying it did but not actually
sgbaird commentedon Jun 26, 2025
Maybe we put this back in the playwright PR discussion
zweaung1014 commentedon Jun 26, 2025
Good point! Let me do that.
sgbaird commentedon Jul 2, 2025
sgbaird commentedon Jul 12, 2025
Like we talked about, Larry - thanks for prioritizing the SDL2 workflows. @Jonathan-Woo is available again to work on this, so he'll pick up from where you left off (thanks for your patience with the relay race).
EDIT: Here's where we left off #343 (comment)
Also, @Jonathan-Woo I think you mentioned there were some ways we may be able to speed up the process? (It would be good to get a sense of the time required to process a single video in general, too). I think you mentioned avoiding encoding the segments that would be removed anyway
Jonathan-Woo commentedon Jul 16, 2025
Yes, so here's how things currently work.
auto-editoris used to detect stale video sectionsffmpegapplies overlayauto-editoris applied to speed-up sectionsCurrently, step 2 naively re-encodes the entire video after adding the overlay which can be accelerated by using stream copy to avoid encoding unedited sections. I'm working on this.
[-]MWE for post-processing videos (still frame removal and speedup) on HF space[/-][+]MWE for post-processing videos (still frame removal and speedup)[/+][-]MWE for post-processing videos (still frame removal and speedup)[/-][+]Post-processing videos (still frame removal and speedup)[/+]sgbaird commentedon Aug 21, 2025
Porting most development and code over to https://github.com/AccelerationConsortium/youtube-livestream-processor, new issues to be raised there.
We may also end up clearing out https://github.com/AccelerationConsortium/ac-training-lab/tree/main/src/ac_training_lab/video_editing since it's redundant now.