Repository navigation
Migrate Bevy Asset System to Distill #708
Description
this is a tracking issue for integrating atelier-assets into bevy
I've been working with "resident asset-expert" @kabergstrom for awhile now to figure out the future of the Bevy asset system. @kabergstrom built atelier-assets, which has basically everything a pro-grade engine needs from an asset system. While learning atelier, I decided to explore the space for a bit by experimenting with directly expanding the current bevy_asset with the features we were interested in. After much discussion about adopting atelier-assets vs expanding the current system, we agreed that atelier was the best long term bet. We merged a subset of my "bevy asset rework" in #693, just to unblock scenarios like GLTF loading and to prepare the way for an atelier migration.
Changes required for a merge
That being said, atelier needs a few features before it can be a drop in replacement for the current system. Atelier support shouldn't be merged until it has the following features:
- A
bevy_assetintegration layer- This should be something like the current
AssetServer,Assets<T>, andHandle<T>. Zero-api breakage is a nice-to-have, but I'm cool with breaking apis if they are significant improvements or they significantly improve the integration clarity / complexity.
- This should be something like the current
- All current Bevy AssetLoaders ported to Atelier Assets
- Loading assets via their paths (Added support for Indirect handles & reworked Loader APIs amethyst/distill#64)
- Wasm publishing support (likely via pack files)
- Currently, the only way to publish a game with the tools atelier provides is to host a daemon server in a separate process from the game. Wasm (and potentially other future platforms) can't host an in-process daemon server. We need some way to publish games on platforms like wasm. @kabergstrom is planning on adding "packfile" loading, which would drop imported assets into a "packfile" which could then be directly loaded (without a daemon server). This should make WASM support easier, but its also just a nice way to package an app on any platform.
- Android publishing support (likely via pack files, but a daemon server might also work)
- iOS publishing support (likely via pack files, but a daemon server might also work)
- Bevy Task Features
- Support for spawn_local and !Send + !Sync futures (Support for !Send tasks #1216)
- file and TCP IO, maybe with async-fs+async-net?
- for web, probably want websocket support for Daemon connectivity
Given that Bevy's current asset system works for most current platforms/user-facing scenarios, I don't want to merge Atelier until we are sure it won't significantly block progress on things like building out platform support or publishing games.
Changes that are "nice to have"
There are also some "nice to have" items that I won't necessarily block a merge on, but should be addressed as soon as possible:
- Reduce the number of dependencies used by atelier as much as possible / cutting clean compile time
- This should include removing large or unnecessary dependencies, but it will also include aligning dependency versions as much as possible. I'm leaving this open-ended because theres a lot to discuss here and plenty of details that need to be handled on a case-by-case basis. For example, Atelier currently uses Tokio, which does introduce a decent amount of dependencies. But Bevy currently doesn't have anything that handles "async io", so theres no way out of adding new dependencies there. We will evaluate all of the options in that space and pick the one that fits our constraints the best.
Changes without any time pressure
And then there are some fun things that have no particular timeline and will likely just get spun off into their own issues:
- Wasm asset hot-reloading via something like websockets or WebRTC
Activity
[-]Migrate Bevy asset system to Atelier Assets[/-][+]Migrate Bevy Asset System to Atelier Assets[/+]zicklag commentedon Oct 21, 2020
I'm not sure about how stuck Atelier is on Tokio, but the
smolasync runtime is pretty amazing and ridiculously tiny. IMHO it's the best design for an async executor out there, but there could very well be reasons that Atelier can't swtich away from Tokio, and that's probably the topic of another thread.cart commentedon Oct 21, 2020
Yeah we've talked a bit about that already. It's definitely on the table!
chemicstry commentedon Oct 21, 2020
Eagerly waiting for the integration!
One note on wasm. While pack files are a simple option for small games, this would not work well for larger ones if each window refresh redownloads 1gb or more of assets. Larger web-based games use local storage to fetch assets just once. This would require some kind of cache layer, which looks for file in local storage and if it's not available uses HTTP to fetch it. If file-level caching is too difficult to implement, then maybe caching pack files would be easier (or just leave it up to the user to download it and provide in-memory file). But I believe with atelier "import and process before load" caching might be easy to implement. Which IIRC was one of the atelier goals to compile a fast asset storage optimized for production (without daemon). Maybe @kabergstrom could comment on this.
kabergstrom commentedon Oct 21, 2020
I want to move atelier-assets to bevy_task, which is based on crates also used by smol. There are a few things missing in bevy_task right now though, preventing the migration:
Would be happy if someone helped on these parts.
kabergstrom commentedon Oct 21, 2020
I think web distribution can be done in a ton of ways, and hopefully it'll be simple to implement different approaches. Would love contributions on this front. The Loader API (LoaderIO) is here:
https://github.com/amethyst/atelier-assets/blob/indirect_handles/loader/src/io.rs
For the packaging step, you can imagine just a function that takes
&[(AssetMetadata, ArtifactMetadata, Vec<u8>)]and outputs whatever is loaded by the LoaderIO impl.Moxinilian commentedon Nov 2, 2020
I would be interested in helping to port current loaders to atelier-assets. Has this effort started? If not, where should it be hosted?
kabergstrom commentedon Nov 2, 2020
I started on an integration here: https://github.com/kabergstrom/bevy/tree/master/crates/bevy_atelier
It's based on pre-asset-rework code though, so might want to just start over.
If you want to help with porting loaders, you can probably pull in
atelier-assetsas a dependency and port them to work with theImportertrait.alec-deason commentedon Jan 6, 2021
This is added by #1216
cart commentedon Jan 7, 2021
Just updated the todo list in the description / added more bullet points. I'll try to keep that up to date from now on.
alec-deason commentedon Jan 7, 2021
@kabergstrom said on discord that task locals are no longer necessary, so that can probably come off the list
[-]Migrate Bevy Asset System to Atelier Assets[/-][+]Migrate Bevy Asset System to Distill[/+]allsey87 commentedon Aug 31, 2021
Perhaps this issue is in need of being updated now that the Tokio dependency has been removed from distill?
cart commentedon Aug 31, 2021
Good call. Just checked the "reduce dependencies" box. I'll give this another pass after Bevy 0.6 drops and try to make a final call on asset system direction.
cart commentedon Aug 31, 2021
While we wait, if anyone is interested in getting Distill working in Bevy, I recommend (initially) doing it as a 3rd-party crate so progress isn't blocked on official reviews and interested users can start experimenting with it.
B-Reif commentedon Apr 22, 2023
I assume this is off the table now, right?
cart commentedon Apr 24, 2023
Less likely at this point, but still technically on the table. I'll be putting out my proposal MVP (both code and design outline) in the very near future. I believe my impl is the right path for Bevy, but I want other peoples' opinions.
Bevy Asset V2 (#8624)
cart commentedon Sep 9, 2023
Closing in favor of #8624
Bevy Asset V2 (bevyengine#8624)
Bevy Asset V2 (#8624)