Repository navigation
Output screen texture to png #1207
Description
What problem does this solve or what need does it fill?
Would enable using Bevy as a general purpose simulation tool for robotics and machine learning research by allowing the screen texture to piped over to a machine learning engine, or for incorporating direct inference in Bevy.
What solution would you like?
The actual solution that would be nice is to have an additional node that can be attached to the render pipeline that will return a png of what is shown on the screen. Being able to scale that also so it's not produced every frame but rather every 10 frames or 100 frames would be helpful for performance issues.
I have tried on my own the following so far:
created in WgpuRenderResourceContext the following code to copy a texture to a buffer:
#[allow(clippy::too_many_arguments)]
pub fn copy_texture_to_buffer(
&self,
command_encoder: &mut wgpu::CommandEncoder,
source_texture: TextureId,
source_origin: [u32; 3], // TODO: replace with math type
source_mip_level: u32,
destination_buffer: BufferId,
destination_offset: u64,
destination_bytes_per_row: u32,
size: Extent3d,
) {
let buffers = self.resources.buffers.read();
let textures = self.resources.textures.read();
let source = textures.get(&source_texture).unwrap();
let destination = buffers.get(&destination_buffer).unwrap();
command_encoder.copy_texture_to_buffer(
wgpu::TextureCopyView {
texture: source,
mip_level: source_mip_level,
origin: wgpu::Origin3d {
x: source_origin[0],
y: source_origin[1],
z: source_origin[2],
},
},
wgpu::BufferCopyView {
buffer: destination,
layout: wgpu::TextureDataLayout {
offset: destination_offset,
bytes_per_row: destination_bytes_per_row,
rows_per_image: size.height,
},
},
size.wgpu_into(),
);
}
and then created in impl RenderResourceContext for WgpuRenderResourceContext the following function to get the buffer out of the gpu:
fn copy_buffer_to_png(&self, buffer_id: BufferId, descriptor: TextureDescriptor) -> () {
let buffers = self.resources.buffers.read();
let buffer = buffers.get(&buffer_id).unwrap();
// reading out from a buffer: https://github.com/gfx-rs/wgpu/issues/239
// now read the buffer
let buffer_slice = buffer.slice(..);
let buffer_future = buffer_slice.map_async(wgpu::MapMode::Read);
self.device.poll(wgpu::Maintain::Wait);
if future::block_on(buffer_future).is_ok() {
let data = buffer_slice.get_mapped_range();
match descriptor.format {
TextureFormat::Bgra8UnormSrgb => {
// saving for testing only, should output to struct and pass that back
image::save_buffer("test.png",
data.as_ref(),
descriptor.size.width,
descriptor.size.height,
image::ColorType::Bgra8);
},
TextureFormat::Depth32Float => {
// todo: convert data to f32 then scale then convert to u16 then convert to &[u8]
/*image::save_buffer("test_depth.png",
data.as_ref(),
descriptor.size.width,
descriptor.size.height,
image::ColorType::???);*/
},
_ => (),
};
drop(data);
buffer.unmap();
}
}
and then tried to implement in impl Node for WindowTextureNode fn update:
if let Some(RenderResourceId::Texture(texture)) = output.get(WINDOW_TEXTURE) {
let render_resource_context = render_context.resources_mut();
let descriptor = self.descriptor;
let width = descriptor.size.width as usize;
let aligned_width =
render_resource_context.get_aligned_texture_size(width);
let format_size = descriptor.format.pixel_size();
println!("{} {:?}", descriptor.format.pixel_info().type_size, descriptor.size);
println!("{:?}", descriptor.format);
let texture_buffer = render_resource_context.create_buffer(BufferInfo {
size: descriptor.size.volume() * format_size,
buffer_usage: BufferUsage::MAP_READ | BufferUsage::COPY_DST,
mapped_at_creation: false,
});
render_context.copy_texture_to_buffer(
texture,
[0, 0, 0],
0,
texture_buffer,
0,
(format_size * aligned_width) as u32,
descriptor.size,
);
let render_resource_context = render_context.resources_mut();
render_resource_context.copy_buffer_to_png(texture_buffer, descriptor);
// remove the created buffer... for now
render_resource_context.remove_buffer(texture_buffer);
}
Unfortunately that is as far as I got as the resulting png image is empty (what comes out is an array of zeros). Ideally this should be it's own node that's attached to the final end of the render pipeline after the screen texture is written.
What alternative(s) have you considered?
Not understanding how the render pipeline works, chose to try window_texture_node.rs as that is the only place with the necessary bevy_render::texture::TextureDescriptor for the screen buffer. From a comment on discord it seems that really the texture to extract is more likely in window_swapchain_node.rs. The problem is, there isn't the necessary information in a TextureDescriptor in that node in order to do the necessary texture -> buffer -> png copying. So some type of information passing from the WindowTextureNode (where the relevant TextureDescriptor is stored) to finally which ever node actually has the access to the final screen texture is needed.
I tried to understand the default render pipeline in base.rs but couldn't make much sense of what was being passed around.
Additional context
The above code causes the game to crash when the window is re-sized.
Additionally, a compute shader that converts the depth buffer to a scaled u16 buffer scaled to match the output from a physical depth camera (like an intel realsense) would be super handy.
Activity
TheRawMeatball commentedon Jan 4, 2021
I have an untested version of this code in this branch. It might be looking over. Also related: #1159
mockersf commentedon Jan 4, 2021
also related: #22
ctangell commentedon Jan 5, 2021
I think Issue #22 is higher priority, or basically the same thing, because it has the advantage of disconnecting the camera from a view screen, and eliminates the bug where re-sizing the window crashes the game.
I'm happy to help out on this issue and get this moving forward, but I would need somebody to walk me through the render pipeline and explain how it works because I spent a day poking at its internals and couldn't make sense of the design decisions, and how all the parts work together.
ctangell commentedon Jan 9, 2021
I started working on this branch, a fork of @TheRawMeatball 's branch where I tried to implement an example based on his code, and fixed some bugs in the code.
The
ReadTextureNodenode currently throws an error where it says the buffer is not large enough to copy in the texture, while the example code (commented out, beneath the mainimpl) worked when I tested it out. The two codes are basically identical now, so I don't know what's wrong there. I could use more experienced eyes on it to figure that out. If I artificially increase the size of the buffer, I then get a different error. Again, the original example code worked, so I don't know what the issue is now.The example I've been working on is
write_to_png.rsin the examples/window folder, which is based off of themultiple_windows.rsexample. I basically just add in aReadTextureNodeto currently read off of theWindowTextureNodenode starting on line 209.It's currently connected to
WindowTextureNode(which has a texture that exists) and notWindowSwapChainNodebecause the texture fromWindowSwapChainNodedoesn't exist by the timeReadTextureNodeis called.Outstanding problems:
WindowSwapChainNodedoesn't exist by the timeReadTextureNoderunsReadTextureNodeand access it in a systemmrk-its commentedon Jan 12, 2021
@ctangell take a look here: https://github.com/mrk-its/bevy/tree/render_to_texture - it adds working 3d/render_to_texture.rs example.
rmsc commentedon Jan 20, 2021
I'm also interested in getting this working.
@ctangell I'm not sure if this still matters, but there's a problem with the code you used to write the PNG file:
The
save_buffer()function is actually failing silently. If you 'unwrap()' the return value you'd get an error, namely that the PNG encoder doesn't support the Bgra8 format.That said, I still haven't figured out how to make it work, and saving a JPEG results in a black image.
@mrk-its That's a great example of rendering to a texture, thanks! I've been playing with this example and trying to save the texture to an image file, but still unsuccessfully.
Not to hijack this issue, but I'm still trying to figure out how the render graph works. There are a few edges in your example that don't seem to be needed for it to work, namely these:
and
Could you please let me know if they're really needed an why? Thanks!
rmsc commentedon Jan 21, 2021
@mrk-its Nevermind, I actually figured out why those edges are needed. I guess I was lucky that it worked without it.
I've finally managed to get it working. I've started with the render_to_texture example, and then implemented a custom render node that inputs the
TEXTURE_NODEand writes it to a file. That node has to depend on theFIRST_PASS, so that it waits for the render to finish.rmsc commentedon Jan 23, 2021
Here's a working (but hacky) example of rendering to a jpeg file: https://github.com/rmsc/bevy/tree/render_to_file
alice-i-cecile commentedon Mar 12, 2021
Summarizing related feedback from
@leonsver1:bevy_renderhas a function that converts fromDynamicImagetoTexture.bevy_renderdepends on should be re-exported, likebevy_mathdoes forglam_rs.Textureshould be methods onTexturefor better visibility.mattdm commentedon Mar 16, 2021
I have a use case which is basically a subset of this. I guess it's similar to the egui issue, but not specific to egui.
I would like to be able to render textures to another texture, like a textureatlas but more generally. (As a one-time thing when called; that texture then shouldn't get re-rendered from its sources.) This is useful in a number of ways, like dynamically generating sprites or panels that don't need to be re-rendered, or generating a map from many tile textures. (Ideally, that texture could then be used in multiple places.)
As I understand the render-to-texture example, that actually does an each-frame render — the texture basically becomes a live view into another world. My need is more like the save-to-file case, except... don't actually write, but instead modify the texture in place.
I'm imagining this working basically like
.add_texture()forTextureAtlas... something likewhich would put the bits from
my_source_texturein thesource_rectwith that image intomy_canvas_textureat the location and scale given bydest_rect.It would be nice for there also to be:
where of course
texture_indexis the index of the actual part of the atlas desired. I guesssource_rectcould be optional in both of these functions, or maybe even omitted in the latter.my_canvas_texturecould then be used as the texture for a Sprite, and drawn in the world.It's my understanding that as currently written, the functions to create a TextureAtlas just use CPU and main memory. So this feature could actually be used to refactor that — TextureAtlas is basically a use-case of what I'm looking for.
There could also be a "blit" version of this which uses fancy material/shader stuff, but I'm a humble old-school 2d person and don't understand any of that. :)
Also beyond my understanding, but: if this is using the same basic rendering pipeline as drawing to the screen (even though only on demand rather than part of the loop), this could then automatically benefit from future improvements like render batching and culling.
I can file this all as a separate RFE if that would be helpful, but it seems everything here is all so related that it might actually be close to being solved as basically a side-effect.
alice-i-cecile commentedon Jul 2, 2021
As a natural extension of this, it would be very nice to be able to natively capture the screen (or a camera view) to a .gif animation file.
ogxd commentedon Mar 7, 2022
Is there a less hacky / simpler solution to read a camera's pixels since? (for saving to a file for instance)
Add screenshot api (bevyengine#7163)