Chroma Radiance (v0.4) #807

Activity

1. Green-Sky commented on Sep 9, 2025

   

   Green-Sky

   on Sep 9, 2025

   ContributorAuthor

   More actions

   Say good bye to VAE.
2. FSSRepo commented on Sep 10, 2025

   

   FSSRepo

   on Sep 10, 2025

   Last edited by FSSRepo

   Contributor

   More actions

   @Green-Sky Is this an alternative to VAEs or an image generation model? I think the best approach would be to implement this alternative applied to the existing models to distinguish the differences (replace the AutoEncoder to this, SDXL, SD1.5). Also, the size (14 GB) of this new alternative catches my attention—I hope it’s perhaps because these are training checkpoints that include the gradients.
3. stduhpf commented on Sep 10, 2025

   

   stduhpf

   on Sep 10, 2025

   Contributor

   More actions

   @Green-Sky Is this an alternative to VAEs or an image generation model?

   I think it's a pixel-space diffusion model (or rather pixel-space flow). So there is no need for a VAE.

   👍React with 👍1Reacted by Erik Scholz
4. Green-Sky commented on Sep 11, 2025

   

   Green-Sky

   on Sep 11, 2025

   ContributorAuthor

   More actions

   Yes, @stduhpf is right. It is an alternative approach, that does not need a VAE, but requires a full finetune.
   If I understood it correctly, it works by reducing the pixel space using DCT, the tech that powers jpeg.
   Here is a section from @lodestone-rock 's reddit chrome release post ( https://old.reddit.com/r/StableDiffusion/comments/1mxwr4e/update_chroma_project_training_is_finished_the/ ):

   My Beef with GAN training.

   GAN is notoriously hard to train and also expensive! It’s so unstable even with a shit ton of math regularization and another mumbojumbo you throw at it. This is the reason behind 2 of the research branches: Radiance is to remove the VAE altogether because you need a GAN to train it, and Flash is to get a few-step speed without needing a GAN to make it fast.

   The instability comes from its core design: it's a min-max game between two networks. You have the Generator (the artist trying to paint fakes) and the Discriminator (the critic trying to spot them). They are locked in a predator-prey cycle. If your critic gets too good, the artist can't learn anything and gives up. If the artist gets too good, it fools the critic easily and stops improving. You're trying to find a perfect, delicate balance but in reality, the training often just oscillates wildly instead of settling down.

   GANs also suffer badly from mode collapse. Imagine your artist discovers one specific type of image that always fools the critic. The smartest thing for it to do is to just produce that one image over and over. It has "collapsed" onto a single or a handful of modes (a single good solution) and has completely given up on learning the true variety of the data. You sacrifice the model's diversity for a few good-looking but repetitive results.

   Honestly, this is probably why you see big labs hand-wave how they train their GANs. The process can be closer to gambling than engineering. They can afford to throw massive resources at hyperparameter sweeps and just pick the one run that works. My goal is different: I want to focus on methods that produce repeatable, reproducible results that can actually benefit everyone!

   That's why I'm exploring ways to get the benefits (like speed) without the GAN headache.

   The Holy Grail of the End-to-End Generation!

   Ideally, we want a model that works directly with pixels, without compressing them into a latent space where information gets lost. Ever notice messed-up eyes or blurry details in an image? That's often the VAE hallucinating details because the original high-frequency information never made it into the latent space.

   This is the whole motivation behind Chroma1-Radiance. It's an end-to-end model that operates directly in pixel space. And the neat thing about this is that it's designed to have the same computational cost as a latent space model! Based on the approach from the PixNerd paper, I've modified Chroma to work directly on pixels, aiming for the best of both worlds: full detail fidelity without the extra overhead. Still training for now but you can play around with it.
5. Green-Sky

   changed the title [-]Chroma Radiance (upcoming)[/-] [+]Chroma Radiance (v0.1)[/+] on Sep 12, 2025
6. FSSRepo commented on Sep 12, 2025

   

   FSSRepo

   on Sep 12, 2025

   Contributor

   More actions

   From what I understand, it doesn’t need a latent image; instead, it works directly on the pixels of an image. In other words, whereas a VAE transforms a latent image into an RGB image, with this new alternative that step isn’t necessary—the diffusion model receives the image pixels directly.

   👍React with 👍2Reacted by Erik Scholz and Shareef
7. stduhpf commented on Oct 16, 2025

   

   stduhpf

   on Oct 16, 2025

   Last edited by stduhpf

   Contributor

   More actions

   I looked a bit deeper into how it works and it's not exactly doing diffusion on pixels directly, it's grouping the pixels into 16x16 "latent" patches with a convolution layer, and then it's decoding these patches back to pixels using a NeRF (Neural Radiance Field ?) instead of a VAE. I don't fully understand what it all exactly means yet.

   Edit: Okay, so the trick is that the output from the DiT is no longer latent pixels, but the weights of a simple MLP that would denoise the corresponding image patch.

   👀React with 👀1Reacted by Erik Scholz
8. Green-Sky commented on Oct 16, 2025

   

   Green-Sky

   on Oct 16, 2025

   Last edited by Green-Sky

   ContributorAuthor

   More actions

   Another cool thing I saw ppl mention, is that loras for Chroma (non-radiance) are somewhat compatible with the the radiance version. But I also saw the Chroma Radiance training is getting continued, we are now at v0.4 :)

   Arch diagram from the repo:

   

   (start at the bottom right)
9. Green-Sky

   changed the title [-]Chroma Radiance (v0.1)[/-] [+]Chroma Radiance (v0.4)[/+] on Oct 16, 2025
10. leejet commented on Oct 22, 2025

    

    leejet

    on Oct 22, 2025

    Owner

    More actions

    stable-diffusion.cpp now supports Chroma Radiance #910.

    🚀React with 🚀2Reacted by stduhpf and Erik Scholz
11. leejet

    closed this as completedon Nov 13, 2025
12. Repository owner locked and limited conversation to collaborators on Nov 13, 2025
13. leejet

    converted this issue into a discussion #971 on Nov 13, 2025