Quantized latents?

[stduhpf]

Has anyone sucessfuly experimented with quantizing the latent image tensor to f16 or q8? I guess this could help a lot with generating high resolution images with limited memory, that is if f32 precision isn't needed, of course.

[Green-Sky]

You can check #386 , i did not mention it there, but the flashattention op in ggml only works with f16 k and v, so it is happening there.

[stduhpf]

That's neat. It's not ot quite what I had in mind, but it should also reduce memory usage for DiT models. Though I think this won't affect U-net based models as much?

Edit: It seems to speed up SD2 which is U-net based, so I'm wrong. I thought attention wasn't used in U-nets.