stable-diffusion-links: useful optimizations

stable-diffusion-links.md

📑 links.sd index

links to useful SD mods and tools

table of contents:

1. optimizations
   1. notes about specific card architectures/families
   2. how to apply the optimizations
2. tools
3. video generation

(link ids are adler32-style sha-1 hashes of the URL)

optimizations

1. https://github.com/CompVis/stable-diffusion/compare/main...Doggettx:stable-diffusion:main

   • link id: d8ad218c
   • 🚀 time/speed optimized
   • Doggettx memory optimization: allows rendering 1024x1024 images and even up to 4096x1024

2. https://github.com/hlky/stable-diffusion/commit/8c885b480055ff4250fad6967a7fca02bd58e152

   • link id: 6a001cf2
   Details

   • outdated, replaced by 459b1456 below

   • alternate link: CompVis/stable-diffusion#177
   • neonsecret's optimizations (seems less extensive than Doggettx's, but still saving it here)

3. https://github.com/basujindal/stable-diffusion/pull/122

   • link id: 459b1456
   • 🧠 VRAM/memory optimized
   • improvement on neonsecret's optimizations
   • merged on 2022-09-07

Notes about specific card architectures/families

NVIDIA Pascal (Quadro P1000, Tesla P40, GTX 1xxx series e.g. GTX 1080)

1. For NVIDIA Pascal GPUs, stable-diffusion is faster in full-precision mode (fp32), not half-precision mode (fp16)!

How to apply the optimizations

Hard mode

1. Open the "Files changed" view in the PR/diff and modify/add the listed files in your copy of stable-diffusion.

Easier mode

1. Open the "Files changed" view in the PR/diff and at the top right of every file, there is a 3-dots menu (•••).
2. Click "View file" in that 3-dots menu. You will be taken to an individual file view.
3. At the top of the individual file view, click the "Raw" button.
4. Save the page as the filename specified in the PR, under the path specified in the PR (but starting from the root directory of your stable-diffusion installation).

tools

1. https://github.com/pharmapsychotic/clip-interrogator
   • link id: 1023145e
   • 🔍 CLIP interrogator
   • determines an approximate text prompt (with style) for a given image
   • replicate runpage: https://replicate.com/methexis-inc/img2prompt
2. https://gist.github.com/Filarius/2c30348cca9266d7dc3a171a1d197e17
   • link id: e58f14c6
   • 🗜 convert Stable Diffusion weights file into float16 (half precision)
   • by Filarius

krita plugins

1. https://github.com/sddebz/stable-diffusion-krita-plugin
   • link id: 446f1509
   • based on AUTOMATIC1111's webUI
   • requires Krita 5.1
2. https://github.com/nousr/koi
   • link id: 98450a94
   • pytorch-based krita plugin

video generation

1. https://github.com/deforum/stable-diffusion
   • link id: 6415107b
   • 🎥 adds the ability to render videos
   • replicate runpage: https://replicate.com/deforum/deforum_stable_diffusion/versions/e22e77495f2fb83c34d5fae2ad8ab63c0a87b6b573b6208e1535b23b89ea66d6

In addition, it may make sense with both @mchaker and the Doggettx mods to have some kind of memory threshold or image dimension size based on GPU VRAM be used to determine if "slow and steady" mode kicks in or not?

I am new to using git. How do I implement the Doggettx optimizations or the basujindal?

CompVis/stable-diffusion#177 worked very well for me

This one is also interesting CompVis/stable-diffusion#142

This seems to utilize CPU -- what stands out about it to you @ilikenwf ?

In addition, it may make sense with both @mchaker and the Doggettx mods to have some kind of memory threshold or image dimension size based on GPU VRAM be used to determine if "slow and steady" mode kicks in or not?

I think that is how the Doggettx optimizations work

I am new to using git. How do I implement the Doggettx optimizations or the basujindal?

For the Doggettx optimizations, replace files in your local copy of stable-diffusion with the files that were changed in the linked PR.

I've added some instructions above to clarify.

CompVis/stable-diffusion#177 worked very well for me

This looks exactly like the changes in item (2) in my list, except that they are in PR format. I will add that as an alternate link. Thank you.

Maybe you want to add basujindal/stable-diffusion#122 I made a pull request to improve upon neonsecret's

Do you think you could come up with a version of it that works with Doggettx's optimizations?

Currently, the PR I mentioned uses less memory (VRAM) than Doggetx's for the same generation time (at least for a 1024 image), so I'd just use that. If you can tell me a specific aspect of their optimization that I should include, I'll consider implementing it. Otherwise, it's just too much work to go comb through the details, and compare all the changes.

Thanks for this. I was previously using the tweak from neonsecret and was able to generate up to 1024x640 images on 8GB; however, this came at the cost of speed where it took multiple seconds per iteration due to the attention splitting.

Results for 512x512 default parameters
Baseline code from following this guide JoshuaKimsey/Linux-StableDiffusion-Script@120a13b :

• 6921MB peak
• 5.54it/s

Using attention.py from https://github.com/basujindal/stable-diffusion/pull/122/files:

• 5992MB peak
• 5.01it/s
• Can generate 1024x640 using 8132MB peak

Will update this once I add in Doggettx tweaks

System: Win10 with wsl2 Ubuntu 22.04, i7 11800H, 16GB of RAM, 3070 mobile 8GB

Quick question!

Can we apply the speed time as well as lower vRam mod both at the same time?

@mrpixelgrapher I have not tried that yet but it looks like some of the changes overlap. I'm not sure if it's possible to combine both approaches -- but perhaps there is and I just don't know enough math to do it 😅