Skip to content

Instantly share code, notes, and snippets.

@fbarretto
Last active February 27, 2025 18:42
Show Gist options
  • Save fbarretto/e61dbf5bb25fd5e98975a42fc7dd5295 to your computer and use it in GitHub Desktop.
Save fbarretto/e61dbf5bb25fd5e98975a42fc7dd5295 to your computer and use it in GitHub Desktop.
StreamDiffusion on a Mac

This is a gist on how to get StreamDiffusion running on a Mac (mps)

  1. Clone the repo

git clone https://github.com/cumulo-autumn/StreamDiffusion.git
  1. Setup the environment

cd StreamDiffusion
python -m venv venv (or python3)
source venv/bin/activate
  1. Install dependencies and StreamDiffusion

pip install --upgrade pip

pip install --pre torch torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu

pip install .          
  1. Edit pipeline.py to support non cuda

  • Go to venv/lib/python3.11/site-packages/streamdiffusion/pipeline.py

Line 439, replace the call function by this:

	@torch.no_grad()
    # condition hack event sync/track for non-cuda devices, RIP profiling etc
    def __call__(
        self, x: Union[torch.Tensor, PIL.Image.Image, np.ndarray] = None
    ) -> torch.Tensor:
        if self.device == "cuda":
            start = torch.cuda.Event(enable_timing=True)
            end = torch.cuda.Event(enable_timing=True)
            start.record()
        if x is not None:
            x = self.image_processor.preprocess(x, self.height, self.width).to(
                device=self.device, dtype=self.dtype
            )
            if self.similar_image_filter:
                x = self.similar_filter(x)
                if x is None:
                    time.sleep(self.inference_time_ema)
                    return self.prev_image_result
            x_t_latent = self.encode_image(x)
        else:
            # TODO: check the dimension of x_t_latent
            x_t_latent = torch.randn((1, 4, self.latent_height, self.latent_width)).to(
                device=self.device, dtype=self.dtype
            )
        x_0_pred_out = self.predict_x0_batch(x_t_latent)
        x_output = self.decode_image(x_0_pred_out).detach().clone()

        self.prev_image_result = x_output
        if self.device == "cuda":
            end.record()
            torch.cuda.synchronize()
            inference_time = start.elapsed_time(end) / 1000
            self.inference_time_ema = 0.9 * self.inference_time_ema + 0.1 * inference_time
        return x_output
  1. Run the demos

txt2img

Make sure you have node installed and npm or pnpn in your path.

  • Edit config.py to set
    torch.device("cuda" if torch.cuda.is_available() else "mps")
    ...
    acceleration: Literal["none", "xformers", "tensorrt"] = "none"
  • Run
cd demo/realtime-txt2img
pip install -r requirements.txt
./start.sh

or

cd demo/realtime-txt2img
pip install -r requirements.txt
cd frontend
pnpm i
pnpm run build
cd ..
python main.py

img2img

  • Edit main.py (line 161)
device = torch.device("cuda" if torch.cuda.is_available() else "mps")
  • Run the demo
cd demo/realtime-img2img
pip install -r requirements.txt
./start.sh
  1. References

[1] cumulo-autumn/StreamDiffusion#34

[2] cumulo-autumn/StreamDiffusion#125

[3] https://developer.apple.com/metal/pytorch/

[4] cumulo-autumn/StreamDiffusion#134

@computationalmama
Copy link

Thanks @fbarretto for this guide - it worked on my m1 air! For Realtime img2img - can you please correct the last code snippet for "Run the demo" to cd demo/realtime-img2img currently it says realtime-txt2img.
And another thing for users from my own test, when using img2img demo, please use localhost:port instead of 0.0.0.0:port as the latter is causing an error in bringing up the webcam!

@fbarretto
Copy link
Author

fbarretto commented May 17, 2024

Thanks @fbarretto for this guide - it worked on my m1 air! For Realtime img2img - can you please correct the last code snippet for "Run the demo" to cd demo/realtime-img2img currently it says realtime-txt2img. And another thing for users from my own test, when using img2img demo, please use localhost:port instead of 0.0.0.0:port as the latter is causing an error in bringing up the webcam!

Thanks for the feedback! I've updated the img2img command.

@yosun
Copy link

yosun commented Jun 30, 2024

what kind of frame rates are u getting?

@fbarretto
Copy link
Author

what kind of frame rates are u getting?

I get around 2.5 fps on a M1 Max.

@yosun
Copy link

yosun commented Jun 30, 2024

ack the huge lag... so i can probably expect no more than 10fps on m3max

@fbarretto
Copy link
Author

Let me know how many FPS you get

@pvjosue
Copy link

pvjosue commented Oct 27, 2024

Got a working example of the img2img demo working on this branch

@urbanonymous
Copy link

Got it working on my M3, havent calculated actual fps, but around 3-4 img2img

@jobeejoba
Copy link

hey folks ! I’ve been trying to set up StreamDiffusion on my Mac, and I’ve encountered a persistent issue. After following the installation steps, I keep running into an error related to cached_download in huggingface_hub.

Specifically, the error says:
ImportError: cannot import name 'cached_download' from 'huggingface_hub'
It seems like there is a conflict with dependancies, librairies (I don't really know...). But I tried different version of Hugging Face, diffusers... But nothing works. I tried to work in Python 3.11 and 3.10 ( actually, I would like to use it with 3.10 because I want to use StreamDiffusion in StreamDiffusion-NDI that requires Python 3.10, according to this page : https://github.com/olegchomp/StreamDiffusion-NDI?tab=readme-ov-file ). That seems so exciting to use StreamDiffusion in a soft like Resolume ! But I can't undertstand why this doesn't work on my MAC. My config is : MACBOOK PRO M1 14" 2021 10 CPU and 16 GPU SONOMA 14.7

Any tiny idea 😊 ???

@jobeejoba
Copy link

Successfully installed Pillow-10.1.0 accelerate-0.24.0 annotated-types-0.7.0 anyio-3.7.1 click-8.1.7 compel-2.0.2 controlnet-aux-0.0.7 einops-0.8.0 exceptiongroup-1.2.2 fastapi-0.104.1 h11-0.14.0 httptools-0.6.4 imageio-2.36.1 lazy-loader-0.4 markdown2-2.5.1 opencv-python-4.10.0.84 peft-0.6.0 pydantic-2.10.2 pydantic-core-2.27.1 pyparsing-3.2.0 python-dotenv-1.0.1 scikit-image-0.24.0 scipy-1.14.1 sniffio-1.3.1 starlette-0.27.0 tifffile-2024.9.20 timm-1.0.11 tokenizers-0.15.2 transformers-4.35.2 uvicorn-0.24.0.post1 uvloop-0.21.0 watchfiles-1.0.0 websockets-14.1

@jobeejoba
Copy link

jobeejoba commented Dec 3, 2024

but : Traceback (most recent call last):
File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/main.py", line 22, in
from img2img import Pipeline
File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/img2img.py", line 12, in
from utils.wrapper import StreamDiffusionWrapper
File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/../../utils/wrapper.py", line 9, in
from diffusers import AutoencoderTiny, StableDiffusionPipeline
File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/init.py", line 5, in
from .utils import (
File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/utils/init.py", line 38, in
from .dynamic_modules_utils import get_class_from_dynamic_module
File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 28, in
from huggingface_hub import HfFolder, cached_download, hf_hub_download, model_info
ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/huggingface_hub/init.py)

@fbarretto
Copy link
Author

but : Traceback (most recent call last): File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/main.py", line 22, in from img2img import Pipeline File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/img2img.py", line 12, in from utils.wrapper import StreamDiffusionWrapper File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/../../utils/wrapper.py", line 9, in from diffusers import AutoencoderTiny, StableDiffusionPipeline File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/init.py", line 5, in from .utils import ( File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/utils/init.py", line 38, in from .dynamic_modules_utils import get_class_from_dynamic_module File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 28, in from huggingface_hub import HfFolder, cached_download, hf_hub_download, model_info ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/huggingface_hub/init.py)

cached_download has been removed from huggingface_hub from 0.26. Maybe if you downgrade it to 0.25 it will work. Try this:

pip install huggingface_hub==0.25.0

@setsun
Copy link

setsun commented Dec 18, 2024

but : Traceback (most recent call last): File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/main.py", line 22, in from img2img import Pipeline File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/img2img.py", line 12, in from utils.wrapper import StreamDiffusionWrapper File "/Users/jonathanlandais/StreamDiffusion/demo/realtime-img2img/../../utils/wrapper.py", line 9, in from diffusers import AutoencoderTiny, StableDiffusionPipeline File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/init.py", line 5, in from .utils import ( File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/utils/init.py", line 38, in from .dynamic_modules_utils import get_class_from_dynamic_module File "/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 28, in from huggingface_hub import HfFolder, cached_download, hf_hub_download, model_info ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/Users/jonathanlandais/StreamDiffusion/venv/lib/python3.10/site-packages/huggingface_hub/init.py)

cached_download has been removed from huggingface_hub from 0.26. Maybe if you downgrade it to 0.25 it will work. Try this:

pip install huggingface_hub==0.25.0

^ confirmed the above worked for me, thanks for investigating this @fbarretto 🙏

@jobeejoba
Copy link

Sorry, for not answering. Yes this works !
Thank's FBarreto

@pbmagnet4
Copy link

pbmagnet4 commented Jan 3, 2025

I am getting the same error after applying pip install huggingface_hub==0.25.0

Anybody able to resolve this issue?

ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/huggingface_hub/init.py)

@fbarretto
Copy link
Author

fbarretto commented Jan 8, 2025

I am getting the same error after applying pip install huggingface_hub==0.25.0

Anybody able to resolve this issue?

ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/huggingface_hub/init.py)

@pbmagnet4 Are you using any Python virtual environments? Are you installing it system-wide? Run pip list and check huggingface_hub installed version

@glxgls
Copy link

glxgls commented Jan 18, 2025

hi!!
I managed to install everything following your instructions! (Thanks omg). However, when starting the stream, I get an error related to pin_memory. (RuntimeError: Need to provide pin_memory allocator to use pin memory.)

I’ve been trying to fix it with chatGPT, but we couldn’t figure it out uwu. Do you have any ideas? I’m trying to use Stream Diffusion TOX.
I edited main_sdtd.py to remove "pin_memory=True", but when I run the stream, the code gets modified again, and it reappears

@Pongomotion
Copy link

Hi guys !

Managed to get it all installed and running but now I ;m having this error on the web portal... any ideas?
thanks in advance!
Screenshot 2025-01-30 at 17 31 21

Ive tried
http://0.0.0.0:7860
and localhost:port

Unsuccessfully :(

@fbarretto
Copy link
Author

hi!! I managed to install everything following your instructions! (Thanks omg). However, when starting the stream, I get an error related to pin_memory. (RuntimeError: Need to provide pin_memory allocator to use pin memory.)

I’ve been trying to fix it with chatGPT, but we couldn’t figure it out uwu. Do you have any ideas? I’m trying to use Stream Diffusion TOX. I edited main_sdtd.py to remove "pin_memory=True", but when I run the stream, the code gets modified again, and it reappears

Sorry, I don't have an answer for that rn. This is a gist to run the original StreamDiffusion github repo though. I did mess with the tox but I haven't played with it in the last couple of weeks. If you have access to Dotsimulate discord channel check this post: https://discord.com/channels/1026335703598104576/1233197067728457758

@fbarretto
Copy link
Author

Hi guys !

Managed to get it all installed and running but now I ;m having this error on the web portal... any ideas? thanks in advance! Screenshot 2025-01-30 at 17 31 21

Ive tried http://0.0.0.0:7860 and localhost:port

Unsuccessfully :(

Hey! Do you also have additional info or messages from the terminal? Apparently, it is an issue on the frontend side (https://github.com/cumulo-autumn/StreamDiffusion/blob/b623251dc055e1fd858d53509aa43e09dfc5cdc0/demo/realtime-img2img/frontend/src/lib/mediaStream.ts#L15)

@jolc
Copy link

jolc commented Feb 4, 2025

Hi guys !

Managed to get it all installed and running but now I ;m having this error on the web portal... any ideas? thanks in advance! Screenshot 2025-01-30 at 17 31 21

Ive tried http://0.0.0.0:7860 and localhost:port

Unsuccessfully :(

I saw the same error on http://0.0.0.0:7860.
works using http://localhost:7860

Screenshot 2025-02-04 at 16 16 23 Screenshot 2025-02-04 at 16 16 37

does that help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment