ollama with rocm (AMD GPU) and webui, in podman

ollama-start.sh

podman run --pull newer --detach --security-opt label=type:container_runtime_t --replace --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm; podman run --replace --pull newer -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main

To run it without GPU acceleration, you need to remove a few things in the first podman command, so it looks like this instead:

podman run --pull newer --detach --replace -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama; podman run --replace --pull newer -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main

(Basically removing the SELinux permission stuff, the GPU device paths, and the :rocm tag from the container. The open-webui part stays the same.)

Depending on your GPU, you can use various models. They'll all be pretty quick. On CPU, it'll be slower, but smaller ones are decent, like mistral, phi3, or granite. When downloading models, it'll take a while depending on the size. The smaller ones are quicker and require less RAM (either vRAM for GPU or RAM for CPU).