Dave's Ground Truth Blog: DGX Spark Local LLM May 2026 Setup and Experiment Details

gistfile1.txt

For the blog, please check https://www.davesgroundtruth.com/

# Setup and Experiment Details

1. Set up each DGX Spark individually using the official NVIDIA getting started and made sure they were both fully up to date through the dashboard.
1. For the two-node setup, follow NVIDIA's Connect Two Sparks playbook: https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/connect-two-sparks
   - Aligned the usernames on both systems, connected the two Sparks with one QSFP cable, used the automatic IP assignment path, and verified the link and SSH between the nodes.
1. Make sure your user is in the `docker` group. If you run `docker ps` and get a permission error:
   ```bash
   sudo usermod -aG docker $USER
   newgrp docker
   ```


# Gemma 4 31B-it on a Single Node

1. Following from https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/vllm
1. Download the vLLM Docker container specifically from: https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html
   ```bash
   docker pull vllm/vllm-openai:gemma4-cu130
   ```
1. Launch the container:
    ```bash
    docker run -d --gpus all -p 8000:8000 \
      -v ~/.cache/huggingface:/root/.cache/huggingface \
      vllm/vllm-openai:gemma4-cu130 \
      google/gemma-4-31B-it \
      --max-model-len 120k \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser gemma4 \
      --reasoning-parser gemma4 \
      --chat-template examples/tool_chat_template_gemma4.jinja
    ```
1. Watch startup logs:
  ```bash
  docker logs -f <container name>
  ```
1. Stop the container:
   ```bash
   # To get containers on the machine
   docker ps -a --size
   docker stop <container_name_or_id>
   ```
1. Cleanup
   ```bash
   docker rm <container_name_or_id>
   # Remove the image
   docker image ls -a
   docker rmi <image_name_or_id>
   ```

# For other Models on a Single Node

Replace the "Launch the container" step with the new model name and parameters. 

## Qwen 3.6

```bash
docker run -d --name qwen36-vllm --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint vllm \
  vllm/vllm-openai:gemma4-cu130 \
  serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --max-model-len 120000 \
    --max-num-seqs 1 \
    --gpu-memory-utilization 0.90 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
```


# Distributed Across Two Nodes

These commands are based on the [DGX Spark vLLM playbook's](https://github.com/NVIDIA/dgx-spark-playbooks) two-Spark instructions. You should follow [Connect Two Sparks](https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/connect-two-sparks) before trying the below.

Values changed for this setup:

- `MN_IF_NAME` is `enp1s0f0np0`, because that is the QSFP interface that is up on both Sparks.
- Spark 1/head `VLLM_HOST_IP` is `<HEAD_QSFP_IP>`.
- Spark 2/worker `VLLM_HOST_IP` is `<WORKER_QSFP_IP>`.
- Worker `HEAD_NODE_IP` and `MASTER_ADDR` are `<HEAD_QSFP_IP>`.
- vLLM image is set to `nvcr.io/nvidia/vllm:26.03.post1-py3`. The newer `26.04-py3` looks tempting (it ships upstream vLLM 0.19.0 with Gemma 4 support), but NVIDIA dropped Ray from that image, so `run_cluster.sh` fails immediately with `ray: command not found`. Stay on `26.03.post1-py3` for the NVIDIA-image-based Nemotron and MiniMax runs. For Gemma 4 distributed, use the separate `vllm/vllm-openai:gemma4-cu130`, described below.
- Ray usage stats are disabled with `RAY_USAGE_STATS_ENABLED=0`.
- Ray dashboard is exposed on the head node at `0.0.0.0:8265`, which should make it available at `http://<HEAD_LAN_IP>:8265` on the local network.
- Nemotron 3 Super requires `--trust-remote-code` because the Hugging Face repo contains custom model code.

To find the placeholder values for your own setup:

- `<HEAD_QSFP_IP>` and `<WORKER_QSFP_IP>` are the link-local IPs (typically `169.254.x.y`) that the connect-two-sparks playbook auto-assigns to the QSFP interface. Run `ip -4 addr show enp1s0f0np0` on each Spark and read the `inet` address. Pick one Spark to be the head; its QSFP IP is `<HEAD_QSFP_IP>` and the other Spark's is `<WORKER_QSFP_IP>`.
- `<HEAD_LAN_IP>` is whatever IP the head Spark has on your regular local network (the one you SSH in over, not the QSFP link). Run `hostname -I` on the head node and pick the address on your LAN subnet.

### Step 1: Download the cluster script on both Sparks

Run on both Spark 1 and Spark 2:

```bash
mkdir ~/vllm-two-spark
cd ~/vllm-two-spark
wget https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/examples/online_serving/run_cluster.sh
chmod +x run_cluster.sh
```

Patch the local script so the Ray dashboard binds to all interfaces on the head node. Run this on both Sparks so both copies are consistent:

```bash
cd ~/vllm-two-spark
cp --update=none run_cluster.sh run_cluster.original.sh
grep -q -- '--dashboard-host=0.0.0.0' run_cluster.sh || \
  sed -i 's/--head --node-ip-address=${HEAD_NODE_ADDRESS} --port=6379/--head --node-ip-address=${HEAD_NODE_ADDRESS} --port=6379 --include-dashboard=true --dashboard-host=0.0.0.0 --dashboard-port=8265/' run_cluster.sh
```

Only expose the Ray dashboard on a trusted network. Ray dashboard is useful for local debugging, but do not expose it directly to the public internet.

### Step 2: Pull the vLLM image on both Sparks

Run on both Spark 1 and Spark 2:
Pulling the image from: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm/tags?version=26.03.post1-py3

```bash
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.03.post1-py3

docker pull "$VLLM_IMAGE"
```


### Step 3: Bring up the Ray Cluster

Run on Spark 1 (head node):

```bash
cd ~/vllm-two-spark

nohup bash run_cluster.sh \
  nvcr.io/nvidia/vllm:26.03.post1-py3 \
  <HEAD_QSFP_IP> \
  --head \
  ~/.cache/huggingface \
  -e RAY_USAGE_STATS_ENABLED=0 \
  -e VLLM_HOST_IP=<HEAD_QSFP_IP> \
  -e UCX_NET_DEVICES=enp1s0f0np0 \
  -e NCCL_SOCKET_IFNAME=enp1s0f0np0 \
  -e OMPI_MCA_btl_tcp_if_include=enp1s0f0np0 \
  -e GLOO_SOCKET_IFNAME=enp1s0f0np0 \
  -e TP_SOCKET_IFNAME=enp1s0f0np0 \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e RAY_CGRAPH_get_timeout=1800 \
  -e RAY_CGRAPH_submit_timeout=1800 \
  -e MASTER_ADDR=<HEAD_QSFP_IP> \
  > ~/ray-head.log 2>&1 &
disown
```

Run on Spark 2 (worker). The only differences from the head are `--worker`, `VLLM_HOST_IP=<WORKER_QSFP_IP>` (Spark 2's QSFP IP), and the log file name:

```bash
cd ~/vllm-two-spark

nohup bash run_cluster.sh \
  nvcr.io/nvidia/vllm:26.03.post1-py3 \
  <HEAD_QSFP_IP> \
  --worker \
  ~/.cache/huggingface \
  -e RAY_USAGE_STATS_ENABLED=0 \
  -e VLLM_HOST_IP=<WORKER_QSFP_IP> \
  -e UCX_NET_DEVICES=enp1s0f0np0 \
  -e NCCL_SOCKET_IFNAME=enp1s0f0np0 \
  -e OMPI_MCA_btl_tcp_if_include=enp1s0f0np0 \
  -e GLOO_SOCKET_IFNAME=enp1s0f0np0 \
  -e TP_SOCKET_IFNAME=enp1s0f0np0 \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e RAY_CGRAPH_get_timeout=1800 \
  -e RAY_CGRAPH_submit_timeout=1800 \
  -e MASTER_ADDR=<HEAD_QSFP_IP> \
  > ~/ray-worker.log 2>&1 &
disown
```

Watch the log on either Spark with `tail -f ~/ray-head.log` (or `~/ray-worker.log`) until the node has joined. Verify the cluster from Spark 1:

```bash
docker exec "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" ray status
```


## Step 4: Start vLLM on the head node

```bash
# Per-model knobs. Edit these four lines per model.
MODEL_HANDLE=nvidia/MiniMax-M2.7-NVFP4
MAX_MODEL_LEN=120000
TOOL_CALL_PARSER=minimax_m2
REASONING_PARSER=minimax_m2

docker exec -d \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -e MAX_MODEL_LEN="$MAX_MODEL_LEN" \
  -e TOOL_CALL_PARSER="$TOOL_CALL_PARSER" \
  -e REASONING_PARSER="$REASONING_PARSER" \
  -e SAFETENSORS_FAST_GPU=1 \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  bash -c '
    vllm serve "$MODEL_HANDLE" \
      --tensor-parallel-size 2 \
      --max-model-len "$MAX_MODEL_LEN" \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser "$TOOL_CALL_PARSER" \
      --reasoning-parser "$REASONING_PARSER" \
      --trust-remote-code \
      > /tmp/vllm-serve.log 2>&1
  '
```

Watch startup logs with:
```bash
docker exec \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  tail -f /tmp/vllm-serve.log
```

### Step 5: Restart vLLM with different flags (cluster stays up)

To swap models or change vLLM flags without tearing down Ray, kill only the `vllm serve` process inside the container, then re-run Step 2 with new values:

```bash
docker exec \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  pkill -f 'vllm serve'
```

### Step 6: Stop everything

```bash
docker stop "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')"
```

Inspect the model cache:

```bash
du -sh ~/.cache/huggingface
```


## Specific Model vLLM Commands/Differences

### Gemma 4 31B (Full Model)

The distributed run uses a different cluster image from the Nemotron and MiniMax runs. The NVIDIA NGC image `nvcr.io/nvidia/vllm:26.03.post1-py3` ships vLLM 0.17.1 (no Gemma 4 support), and the newer `26.04-py3` drops Ray entirely, breaking `run_cluster.sh`. The upstream `vllm/vllm-openai:gemma4-cu130` has Gemma 4 support, both `gemma4` parsers, and `/vllm-workspace/examples/tool_chat_template_gemma4.jinja`, but it does not include Ray either, so `run_cluster.sh` still fails on it out of the box. The fix is to build a thin derived image that adds `ray[default]` on top.

If the Nemotron/MiniMax Ray cluster is still running, stop it first on both Sparks:

```bash
docker stop "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')"
```

Build the derived image on both Sparks:

```bash
mkdir -p ~/gemma4-ray && cd ~/gemma4-ray

cat > Dockerfile <<'DOCKER'
FROM vllm/vllm-openai:gemma4-cu130
RUN pip install --no-cache-dir "ray[default]>=2.48"
DOCKER

docker build -t local/vllm-gemma4-ray:cu130 .
```

Then re-run the Step 3 Ray cluster bring-up from above, but pass `local/vllm-gemma4-ray:cu130` as the image argument to `run_cluster.sh` on both the head and worker (in place of `nvcr.io/nvidia/vllm:26.03.post1-py3`). Once `ray status` shows both nodes joined, launch vLLM on the head:

```bash
MODEL_HANDLE=google/gemma-4-31B-it
MAX_MODEL_LEN=64000
TOOL_CALL_PARSER=gemma4
REASONING_PARSER=gemma4

docker exec -d \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -e MAX_MODEL_LEN="$MAX_MODEL_LEN" \
  -e TOOL_CALL_PARSER="$TOOL_CALL_PARSER" \
  -e REASONING_PARSER="$REASONING_PARSER" \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  bash -c '
    vllm serve "$MODEL_HANDLE" \
      --tensor-parallel-size 2 \
      --distributed-executor-backend ray \
      --enforce-eager \
      --max-model-len "$MAX_MODEL_LEN" \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser "$TOOL_CALL_PARSER" \
      --reasoning-parser "$REASONING_PARSER" \
      --chat-template /vllm-workspace/examples/tool_chat_template_gemma4.jinja \
      > /tmp/vllm-serve.log 2>&1
  '
```


### Gemma 4 31B (NVFP4)

```bash
MODEL_HANDLE=nvidia/Gemma-4-31B-IT-NVFP4
MAX_MODEL_LEN=64000
TOOL_CALL_PARSER=gemma4
REASONING_PARSER=gemma4

docker exec -d \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -e MAX_MODEL_LEN="$MAX_MODEL_LEN" \
  -e TOOL_CALL_PARSER="$TOOL_CALL_PARSER" \
  -e REASONING_PARSER="$REASONING_PARSER" \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  bash -c '
    vllm serve "$MODEL_HANDLE" \
      --tensor-parallel-size 2 \
      --distributed-executor-backend ray \
      --enforce-eager \
      --max-model-len "$MAX_MODEL_LEN" \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser "$TOOL_CALL_PARSER" \
      --reasoning-parser "$REASONING_PARSER" \
      --chat-template /vllm-workspace/examples/tool_chat_template_gemma4.jinja \
      > /tmp/vllm-serve.log 2>&1
  '
```


### Nemotron 3 Super 120B

```bash
MODEL_HANDLE=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
MAX_MODEL_LEN=120000
TOOL_CALL_PARSER=qwen3_coder
REASONING_PARSER=nemotron_v3

docker exec -d \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -e MAX_MODEL_LEN="$MAX_MODEL_LEN" \
  -e TOOL_CALL_PARSER="$TOOL_CALL_PARSER" \
  -e REASONING_PARSER="$REASONING_PARSER" \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  bash -c '
    vllm serve "$MODEL_HANDLE" \
      --tensor-parallel-size 2 \
      --max-model-len "$MAX_MODEL_LEN" \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser "$TOOL_CALL_PARSER" \
      --reasoning-parser "$REASONING_PARSER" \
      --trust-remote-code \
      > /tmp/vllm-serve.log 2>&1
  '

```


### MiniMax-M2.7

```bash
MODEL_HANDLE=nvidia/MiniMax-M2.7-NVFP4
MAX_MODEL_LEN=120000
TOOL_CALL_PARSER=minimax_m2
REASONING_PARSER=minimax_m2

docker exec -d \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -e MAX_MODEL_LEN="$MAX_MODEL_LEN" \
  -e TOOL_CALL_PARSER="$TOOL_CALL_PARSER" \
  -e REASONING_PARSER="$REASONING_PARSER" \
  -e SAFETENSORS_FAST_GPU=1 \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  bash -c '
    vllm serve "$MODEL_HANDLE" \
      --tensor-parallel-size 2 \
      --max-model-len "$MAX_MODEL_LEN" \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser "$TOOL_CALL_PARSER" \
      --reasoning-parser "$REASONING_PARSER" \
      --trust-remote-code \
      > /tmp/vllm-serve.log 2>&1
  '
```


## Qwen3-Coder-Next FP8 Dynamic

```bash
MODEL_HANDLE=unsloth/Qwen3-Coder-Next-FP8-Dynamic
MAX_MODEL_LEN=120000
TOOL_CALL_PARSER=qwen3_coder

docker exec -d \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -e MAX_MODEL_LEN="$MAX_MODEL_LEN" \
  -e TOOL_CALL_PARSER="$TOOL_CALL_PARSER" \
  -e SAFETENSORS_FAST_GPU=1 \
  "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
  bash -c '
    vllm serve "$MODEL_HANDLE" \
      --tensor-parallel-size 2 \
      --distributed-executor-backend ray \
      --enforce-eager \
      --max-model-len "$MAX_MODEL_LEN" \
      --max-num-seqs 1 \
      --gpu-memory-utilization 0.90 \
      --enable-auto-tool-choice \
      --tool-call-parser "$TOOL_CALL_PARSER" \
      > /tmp/vllm-serve.log 2>&1
  '
```