Skip to content

Instantly share code, notes, and snippets.

@comfyanonymous

comfyanonymous/README.md Secret

Last active March 4, 2025 09:50
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save comfyanonymous/2f57adabe5a22b36a21ae024306daddb to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save comfyanonymous/2f57adabe5a22b36a21ae024306daddb to your computer and use it in GitHub Desktop.
Download ZIP
Test workflow for Nvidia Cosmos model
Raw
README.md

Test workflow for Nvidia Cosmos model

You will need

Text encoder and VAE:

https://huggingface.co/comfyanonymous/cosmos_1.0_text_encoder_and_VAE_ComfyUI/tree/main

oldt5_xxl_fp8_e4m3fn_scaled.safetensors -> ComfyUI/models/text_encoders

cosmos_cv8x8x8_1.0.safetensors -> ComfyUI/models/vae

Note: oldt5_xxl is not the same as the t5xxl used in flux and other models. oldt5_xxl is t5xxl 1.0 while the one used in flux and others is t5xxl 1.1

diffusion models repackaged in safetensors:

https://huggingface.co/mcmonkey/cosmos-1.0/tree/main

Goes in: ComfyUI/models/diffusion_models

Note: "Text to World" means Text to video and "Video to World" means image/video to video.

If you want the original diffusion models in .pt format the official links are:

https://huggingface.co/nvidia/Cosmos-1.0-Diffusion-7B-Text2World

https://huggingface.co/nvidia/Cosmos-1.0-Diffusion-14B-Text2World

https://huggingface.co/nvidia/Cosmos-1.0-Diffusion-7B-Video2World

https://huggingface.co/nvidia/Cosmos-1.0-Diffusion-14B-Video2World

Workflows

Text to Video

Image to Video

Raw
image_to_video.json
{
"last_node_id": 75,
"last_link_id": 190,
"nodes": [
{
"id": 61,
"type": "ModelSamplingContinuousEDM",
"pos": [
415.2213439941406,
-0.6805005669593811
],
"size": [
327.5999755859375,
106
],
"flags": {},
"order": 7,
"mode": 4,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 157
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
189
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ModelSamplingContinuousEDM"
},
"widgets_values": [
"edm",
120,
0.002
]
},
{
"id": 39,
"type": "VAELoader",
"pos": [
75.90576171875,
610.438232421875
],
"size": [
278.68310546875,
58
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
76,
173
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoader"
},
"widgets_values": [
"cosmos_cv8x8x8_1.0.safetensors"
]
},
{
"id": 65,
"type": "LoadImage",
"pos": [
-280.9966735839844,
612.0447387695312
],
"size": [
315,
314
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
184
],
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": null,
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"ComfyUI_21752_.png",
"image"
]
},
{
"id": 70,
"type": "CosmosImageToVideoLatent",
"pos": [
396.11029052734375,
617.0000610351562
],
"size": [
319.07012939453125,
150
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "vae",
"type": "VAE",
"link": 173
},
{
"name": "image",
"type": "IMAGE",
"link": 184
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
180
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CosmosImageToVideoLatent"
},
"widgets_values": [
1024,
1024,
121,
1
]
},
{
"id": 38,
"type": "CLIPLoader",
"pos": [
-332.08502197265625,
231.04571533203125
],
"size": [
315,
82
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
75,
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPLoader"
},
"widgets_values": [
"oldt5_xxl_fp8_e4m3fn_scaled.safetensors",
"cosmos",
"default"
]
},
{
"id": 42,
"type": "CLIPTextEncode",
"pos": [
0,
120
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 99
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
185
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"very cute anime girl with massive fennec fox ears with long wavy blonde hair and large blue eyes and is wearing a pink coat and a long black skirt and is sitting in the middle of a beautiful scenery amazing view in japanese anime style standing up and walking away"
]
},
{
"id": 7,
"type": "CLIPTextEncode",
"pos": [
0,
330
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 75
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
186
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"The video captures a series of frames showing ugly scenes, static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and flickering. Overall, the video is of poor quality."
]
},
{
"id": 73,
"type": "SaveAnimatedPNG",
"pos": [
2297.135986328125,
123.6446762084961
],
"size": [
959.7330932617188,
960.6917114257812
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 181
}
],
"outputs": [],
"properties": {
"Node name for S&R": "SaveAnimatedPNG"
},
"widgets_values": [
"ComfyUI",
24,
4,
null
]
},
{
"id": 69,
"type": "SaveAnimatedWEBP",
"pos": [
1512.938720703125,
115.3922119140625
],
"size": [
742.2177734375,
887.9819946289062
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 172
}
],
"outputs": [],
"properties": {},
"widgets_values": [
"ComfyUI",
24,
false,
80,
"default",
null
]
},
{
"id": 3,
"type": "KSampler",
"pos": [
843.46337890625,
122.69183349609375
],
"size": [
385.8114318847656,
262
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 190
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 187
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 188
},
{
"name": "latent_image",
"type": "LATENT",
"link": 180
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
87
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KSampler"
},
"widgets_values": [
192220229712561,
"randomize",
30,
7,
"res_multistep",
"karras",
1
]
},
{
"id": 75,
"type": "TorchCompileModel",
"pos": [
767.4100952148438,
-8.771673202514648
],
"size": [
210,
58
],
"flags": {},
"order": 9,
"mode": 4,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 189
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
190
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "TorchCompileModel"
},
"widgets_values": [
"inductor"
]
},
{
"id": 8,
"type": "VAEDecode",
"pos": [
1270.6927490234375,
120.51702117919922
],
"size": [
210,
46
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 87
},
{
"name": "vae",
"type": "VAE",
"link": 76
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
172,
181
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 37,
"type": "UNETLoader",
"pos": [
60.60690689086914,
-9.085861206054688
],
"size": [
315,
82
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
157
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UNETLoader"
},
"widgets_values": [
"cosmos_1.0_7B_i2v.pt",
"default"
]
},
{
"id": 74,
"type": "LTXVConditioning",
"pos": [
505.62652587890625,
174.38633728027344
],
"size": [
210,
78
],
"flags": {},
"order": 8,
"mode": 4,
"inputs": [
{
"name": "positive",
"type": "CONDITIONING",
"link": 185
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 186
}
],
"outputs": [
{
"name": "positive",
"type": "CONDITIONING",
"links": [
187
],
"slot_index": 0
},
{
"name": "negative",
"type": "CONDITIONING",
"links": [
188
],
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "LTXVConditioning"
},
"widgets_values": [
24
]
}
],
"links": [
[
75,
38,
0,
7,
0,
"CLIP"
],
[
76,
39,
0,
8,
1,
"VAE"
],
[
87,
3,
0,
8,
0,
"LATENT"
],
[
99,
38,
0,
42,
0,
"CLIP"
],
[
157,
37,
0,
61,
0,
"MODEL"
],
[
172,
8,
0,
69,
0,
"IMAGE"
],
[
173,
39,
0,
70,
0,
"VAE"
],
[
180,
70,
0,
3,
3,
"LATENT"
],
[
181,
8,
0,
73,
0,
"IMAGE"
],
[
184,
65,
0,
70,
1,
"IMAGE"
],
[
185,
42,
0,
74,
0,
"CONDITIONING"
],
[
186,
7,
0,
74,
1,
"CONDITIONING"
],
[
187,
74,
0,
3,
1,
"CONDITIONING"
],
[
188,
74,
1,
3,
2,
"CONDITIONING"
],
[
189,
61,
0,
75,
0,
"MODEL"
],
[
190,
75,
0,
3,
0,
"MODEL"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1167815779425856,
"offset": [
347.0649226455528,
63.53046012379835
]
}
},
"version": 0.4
}
Raw
text_to_video.json
{
"last_node_id": 76,
"last_link_id": 191,
"nodes": [
{
"id": 61,
"type": "ModelSamplingContinuousEDM",
"pos": [
415.2213439941406,
-0.6805005669593811
],
"size": [
327.5999755859375,
106
],
"flags": {},
"order": 6,
"mode": 4,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 157
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
189
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ModelSamplingContinuousEDM"
},
"widgets_values": [
"edm",
120,
0.002
]
},
{
"id": 38,
"type": "CLIPLoader",
"pos": [
-332.08502197265625,
231.04571533203125
],
"size": [
315,
82
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
75,
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPLoader"
},
"widgets_values": [
"oldt5_xxl_fp8_e4m3fn_scaled.safetensors",
"cosmos",
"default"
]
},
{
"id": 7,
"type": "CLIPTextEncode",
"pos": [
0,
330
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 75
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
186
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"The video captures a series of frames showing ugly scenes, static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and flickering. Overall, the video is of poor quality."
]
},
{
"id": 73,
"type": "SaveAnimatedPNG",
"pos": [
2297.135986328125,
123.6446762084961
],
"size": [
959.7330932617188,
960.6917114257812
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 181
}
],
"outputs": [],
"properties": {
"Node name for S&R": "SaveAnimatedPNG"
},
"widgets_values": [
"ComfyUI",
24,
4
]
},
{
"id": 69,
"type": "SaveAnimatedWEBP",
"pos": [
1512.938720703125,
115.3922119140625
],
"size": [
742.2177734375,
887.9819946289062
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 172
}
],
"outputs": [],
"properties": {},
"widgets_values": [
"ComfyUI",
24,
false,
80,
"default"
]
},
{
"id": 75,
"type": "TorchCompileModel",
"pos": [
767.4100952148438,
-8.771673202514648
],
"size": [
210,
58
],
"flags": {},
"order": 8,
"mode": 4,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 189
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
190
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "TorchCompileModel"
},
"widgets_values": [
"inductor"
]
},
{
"id": 8,
"type": "VAEDecode",
"pos": [
1270.6927490234375,
120.51702117919922
],
"size": [
210,
46
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 87
},
{
"name": "vae",
"type": "VAE",
"link": 76
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
172,
181
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 37,
"type": "UNETLoader",
"pos": [
60.60690689086914,
-9.085861206054688
],
"size": [
315,
82
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
157
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UNETLoader"
},
"widgets_values": [
"cosmos_1.0_7B_t2v.pt",
"default"
]
},
{
"id": 74,
"type": "LTXVConditioning",
"pos": [
505.62652587890625,
174.38633728027344
],
"size": [
210,
78
],
"flags": {},
"order": 7,
"mode": 4,
"inputs": [
{
"name": "positive",
"type": "CONDITIONING",
"link": 185
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 186
}
],
"outputs": [
{
"name": "positive",
"type": "CONDITIONING",
"links": [
187
],
"slot_index": 0
},
{
"name": "negative",
"type": "CONDITIONING",
"links": [
188
],
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "LTXVConditioning"
},
"widgets_values": [
24
]
},
{
"id": 39,
"type": "VAELoader",
"pos": [
952.5319213867188,
460.901611328125
],
"size": [
278.68310546875,
58
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
76
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoader"
},
"widgets_values": [
"cosmos_cv8x8x8_1.0.safetensors"
]
},
{
"id": 3,
"type": "KSampler",
"pos": [
843.46337890625,
122.69183349609375
],
"size": [
385.8114318847656,
262
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 190
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 187
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 188
},
{
"name": "latent_image",
"type": "LATENT",
"link": 191
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
87
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KSampler"
},
"widgets_values": [
192220229712561,
"randomize",
30,
7,
"res_multistep",
"karras",
1
]
},
{
"id": 76,
"type": "EmptyCosmosLatentVideo",
"pos": [
109.6046142578125,
575.8067016601562
],
"size": [
315,
130
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
191
]
}
],
"properties": {
"Node name for S&R": "EmptyCosmosLatentVideo"
},
"widgets_values": [
1280,
704,
121,
1
]
},
{
"id": 42,
"type": "CLIPTextEncode",
"pos": [
0,
120
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 99
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
185
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"The video is a first-person perspective from the viewpoint of a large, humanoid robot navigating through a chemical plant. The robot is equipped with a camera mounted on its head, providing a view of the surroundings. The environment is industrial, with large metal structures and shelves filled with various boxes and supplies. The robot is seen moving forward, with its camera capturing the scene from a height of about 1 meter above the floor. The camera remains mostly static, with slight movements as the robot advances. The robot's body is metallic, with a large, boxy structure and a prominent head with a camera. The background is filled with industrial equipment and storage shelves, indicating a busy and functional workspace. The lighting is bright, typical of an industrial setting, with overhead lights illuminating the area. The robot's movement is steady and deliberate, suggesting a purposeful task, possibly involving inspection or maintenance. The video does not contain any text overlays or channel logos, focusing solely on the visual experience of the robot's journey through the plant."
]
}
],
"links": [
[
75,
38,
0,
7,
0,
"CLIP"
],
[
76,
39,
0,
8,
1,
"VAE"
],
[
87,
3,
0,
8,
0,
"LATENT"
],
[
99,
38,
0,
42,
0,
"CLIP"
],
[
157,
37,
0,
61,
0,
"MODEL"
],
[
172,
8,
0,
69,
0,
"IMAGE"
],
[
181,
8,
0,
73,
0,
"IMAGE"
],
[
185,
42,
0,
74,
0,
"CONDITIONING"
],
[
186,
7,
0,
74,
1,
"CONDITIONING"
],
[
187,
74,
0,
3,
1,
"CONDITIONING"
],
[
188,
74,
1,
3,
2,
"CONDITIONING"
],
[
189,
61,
0,
75,
0,
"MODEL"
],
[
190,
75,
0,
3,
0,
"MODEL"
],
[
191,
76,
0,
3,
3,
"LATENT"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1167815779425856,
"offset": [
333.63346899672575,
67.11218109681896
]
}
},
"version": 0.4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment