Many workloads running in Nomad need a means for acquiring a secret or sensitive configuration information. HashiCorp Vault provides industry leading secrets management and Nomad has an integration that enables Vault to be a backing store for this type of information.
However, we kept hearing from the community that deploying, managing, and integrating with Vault was a lot of effort for smaller use-cases, early proof-of-concept work, and home labs. Nomad users wanted something easier to use out of the box for these types of projects. To meet these needs, we created Nomad Variables.
Nomad Variables enables you to provide secrets and configuration to your job:
- Without having to learn and support an additional product during your application rollout on Nomad
- Without the risk of static secrets in Nomad job specification or secret disclosure when using command-line arguments
- Using a familiar job specification element—the
templateblock
Let's look at the sample job that is created when you run nomad job init -short
job "example" {
datacenters = ["dc1"]
group "cache" {
network {
port "db" {
to = 6379
}
}
task "redis" {
driver = "docker"
config {
image = "redis:7"
ports = ["db"]
auth_soft_fail = true
}
resources {
cpu = 500
memory = 256
}
}
}
}This is a basic Nomad job specification that creates a single Docker instance of Redis 7 and schedules a dynamic port on the client that will be forwarded to the redis port (6379) in the container.
Now, suppose you would like to enable security by providing a redis.conf. Nomad provides a template block in the job specification that you can use to create files that the running application can consume.
Let's add one now inside of the task "redis" block.
template {
destination = "local/conf/redis.conf"
change_mode = "signal"
change_signal = "SIGINT"
data = <<EOT
requirepass foobared
EOT
}We'll add our configuration into the Docker container using a bind mount by adding a volume attribute inside of the task "redis" > config block.
volumes = ["local/conf:/usr/local/etc/redis"]and add an args list to point Redis to the mounted configuration
args = ["/usr/local/etc/redis/redis.conf"]So by this time our job specification looks like this.
job "example" {
datacenters = ["dc1"]
group "cache" {
network {
port "db" {
to = 6379
}
}
task "redis" {
driver = "docker"
config {
image = "redis:7"
ports = ["db"]
auth_soft_fail = true
args = ["/usr/local/etc/redis/redis.conf"]
volumes = ["local/conf:/usr/local/etc/redis"]
}
resources {
cpu = 500
memory = 256
}
template {
destination = "local/conf/redis.conf"
change_mode = "signal"
change_signal = "SIGINT"
data = <<EOT
requirepass foobared
EOT
}
}
}
}Deploy the job into your Nomad cluster with the nomad job run command.
$ nomad job run example2.nomad
==> 2022-09-16T14:50:18-04:00: Monitoring evaluation "dd68b093"
2022-09-16T14:50:18-04:00: Evaluation triggered by job "example"
2022-09-16T14:50:18-04:00: Allocation "a006a7fd" created: node "e147bb46", group "cache"
2022-09-16T14:50:19-04:00: Evaluation within deployment: "50d81a1c"
2022-09-16T14:50:19-04:00: Allocation "a006a7fd" status changed: "pending" -> "running" (Tasks are running)
2022-09-16T14:50:19-04:00: Evaluation status changed: "pending" -> "complete"
==> 2022-09-16T14:50:19-04:00: Evaluation "dd68b093" finished with status "complete"
==> 2022-09-16T14:50:19-04:00: Monitoring deployment "50d81a1c"
✓ Deployment "50d81a1c" successful
2022-09-16T14:50:29-04:00
ID = 50d81a1c
Job ID = example
Job Version = 0
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
cache 1 1 1 0 2022-09-16T15:00:28-04:00
Find the IP address and port that Nomad has scheduled the allocation on by using the nomad alloc status command. Use the Allocation ID provided in the nomad job run command output in the previous step—in the sample output it is a006a7fd
$ nomad alloc status a006a7fd
ID = a006a7fd-5b07-26d5-f9d0-f22c7ddac721
Eval ID = dd68b093
Name = example.cache[0]
Node ID = e147bb46
Node Name = voiselle-V2R6QCTV92
Job ID = example
Job Version = 0
Client Status = running
Client Description = Tasks are running
Desired Status = run
Desired Description = <none>
Created = 21s ago
Modified = 10s ago
Deployment ID = 50d81a1c
Deployment Health = healthy
Allocation Addresses:
Label Dynamic Address
*db yes 10.0.0.254:21948 -> 6379
Task "redis" is "running"
Task Resources:
CPU Memory Disk Addresses
0/500 MHz 2.6 MiB/256 MiB 300 MiB
Task Events:
Started At = 2022-09-16T18:50:18Z
Finished At = N/A
Total Restarts = 0
Last Restart = N/A
Recent Events:
Time Type Description
2022-09-16T14:50:18-04:00 Started Task started by client
2022-09-16T14:50:18-04:00 Task Setup Building Task Directory
2022-09-16T14:50:18-04:00 Received Task received by client
Note the Allocation Addresses: section of the output. The Address column for the db row will contain the IP address and port on the client with an arrow pointing to 6379. In the above sample output it is 10.0.0.254:21948. Your address and port will vary.
NOTE: When using a dev agent, this will be a 127.0.0.1 address. This will interfere with the following Docker test since the two containers will not "share" the same localhost. You can add the
-network-interfaceflag pointing to a network interface with a real IP address on it and restart the demo or you can use telnet to connect to the Redis instance and test it, but that process won't be demonstrated.
If you have Docker on your machine, you can run the redis-cli and connect to the instance.
$ docker run -it --rm redis redis-cli -h 10.0.0.254 -p 21948
10.0.0.254:21948> PING
(error) NOAUTH Authentication required.
10.0.0.254:21948> AUTH foobared
OK
10.0.0.254:21948> PING
PONG
10.0.0.254:21948> exit
At this point we have an authenticated Redis instance, but it is far from secure since the secret is sitting in the jobspec in the clear.
Nomad Variables provide us a way to store and consume secrets without having to risk embedding them in the job specification.
If you're using a private source code repository, you might not be as worried about writing sensitive material into the jobspec. You might think "But I'm using HCL variables to pass secrets so I'm not worried about sensitive material leaking." However, the entire job specification as submitted is stored in the clear inside of the Nomad server state; so it's possible that secrets are being captured in your regular Nomad database backups. Nomad variables are written in an encrypted form into the Server's data protecting you from an accidental exposure via a backup.
Step 1) Store secrets as a Nomad Variable Step 2) Update templates to consume secrets Step 3) Profit!!
The nomad var commands are used when working with Nomad Variables. Let's take our Redis authentication secret as a Nomad Variable. To store a variable, we use the nomad var put command. It takes a path, which can be arbitrary; however there is a specially-shaped path that is automatically available to matching workloads. We'll use that path to store our Redis password.
Additionally, the nomad var put command typically does not return output, but for this demo let's have the command return the information about the newly created variable in table format.
$ nomad var put -out=table nomad/jobs/example/cache/redis password=foobared
Namespace = default
Path = nomad/jobs/example/cache/redis
Create Time = 2022-09-16T15:23:22-04:00
Check Index = 56
Items
password = foobared
Awesome, now we've stored the secret into Nomad. We need to update the job specification to use the stored variable.
Edit the template block in the job specification.
Change
requirepass foobared to
requirepass {{with nomadVar "nomad/jobs/example/cache/redis"}}{{.password}}{{end}} Now, run the job again, noting the Allocation ID in the output as before.
$ nomad job run example.nomad
==> 2022-09-16T15:30:59-04:00: Monitoring evaluation "253cb1c4"
2022-09-16T15:30:59-04:00: Evaluation triggered by job "example"
2022-09-16T15:31:00-04:00: Evaluation within deployment: "5689aee4"
2022-09-16T15:31:00-04:00: Allocation "12a63af7" created: node "e147bb46", group "cache"
2022-09-16T15:31:00-04:00: Evaluation status changed: "pending" -> "complete"
==> 2022-09-16T15:31:00-04:00: Evaluation "253cb1c4" finished with status "complete"
==> 2022-09-16T15:31:00-04:00: Monitoring deployment "5689aee4"
✓ Deployment "5689aee4" successful
2022-09-16T15:31:11-04:00
ID = 5689aee4
Job ID = example
Job Version = 1
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
cache 1 1 1 0 2022-09-16T15:41:10-04:00
Get the address of the Redis instance by running nomad alloc status with the new allocation ID.
$ nomad alloc status 12a63af7
ID = 12a63af7-489b-dd93-44b8-564cdfa1551c
Eval ID = 253cb1c4
Name = example.cache[0]
Node ID = e147bb46
Node Name = voiselle-V2R6QCTV92
Job ID = example
Job Version = 1
Client Status = running
Client Description = Tasks are running
Desired Status = run
Desired Description = <none>
Created = 1m8s ago
Modified = 57s ago
Deployment ID = 5689aee4
Deployment Health = healthy
Allocation Addresses:
Label Dynamic Address
*db yes 10.0.0.254:22842 -> 6379
Task "redis" is "running"
Task Resources:
CPU Memory Disk Addresses
0/500 MHz 2.6 MiB/256 MiB 300 MiB
Task Events:
Started At = 2022-09-16T19:31:00Z
Finished At = N/A
Total Restarts = 0
Last Restart = N/A
Recent Events:
Time Type Description
2022-09-16T15:31:00-04:00 Started Task started by client
2022-09-16T15:31:00-04:00 Task Setup Building Task Directory
2022-09-16T15:30:59-04:00 Received Task received by client
Finally, use the redis-cli to test that the secret was read from the Nomad Variable.
$ docker run -it --rm redis redis-cli -h 10.0.0.254 -p 22842
10.0.0.254:22842> PING
(error) NOAUTH Authentication required.
10.0.0.254:22842> AUTH foobared
OK
10.0.0.254:22842> PING
PONG
10.0.0.254:22842> EXIT
According to sample redis.conf file:
Warning: since Redis is pretty fast, an outside user can try up to 1 million passwords per second against a modern box. This means that you should use very strong passwords, otherwise they will be very easy to break. Note that because the password is really a shared secret between the client and the server, and should not be memorized by any human, the password can be easily a long string from /dev/urandom or whatever, so by using a long and unguessable password no brute force attack will be possible.
So lets make a really big random password for our Redis instance using data from /dev/urandom processed using the base64 command. The nomad var put command can take data from standard input so you can use it in pipelined commands like the following—some things to note in the command:
- the value assigned to
passwordis-, which tells Nomad to get the value from standard input - the command has a
-forceflag to write the variable without performing a check-and-set operation which we will discuss later.
$ head -c 256 /dev/urandom | xxd -p -c 256 | nomad var put -force -out=table nomad/jobs/example/cache/redis password=-
Namespace = default
Path = nomad/jobs/example/cache/redis
Create Time = 2022-09-16T16:02:40-04:00
Modify Time = 2022-09-16 16:02:40.505941 -0400 EDT
Check Index = 113
Items
password = 2078c73fdff58e5002e0b5457af2b710c137e76912dd4ef00e208cb91d515a9bb95177685ce88867b666c23869e819e4d0b5eaa58790ea766e2087bbaa57eff437f7f56d7b8d3d8685ff97c39cb2f5faf0f4c149ec2af581de7bfb9b4f633bae98bc5793cc7eeee5fb9df5e947653f774933a73bee0c7c106c86821d429488f181f71f7627cf9df0dd03bd943b1827e7e70896ab62d3e1c0f40f19ba5228fe5c8d2dc7f3c1c344dbb5ebbe17ca701b9b7f0ac77e2e98885eec32bce09e7cd301b6eb9aa1adb566b650b2e7e0a7a83f03795ef94f5701a76f100796db71fd96a84d760356346ab88bbd47e1a867affa28013db54c9d8f56ce9c308cd9d1758f92
The command you just ran collects 256 bytes of random data from /dev/urandom and prints them out as 512 hexadecimal digits. That string is then piped into the nomad var put command and stored as the password value for our variable stored at nomad/jobs/example/cache/redis
If we rerun the nomad alloc status command for our Redis allocation, we can observe that the allocation was restarted since we changed the secret.
$ nomad alloc status 12a63af7
ID = 12a63af7-489b-dd93-44b8-564cdfa1551c
Eval ID = 253cb1c4
Name = example.cache[0]
Node ID = e147bb46
Node Name = voiselle-V2R6QCTV92
Job ID = example
Job Version = 1
Client Status = running
Client Description = Tasks are running
Desired Status = run
Desired Description = <none>
Created = 36m16s ago
Modified = 4m31s ago
Deployment ID = 5689aee4
Deployment Health = healthy
Allocation Addresses:
Label Dynamic Address
*db yes 10.0.0.254:22842 -> 6379
Task "redis" is "running"
Task Resources:
CPU Memory Disk Addresses
0/500 MHz 2.6 MiB/256 MiB 300 MiB
Task Events:
Started At = 2022-09-16T20:02:44Z
Finished At = N/A
Total Restarts = 1
Last Restart = 2022-09-16T16:02:44-04:00
Recent Events:
Time Type Description
2022-09-16T16:02:44-04:00 Started Task started by client
2022-09-16T16:02:44-04:00 Restarting Task restarting in 16.692545228s
2022-09-16T16:02:44-04:00 Terminated Exit Code: 0
2022-09-16T16:02:44-04:00 Signaling Template re-rendered
2022-09-16T15:31:00-04:00 Started Task started by client
2022-09-16T15:31:00-04:00 Task Setup Building Task Directory
2022-09-16T15:30:59-04:00 Received Task received by client
Let's try using the new secret to connect to the Redis instance. Rather than using copy/paste to fetch the password, we can use nomad var get with the -item flag to fetch the password value from the nomad/jobs/example/cache/redis variable.
$ docker run -it --rm redis redis-cli -h 10.0.0.254 -p 22842 -a $(nomad var get -item=password nomad/jobs/example/cache/redis)
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.254:22842> PING
PONG
10.0.0.254:22842> EXIT