Skip to content

Instantly share code, notes, and snippets.

@npearce

npearce/raspberry_pi_consul_cluster.md

Last active August 27, 2023 08:03
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save npearce/d9eaea6cd024ff2528118853a9689962 to your computer and use it in GitHub Desktop.
Save npearce/d9eaea6cd024ff2528118853a9689962 to your computer and use it in GitHub Desktop.
Download ZIP
Build a Raspberry Pi Consul Cluster
Raw
raspberry_pi_consul_cluster.md

Configure Raspberry Pi Networking

It is possible to run 3 consul servers on the same IP address if you shift the various ports to unique numbers to avoid conflict, but you don't have to do that.

Instead, add interface alias's in your /etc/rc.local (just before the exit 0), e.g.:

In my case, wlan0 is a dhcp assigned interface. The following commands add three new alias interfaces, wlan0:1, etc, with their own stat IP Addresses.

ifconfig wlan0:1 inet 10.0.0.41 netmask 255.255.255.0
ifconfig wlan0:2 inet 10.0.0.42 netmask 255.255.255.0
ifconfig wlan0:3 inet 10.0.0.43 netmask 255.255.255.0

Reboot and ensure you can reach these IP Addresses on your network.

NOTE: to get around peculiarities of my cable providers terminating device, wlan0 remains a dhcp interface so that it's MAC address is registerd and packets are routed - it does wifi mac filtering even though MAC filtering is dsiabled.

Install Consul

Download the binary:

NOTE: 'armelv5' for 32 bit Raspberry Pi, 'armelv6' for 64bit.

cd ~
export CONSUL_URL="https://releases.hashicorp.com/consul"
export CONSUL_VERSION="1.8.3"
curl --silent --remote-name   ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_armelv5.zip
unzip consul_${CONSUL_VERSION}_linux_armelv5.zip
sudo chown root:root consul
sudo mv consul /usr/local/bin/

Verify path: consul --version

Install consul's autocomplete CLI capabilities:

consul -autocomplete-install
complete -C /usr/local/bin/consul consul

Creating an operating environment for consul:

sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo mkdir --parents /opt/consul1
sudo mkdir --parents /opt/consul2
sudo mkdir --parents /opt/consul3
sudo chown --recursive consul:consul /opt/consul1
sudo chown --recursive consul:consul /opt/consul2
sudo chown --recursive consul:consul /opt/consul3

Setup the interconnectivity encryption

Save the output from this file for the server configuration files below server[1-3].hcl: consul keygen

Generate a consul-agent-ca.pem and consul-agent-ca-key.pem using: consul tls ca create

Create server & client certs for you Raspberry Pi DC, e.g. for a DC named 'raspberrydc':

consul tls cert create -server -dc raspberrydc
consul tls cert create -client -dc raspberrydc

You will now have four new files:

raspberrydc-server-consul-0.pem
raspberrydc-server-consul-0-key.pem
raspberrydc-client-consul-0.pem
raspberrydc-client-consul-0-key.pem

We will copy these certs to the apprpriate location later.

Create the configuration directory and files

sudo mkdir --parents /etc/consul.d
sudo touch /etc/consul.d/server1.hcl
sudo touch /etc/consul.d/server2.hcl
sudo touch /etc/consul.d/server3.hcl
sudo chown --recursive consul:consul /etc/consul.d
sudo chmod 640 /etc/consul.d/server*

Copy the certs/keys to the configuration directory:

cp consul-agent-ca.pem raspberrydc-server-consul-0.pem raspberrydc-client-consul-0.pem raspberrydc-server-consul-0-key.pem raspberrydc-client-consul-0-key.pem /etc/consul.d/

Populate the configuration files:

NOTE: the 'datacenter' value must match the dc specified when creating the certificates previously. NOTE: the 'encrpt' value must match the out from the consul keygen command earlier.

server1.hcl

node_name = "server1"
datacenter = "raspberrydc"
data_dir = "/opt/consul1"
encrypt = "XXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxx="
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/raspberrydc-server-consul-0.pem"
key_file = "/etc/consul.d/raspberrydc-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
server = true
bootstrap_expect = 3
ui = true
bind_addr = "10.0.0.41"
client_addr = "10.0.0.41"
retry_join = ["10.0.0.42","10.0.0.43"]

server2.hcl

node_name = "server2"
datacenter = "raspberrydc"
data_dir = "/opt/consul2"
encrypt = "XXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxx="
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/raspberrydc-server-consul-0.pem"
key_file = "/etc/consul.d/raspberrydc-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
server = true
bootstrap_expect = 3
ui = true
bind_addr = "10.0.0.42"
client_addr = "10.0.0.42"
retry_join = ["10.0.0.41","10.0.0.43"]

server3.hcl

node_name = "server3"
datacenter = "raspberrydc"
data_dir = "/opt/consul3"
encrypt = "XXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxx="
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/raspberrydc-server-consul-0.pem"
key_file = "/etc/consul.d/raspberrydc-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
server = true
bootstrap_expect = 3
ui = true
bind_addr = "10.0.0.43"
client_addr = "10.0.0.43"
retry_join = ["10.0.0.41","10.0.0.42"]

Configure systemd to launch the cluster

sudo touch /etc/systemd/system/consul1.service
sudo touch /etc/systemd/system/consul2.service
sudo touch /etc/systemd/system/consul3.service

Populate the files as follows:

consul1.service

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/server1.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ -config-file=/etc/consul.d/server1.hcl
ExecReload=/usr/local/bin/consul reload
ExecStop=/usr/local/bin/consul leave
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

consul2.service

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/server2.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ -config-file=/etc/consul.d/server2.hcl
ExecReload=/usr/local/bin/consul reload
ExecStop=/usr/local/bin/consul leave
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

consul3.service

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/server3.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ -config-file=/etc/consul.d/server3.hcl
ExecReload=/usr/local/bin/consul reload
ExecStop=/usr/local/bin/consul leave
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Validate the configs:

consul validate /etc/consul.d/server1.hcl
consul validate /etc/consul.d/server2.hcl
consul validate /etc/consul.d/server3.hcl

Enalbe the systemctl services:

sudo systemctl enable consul1
sudo systemctl enable consul2
sudo systemctl enable consul3

Start the services:

sudo systemctl start consul1 consul2 consul3

Configure the CLI

The consul command defaults to 127.0.0.1:8500, but to support a consul cluster on a single device we aren't listening on the loopbak address. Reveiew the server[1-3].hcl files. Specifically the bind_addr and client_addr values. The default is 0.0.0.0.

Add the following command to the end of ~/.profile

export CONSUL_HTTP_ADDR=10.0.0.41:8500

Either logout and login again, or execute the following for this to take effect:

source ~/.profile

You should now be able to run consul members and see all three servers.

You can also verify that all is running with:

sudo systemctl status consul1 consul2 consul3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment