Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Minecraft on Raspberry Pi cluster with metrics

alt

Minecraft on Raspberry Pi cluster with metrics

Ever wanted to put your Rapsberry Pi cluster to great use? Our team is working remotely, so we started to play Minecraft. I decided I would host the Minecraft server on my Raspberry Pi cluster. This gist will guide you through the steps I took to get a k3s cluster up with k3sup and later installed Minecraft as well as metrics exporter and Prometheus Operator

Why?

Quoniam Possumus - Because we can

You'll need

  • Raspberry Pis to run kubernetes on. I used 3 Raspberry Pi 3 and one Raspberry Pi 4.

  • A copy of the game Minecraft Java Edition so you can play on your server.

  • Helm

Will it cluster?

Read about how you can install kubernetes to your Raspberry Pi in 15 minutes

Or visit https://k3sup.dev/ to setup your cluster.

Note, If you are using Raspberry Pi 4, consider using a 64 bit kernel to utilize more than 2 GB memory per process.

Taint and label the Minecraft node

To give the node where Minecraft is running access to the whole CPU and RAM you'll need to taint it so it only tolerates the Minecraft Server

kubectl taint nodes minecraft-node app=minecraft:NoSchedule

And to make sure that the minecraft installation will select the node it tolerates you'll also need to label the node.

kubectl label nodes minecraft-node role=minecraft

Install Minecraft Server

This is the helmchart that we will use https://github.com/helm/charts/tree/master/stable/minecraft

Minecraft Server comes in many flavours.

  • To install Minecraft and later install Minecraft Prometheus Export plugin we need to use the SPIGOT distribution of Minecraft Server, you can also run PAPER. This is also running more lightweight than VANILLA Minecraft which is a perfect fit for Raspberry Pi.

  • We need to make sure we are using the correct Minecraft Server image itzg/minecraft-server:multiarch

  • Further more Minecraft needs to have the tolerations that we tainted the minecraft-node with and the nodeSelector

  • Minecraft on Raspberry Pi takes longer time to start than on modern servers, the readyness and liveness probe is turned up a lot. If you have problems with restarts, try to increase the probes even more.

All of these settings are represented in the values file that will be used with the helm chart for Minecraft.

Create the values.yaml below to use with your helm installation.

kubectl create namespace minecraft
helm install --namespace minecraft minecraft -f values.yaml stable/minecraft

Checkpoint

Make sure Minecraft has started and is in Ready state

kubectl get pods -n minecraft

Lookup the IP address for your Minecraft Server.

kubectl get service -n minecraft

Use the IP of the loadbalancer to start playing right away. Or continue with metrics.


We're not done yet, metrics metrics metrics

Now we will install

  • One plugin to expose metrics that can be scraped by Prometheus
  • Prometheus Operator to get a full fledged monitoring system

Minecraft Prometheus Exporter

To get metrics scraped we need to install a Bukkit plugin for just this.

The plugin we are going to install can be found here https://github.com/sladkoff/minecraft-prometheus-exporter

This step requires manual labour inside the minecraft containers.

Get the latest release from minecraft-prometheus-exporter, it is the .jar file that you want to use. Copy that into the Minecraft pod's mounted volume, where the plugins should go.

kubectl cp minecraft-prometheus-exporter-<VERSION>.jar minecraft/minecraft-minecraft-<POD-ID>:/data/plugins

Next restart the pod by deleteing the pod, and it will restart again

We need to make one more change, make sure the exporter listens to all traffic. Exec into the pod and change the config.yml of the minecraft-prometheus-exporter, as described on the plugins README.md file.

# Note that the HTTP server binds to localhost by default.
# If your Prometheus runs on another host or inside a Kubernetes cluster 
# set this to any reachable IP or 0.0.0.0 to listen on all interfaces.
host: localhost
port: 9225

Connect to your pod:

kubectl exec -it -n minecraft minecraft-minecraft-<POD-ID> -- /bin/bash

Edit the config file as per our needs set host to 0.0.0.0 instead of localhost

nano plugins/PrometheusExporter/config.yml

host: 0.0.0.0
port: 9225
...

and exit the pod

exit

Now we need to expose the 9225 port from the deployment as well

kubectl patch deployment -n minecraft minecraft-minecraft --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

This adds a port entry with containerPort: 9225 and name: metrics.

The end result will look like this

        
        ...e: minecraft-minecraft
        ports:
        - containerPort: 25565
          name: minecraft
          protocol: TCP
        - containerPort: 9225
          name: metrics
          protocol: TCP
        readine...

Scrape Minecraft

Install the Prometheus Operator. To do this an ARM (Raspberry Pi architecture) I used https://github.com/uocxp/prometheus-operator-chart-arm.

Prometheus is using ServiceMonitors to scrape endpoints. We need to create a ServiceMonitor that let us scrape a port that we now produce metrics on.

Create a servicemonitor.yaml below.

kubectl apply -f servicemonitor.yaml

Note that the namespace of the service monitor should be the same as where you installed prometheus operator

  • The label release: prometheus-operator needs to be there because the operator only listens to ServiceMonitors with that label by default.

  • The targetPort is the one that we just created in the previous step and the app: minecraft-minecraft comes from the helm chart.

  • The relabeling is to get the node as server_name on the dashboards.

Now you are ready to open up Grafana and install the two dashboards that can be found on the minecraft-prometheus-exporter repository.

kubectl port-forward -n monitor svc/prometheus-operator-grafana 3000:80

Open a browser to http://localhost:3000 and login with the default username and password for prometheus-operator admin prom-operator

Hit the + and paste the Server and the Player dashboards.

grafana.png Enjoy!

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: minecraft-servicemonitor
namespace: monitor
labels:
release: prometheus-operator
spec:
endpoints:
- interval: 10s
path: /metrics
relabelings:
- sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: server_name
targetPort: metrics
namespaceSelector:
matchNames:
- minecraft
selector:
matchLabels:
app: minecraft-minecraft
imageTag: multiarch
resources:
requests: {}
tolerations:
- key: "app"
operator: "Equal"
value: "minecraft"
effect: "NoSchedule"
nodeSelector:
role: minecraft
livenessProbe:
initialDelaySeconds: 120
periodSeconds: 60
readinessProbe:
initialDelaySeconds: 120
periodSeconds: 60
minecraftServer:
# This must be overridden, since we can't accept this for the user.
eula: "TRUE"
# One of: LATEST, SNAPSHOT, or a specific version (ie: "1.7.9").
version: "1.15.2"
# This can be one of "VANILLA", "FORGE", "SPIGOT", "BUKKIT", "PAPER", "FTB", "SPONGEVANILLA"
type: "SPIGOT"
# Max view distance (in chunks).
viewDistance: 7
# Message of the Day
motd: "Welcome to Minecraft on Kubernetes on Rapsberry Pi!"
# If you adjust this, you may need to adjust resources.requests above to match.
# If you are running on raspberry pi 4 you can utilize a lot more
memory: 512M
persistence:
## minecraft data Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: "-"
dataDir:
# Set this to false if you don't care to persist state between restarts.
enabled: true
Size: 10Gi
@alexellis

This comment has been minimized.

@Younday

This comment has been minimized.

Copy link

Younday commented Apr 1, 2020

Love the post! Just one suggestion: in case you install the prometheus operator chart according to https://github.com/uocxp/prometheus-operator-chart-arm, it will be in the namespace 'monitor'. The port-forward command won't work immediately, but when you change into this:

kubectl port-forward svc/prometheus-operator-grafana 3000:80 -n monitor

It will work, also depending on what your current namespace is of course.

I tried it out on my RPi4 cluster with K3s and it works like a charm!

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 1, 2020

Glad you liked it and thanks for pointing that out.

@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 4, 2020

I like how a lot of the details are left out

The 15 minute cluster guide shows us how to setup the cluster and install arkade.

Not only do you not use arkade in your gist here like I assumed you would since it was in the setup guide you included, you use helm and omit the configuration you used for it.

I guess the actual helm documentation wasn't written for RPi because there is nothing about troubleshooting tiller being stuck in a CrashLoopBackoff state. So now I am stuck at the helm install of minecraft.

Screen Shot 2020-04-04 at 3 10 32 PM

Sick ok lets fix tiller then.

After installing helm by following the helm documentation you linked
Screen Shot 2020-04-04 at 3 13 05 PM

...and here we are. Yea I have run describe and checked the logs and googled my ass off. Pretty much it all just comes back to CrashLookBackoff

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 5, 2020

@justcallmekoko I realize that the instructions are quite unclear on what exactly I did to get the cluster up and running. I like the writeup on how to get a cluster up.

I actually just installed k3s with k3sup and not worrying about installing any other apps. Also maybe you are running helm version 2?
So I only did this to get the cluster up. I'll revise the instructions accordingly.

k3sup install --ip $IP --user pi

since I have several clusters that I work with I actually used the below command, but this is not mandatory

k3sup install --ip $IP --user pi --local-path ~/.kube/config --merge --context rpi-cluster

and then for each node.

k3sup join --ip $NODE_IP --server-ip $IP--user pi

Then you should have a cluster up and you can check the nodes in ready state with

For minecraft to work I didn't install anything in this cluster apart from what is described in the instructions above.

@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 5, 2020

Thanks for the clear up. I managed to get the cluster working properly with k3sup and I was able to run the helm install with the yml file.

I have a new problem now where the minecraft pod is a running state however the ready column shows 0/1.
Screen Shot 2020-04-05 at 10 01 55 AM
netstat does not show the default minecraft port open and I am unable to connect using the minecraft client.

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 6, 2020

@justcallmekoko what did the logs* say about minecraft? You can update the values file with more memory, the 512 mb is default from the stable helm chart. Running on rpi 4 you can allocate 1800M for the minecraft server (everything up to 2G is accepted on a 32-bit kernel, the standard raspbian).

  • logs
kubectl logs -n minecraft minecraft-minecraft-dd8b5d849-phvnb
@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 6, 2020

I actually went into the helm charts repo and posted this issue with all of the information I could scrape together. You see the full log of the minecraft pod there.

@dasNavy

This comment has been minimized.

Copy link

dasNavy commented Apr 8, 2020

Can you simply add a modpack or single mods?
And what was the maximum cpu and ram usage with the highest amount of players connected so far?

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 8, 2020

  1. You can add those by just copy those in into the plugins folder as described above. Those get persisted on the PVC.
kubectl cp your-plugin.jar minecraft/minecraft-minecraft-<POD-ID>:/data/plugins
  1. I am running the minecraft on a raspberry pi 4. With 4 players maximum allocated has been 1600 mb, but that is of course because I am allowing it to utilize a lot more. The server with PAPER instead of SPIGOT can probably run a lot lower. The 512mb is from the stable helm chart and the the typicall hello world server allocates 1 G. CPU wise the node idles around 10%, and I am guessing minecraft utilizes what it needs on any of the 4 CPUs, I haven't got any good numbers there.
@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 8, 2020

So any idea why the minecraft server won't start properly?

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 8, 2020

@justcallmekoko You can first try to uninstall the helm chart and then reinstall it with higher memory (792 if you are on a raspberry pi 3 or 1800 if you are on raspberry pi 4). Try VANILLA instead of spigot or try PAPER. With vanila Minecraft you will get better logs so maybe that is an option when trying to install it.

@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 8, 2020

Alright I will try 1800 and I will try VANILLA and PAPER if it doesn't work. I tried 1500, but that didn't seem to do it.

Also how can I verify the resources of the nodes are being used to run the server?

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 8, 2020

kubectl top node minecraft-node

and to see what is running on that node

kubectl get node minecraft-node -o yaml

Maybe when it is up you can portforward into the pod and try to connect your minecraft app to 127.0.0.1 like

kubectl port-forward -n minecraft minecraft-minecraft-<POD_ID> 25565
@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 8, 2020

when tainting the nodes, do you taint the master as well?

@justcallmekoko

This comment has been minimized.

Copy link

justcallmekoko commented Apr 8, 2020

Running PAPER and VANILLA gives me these results

pi@cluster-master:~ $ kubectl get pods -A
NAMESPACE     NAME                                           READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-58fb86bdfd-chzkh        1/1     Running     0          40m
kube-system   metrics-server-6d684c7b5-k2rvh                 1/1     Running     0          40m
kube-system   coredns-d798c9dd-h62wk                         1/1     Running     0          40m
kube-system   helm-install-traefik-csnsr                     0/1     Completed   2          41m
kube-system   svclb-traefik-r52n8                            2/2     Running     0          40m
kube-system   traefik-6787cddb4b-gmzmv                       1/1     Running     0          40m
kube-system   svclb-traefik-m6xf7                            2/2     Running     0          38m
minecraft     svclb-cluster-minecraft-minecraft-mmsfd        1/1     Running     0          13s
minecraft     cluster-minecraft-minecraft-7578b99799-j7mn8   0/1     Running     0          13s
pi@cluster-master:~ $ kubectl logs cluster-minecraft-minecraft-7578b99799-j7mn8 --namespace=minecraft --follow=true
chmod: changing permissions of '/data': Operation not permitted
Running as uid=1000 gid=1000 with /data as 'drwxrwxrwx 2 0 0 4096 Apr  8 21:35 /data'
Checking version information.
Checking type information.
Downloading Paper 1.15.2 from https://papermc.io/api/v1/paper/1.15.2/latest/download ...
Creating server.properties in /data/server.properties
Setting server-name to 'Dedicated Server' in /data/server.properties
Setting server-port to '25565' in /data/server.properties
Setting motd to 'Welcome to Minecraft on Kubernetes on Rapsberry Pi!' in /data/server.properties
Setting allow-nether to 'true' in /data/server.properties
Setting announce-player-achievements to 'true' in /data/server.properties
Setting enable-command-block to 'true' in /data/server.properties
Setting spawn-animals to 'true' in /data/server.properties
Setting spawn-monsters to 'true' in /data/server.properties
Setting spawn-npcs to 'true' in /data/server.properties
Skip setting spawn-protection
Setting generate-structures to 'true' in /data/server.properties
Setting view-distance to '7' in /data/server.properties
Setting hardcore to 'false' in /data/server.properties
Skip setting snooper-enabled
Setting max-build-height to '256' in /data/server.properties
Skip setting force-gamemode
Setting max-tick-time to '60000' in /data/server.properties
Skip setting enable-query
Skip setting query.port
Setting enable-rcon to 'true' in /data/server.properties
Setting rcon.password to 'minecraft' in /data/server.properties
Setting rcon.port to '25575' in /data/server.properties
Setting max-players to '20' in /data/server.properties
Setting max-world-size to '10000' in /data/server.properties
Setting level-name to 'world' in /data/server.properties
Skip setting level-seed
Setting pvp to 'false' in /data/server.properties
Skip setting generator-settings
Setting online-mode to 'true' in /data/server.properties
Skip setting allow-flight
Setting level-type to 'DEFAULT' in /data/server.properties
Skip setting resource-pack
Skip setting resource-pack-sha1
Setting difficulty to '1' in /data/server.properties
Setting mode
Setting gamemode to '0' in /data/server.properties
Checking for JSON files.
Copying any mods over...
Copying any Bukkit plugins over...
Setting initial memory to 1800M and max to 1800M
Starting the Minecraft server...
Downloading vanilla jar...
Patching vanilla jar...
2020/04/08 21:46:41 Stopping with rcon-cli
2020/04/08 21:46:41 ERROR Failed to stop using rcon-cli exit status 1
2020/04/08 21:46:41 Sending 'stop' to Minecraft server...
2020/04/08 21:46:41 Waiting for completion...
@texas-home-pros

This comment has been minimized.

Copy link

texas-home-pros commented Apr 17, 2020

Can this be used to run a vanilla server (no spigot) regardless of monitoring tools just need to run a vanilla on a cluster

@gabrielsson

This comment has been minimized.

Copy link
Owner Author

gabrielsson commented Apr 17, 2020

@texas-home-pros Yes no problem, just run the guide up until the checkpoint. SPIGOT or PAPER is running a bit more lightweight in regards to memory pressure supposedly but I have used VANILLA as well.

You set it in the values.yaml file.

@texas-home-pros

This comment has been minimized.

Copy link

texas-home-pros commented Apr 18, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.