Spigot restart script issues and solution

README.md

⚠️ This guide is for self-managed servers. If you rented a server and got a website through which you manage your server - you are on so called shared-hosting and this guide won't work for you.

What is spigot restart script

Spigot added a feature allowing for server to automatically restart on crashes, as well as when authorized user executes /restart command.

How it works

When server starts to shut down, it spawns a new process which runs your restart script, as specified in spigot.yml under settings.restart-script, which defaults to ./start.sh. The idea is that the original server will stop completely, and the new instance will take its place.

Why is it an issue

Because the new server is created before old one fully stopped, this means:

1. You need enough spare resources, to temporarily run 2 servers at the same time, as the old instance haven't freed up CPU/RAM while the new one starts and prepares itself.
2. You might hit Unable to bind to port errors, because old server is still occupying the port.
3. You might hit Is another instance running error, mentioning that world/session.lock is in use

Because the new server is being detached, that means that if you use tmux or screen to start your server, it no longer will give you access to server console after restart.

How can this be fixed

The solution is simple - don't let the restart script start any server, but instead have it communicate with your start script that when old instance is shut down, it should start a new one instead.

How can I do it?

The easiest way is to simply create 2 scripts - start.sh which will be the start script, and restart.sh which will be used by Spigot to notify that a restart is needed.

First, configure spigot.yml to use your restart script instead:

settings:
  restart-script: ./restart.sh

Finally, copy the attached to this gist start.sh and restart.sh scripts and place them in your server folder from where you start it, which usually is the same directory where server jar, settings and worlds are in. Make sure to understand what they do and change memory setting and other flags to your liking. You'll also need to mark them as executable, by running chmod +x *.sh.

Now you can start your tmux/screen session as usual and run ./start.sh. That's it!

But what about Windows?

I haven't used Windows in years, so I don't know batch or PS language well enough. If someone can rewrite these scripts to Windows equivalents, I'll gladly add them.

I have questions / issues

If you have questions, its best to ask them here so others can see. Alternatively, reach me over on Discord Prof_Bloodstone#0123 or email papermc@bloodstone.dev.

Known issues

Illegal option -o pipefail

Error: start.sh: 4: set: Illegal option -o pipefail

It means that the script was started with a shell not supporting pipefail option. This is most common, when you start the script as sh start.sh while sh is symlinked to dash or other minimal shell:

$ which sh
/bin/sh
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Feb 23 18:52 /bin/sh -> dash

Solution: Start it using bash - either use bash start.sh, or first mark the file as executable with chmod +x start.sh and then execute the file directly using ./start.sh.

bad substitution

Error: start.sh: line 55: ${restart_on_crash,,}: bad substitution

It means that you sucessfully started the script using bash, but it's probably too old. Run bash --version to confirm - I believe required features were added in version 4.

Solutions:

1. Upgrade bash to more modern version
2. Replace the mentioned substitution to remove the lowercasing (remove the two ,,). From now on you need to make sure that restart_on_crash setting is lowercase.

Restart doesn't work

Script copied from Windows

If you copied the script from Windows machine, there's a chance that DOS line endings were added. These endings are called CRLF typed as \r\n. Many Linux editors will display ^M in place of CR/\r.

Solutions:

1. Save the file using Linux line ending LF/\n
2. Run a tool like dos2unix on the file to convert it for you

restart.sh

#!/usr/bin/env sh
# We need to use sh, since it's hardcoded in spigot
# Doesn't matter much, since it doesn't do much anyway

# This script only creates a file which tells `start.sh` that it should restart, instead of exiting.

# Make sure to change this, if you modified `restart_flag` in `start.sh`!
touch .restart_flag

start.sh

#!/usr/bin/env bash
# NOTE: We use bash for better readability and error handling here
# but it's not hard to make it work with regular shell
set -euo pipefail

# SETTINGS
# Path to file used to communicate from restart script
readonly restart_flag='.restart_flag'
# How long (in seconds) to wait before restarting
readonly restart_delay=10
# Whether to restart on crash or not
# The `settings.restart-on-crash` setting in spigot.yml doesn't always work
# but also sometimes server might not return proper exit code,
# so it's best to keep both options enabled
# Accepted values: y/yes/true/n/no/false
readonly restart_on_crash='yes'
# The name of your server jar
readonly server_jar='paperclip.jar'
# What will be passed to `-Xms` and `-Xmx`
readonly heap_size='1G'
# JVM startup flags, one per line for better readability
# NOTE: -Xms and -Xmx are set separately
# These are mostly "Aikar flags"
# taken from: https://mcflags.emc.gs/
readonly jvm_flags=(
  -XX:+UseG1GC
  -XX:+ParallelRefProcEnabled
  -XX:MaxGCPauseMillis=200
  -XX:+UnlockExperimentalVMOptions
  -XX:+DisableExplicitGC
  -XX:+AlwaysPreTouch
  -XX:G1NewSizePercent=30
  -XX:G1MaxNewSizePercent=40
  -XX:G1HeapRegionSize=8M
  -XX:G1ReservePercent=20
  -XX:G1HeapWastePercent=5
  -XX:G1MixedGCCountTarget=4
  -XX:InitiatingHeapOccupancyPercent=15
  -XX:G1MixedGCLiveThresholdPercent=90
  -XX:G1RSetUpdatingPauseTimePercent=5
  -XX:SurvivorRatio=32
  -XX:+PerfDisableSharedMem
  -XX:MaxTenuringThreshold=1
  -Dusing.aikars.flags=https://mcflags.emc.gs
  -Daikars.new.flags=true
)
# Minecraft args you might want to start your server with
# Usually there isn't much to configure here:
readonly mc_args=(
  --nogui # Start the server without GUI
)
# END OF SETTINGS

should_restart_on_crash() {
  case "${restart_on_crash,,}" in
    y|yes|true) return 0;;
    n|no|false) return 1;;
    *)
      printf 'ERROR: Invalid value for "restart_on_crash" variable: %s\n' "${restart_on_crash}" >&2
      exit 1
      ;;
  esac
}

# The arguments that will be passed to java:
readonly java_args=(
  -Xms"${heap_size}" # Set heap min size
  -Xmx"${heap_size}" # Set heap max size
  "${jvm_flags[@]}" # Use jvm flags specified above
  -jar "${server_jar}" # Run the server
  "${mc_args[@]}" # And pass it these settings
)

# Remove restart flag, if it exists,
# so that we won't restart the server after first stop,
# unless restart script was called
rm "${restart_flag}" &>/dev/null || true

# Check if `restart_on_crash` has valid value
should_restart_on_crash || true

while :; do # Loop infinitely
  # Run server
  java "${java_args[@]}" || {
    # Oops, server didn't exit gracefully
    printf 'Detected server crash (exit code: %s)\n' "${?}" >&2
    # Check if we should restart on crash or not
    if should_restart_on_crash; then
      touch "${restart_flag}"
    fi
  }
  # Check if restart file exists or exit
  if [ -e "${restart_flag}" ]; then
    # The flag exists - try to remove it
    rm "${restart_flag}" || {
      # If we can't remove it (permissions?), then exit to avoid endless restart loop
      printf 'Error removing restart flag (exit code: %s) - cowardly exiting\n' "${?}" >&2
      exit 1
    }
  else
    break # Flag doesn't exist, so break out of the loop
  fi
  printf 'Restarting server in 10 seconds, press Ctrl+C to abort.\n' >&2
  sleep "${restart_delay}" || break # Exit if sleep is interrupted (for example Ctrl+C)
done

printf 'Server stopped\n' >&2