Spigot restart script issues and solution

README.md

⚠️ This guide is for self-managed servers. If you rented a server and got a website through which you manage your server - you are on so called shared-hosting and this guide won't work for you.

What is spigot restart script

Spigot added a feature allowing for server to automatically restart on crashes, as well as when authorized user executes /restart command.

How it works

When server starts to shut down, it spawns a new process which runs your restart script, as specified in spigot.yml under settings.restart-script, which defaults to ./start.sh. The idea is that the original server will stop completely, and the new instance will take its place.

Why is it an issue

Because the new server is created before old one fully stopped, this means:

1. You need enough spare resources, to temporarily run 2 servers at the same time, as the old instance haven't freed up CPU/RAM while the new one starts and prepares itself.
2. You might hit Unable to bind to port errors, because old server is still occupying the port.
3. You might hit Is another instance running error, mentioning that world/session.lock is in use

Because the new server is being detached, that means that if you use tmux or screen to start your server, it no longer will give you access to server console after restart.

How can this be fixed

The solution is simple - don't let the restart script start any server, but instead have it communicate with your start script that when old instance is shut down, it should start a new one instead.

How can I do it?

The easiest way is to simply create 2 scripts - start.sh which will be the start script, and restart.sh which will be used by Spigot to notify that a restart is needed.

First, configure spigot.yml to use your restart script instead:

settings:
  restart-script: ./restart.sh

Finally, copy the attached to this gist start.sh and restart.sh scripts and place them in your server folder from where you start it, which usually is the same directory where server jar, settings and worlds are in. Make sure to understand what they do and change memory setting and other flags to your liking. You'll also need to mark them as executable, by running chmod +x *.sh.

Now you can start your tmux/screen session as usual and run ./start.sh. That's it!

What about Windows?

I haven't used Windows in years, so I don't know batch or PS language well enough. @ipiepiepie rewrote the original Linux scripts to Windows batch, and they are attached below. Download start.bat and restart.bat instead, and everywhere you see start.sh and restart.bat in this readme, replace it .sh with .bat. Also make sure to remove the leading ./, for example ./start.sh -> start.bat. And there's no chmod on Windows, so skip that step too :)

I have questions / issues

If you have questions, its best to ask them here so others can see. Alternatively, reach me over on Discord Prof_Bloodstone#0123 or email papermc@bloodstone.dev.

Known issues - Linux

Illegal option -o pipefail

Error: start.sh: 4: set: Illegal option -o pipefail

It means that the script was started with a shell not supporting pipefail option. This is most common, when you start the script as sh start.sh while sh is symlinked to dash or other minimal shell:

$ which sh
/bin/sh
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Feb 23 18:52 /bin/sh -> dash

Solution: Start it using bash - either use bash start.sh, or first mark the file as executable with chmod +x start.sh and then execute the file directly using ./start.sh.

bad substitution

Error: start.sh: line 55: ${restart_on_crash,,}: bad substitution

It means that you sucessfully started the script using bash, but it's probably too old. Run bash --version to confirm - I believe required features were added in version 4.

Solutions:

1. Upgrade bash to more modern version
2. Replace the mentioned substitution to remove the lowercasing (remove the two ,,). From now on you need to make sure that restart_on_crash setting is lowercase.

Restart doesn't work

Script copied from Windows

If you copied the script from Windows machine, there's a chance that DOS line endings were added. These endings are called CRLF typed as \r\n. Many Linux editors will display ^M in place of CR/\r.

Solutions:

1. Save the file using Linux line ending LF/\n
2. Run a tool like dos2unix on the file to convert it for you

Known issues - Windows

'.' is not recognized as an internal or external command

It can appear on restart if you forgot to remove ./ in start of your restart-script in spigot.yml

Solutions:

1. Change ./restart.bat to restart.bat in spigot.yml

restart.bat

@echo off
:: We need to use script, since it's hardcoded in spigot
:: Doesn't matter much, since it doesn't do much anyway

:: This script only creates a file which tells `start.bat` that it should restart, instead of exiting.

:: Make sure to change this, if you modified `restart_flag` in `start.bat`!
type nul > .restart_flag

exit 0

restart.sh

#!/usr/bin/env sh
# We need to use sh, since it's hardcoded in spigot
# Doesn't matter much, since it doesn't do much anyway

# This script only creates a file which tells `start.sh` that it should restart, instead of exiting.

# Make sure to change this, if you modified `restart_flag` in `start.sh`!
touch .restart_flag

start.bat

@echo off

:: SETTINGS
:: Path to file used to communicate from restart script
set "restart_flag=.restart_flag"
:: How long (in seconds) to wait before restarting
set "restart_delay=5"
:: Whether to restart on crash or not
:: The `settings.restart-on-crash` setting in spigot.yml doesn't always work
:: but also sometimes server might not return proper exit code,
:: so it's best to keep both options enabled
:: Accepted values: y/yes/true/n/no/false
set "restart_on_crash=yes"
:: The name of your server jar
set "server_jar=craftbukkit.jar"
:: What will be passed to `-Xms` and `-Xmx`
set "heap_size=1G"
:: JVM startup flags
:: NOTE: -Xms and -Xmx are set separately
:: These are mostly "Aikar flags"
:: taken from: https://mcflags.emc.gs/
:: TODO try to add one per line for better readability
set "jvm_flags=-XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true"
:: Minecraft args you might want to start your server with
:: Usually there isn't much to configure here:
set "mc_args=--nogui"

:: END OF SETTINGS

:: Parse yes
if %restart_on_crash% == yes (set "should_restart_on_crash=1")
if %restart_on_crash% == y (set "should_restart_on_crash=1")
if %restart_on_crash% == true (set "should_restart_on_crash=1")
:: Parse no
if %restart_on_crash% == no (set "should_restart_on_crash=0")
if %restart_on_crash% == n (set "should_restart_on_crash=0")
if %restart_on_crash% == false (set "should_restart_on_crash=0")
:: If we can't initialise %restart_on_crash%, then user input is incorrect
if not defined should_restart_on_crash (
  echo ERROR: Invalid value for ^"restart_on_crash^" variable: %restart_on_crash%
  exit /b 1
)

:: The arguments that will be passed to java:
set jvm_args=-Xms%heap_size% -Xmx%heap_size% %jvm_flags% -jar %server_jar% %mc_args%

:: Remove restart flag, if it exists,
:: so that we won't restart the server after first stop,
:: unless restart script was called
del %restart_flag% 2>nul

:: Loop infinitely
:loop
  :: Run server
  java %jvm_args% || (
    :: Oops, server didn't exit gracefully
    echo Detected server crash ^(exit code: %ERRORLEVEL%^)
    :: Check if we should restart on crash or not
    if %should_restart_on_crash% == 1 (
      type nul > %restart_flag%
    )
  )

  :: Check if restart file exists or exit
  if exist %restart_flag% (
    :: The flag exists - try to remove it
    del %restart_flag% 2>nul || (
      :: If we can't remove it (permissions?), then exit to avoid endless restart loop
      echo Error removing restart flag ^(exit code: %ERRORLEVEL%^)
      exit /b 1
    )
  ) else (
    :: Flag doesn't exist, so break out of the loop
    exit /b 0
  )

  :: Restart server with delay
  echo Restarting server in %restart_delay% seconds, press Ctrl+C to abort.
  timeout /T %restart_delay% /NOBREAK > nul || exit /b 0 REM Exit if timeout is interrupted (for example Ctrl+C)
goto loop

echo Server stopped.

start.sh

#!/usr/bin/env bash
# NOTE: We use bash for better readability and error handling here
# but it's not hard to make it work with regular shell
set -euo pipefail

# SETTINGS
# Path to file used to communicate from restart script
readonly restart_flag='.restart_flag'
# How long (in seconds) to wait before restarting
readonly restart_delay=10
# Whether to restart on crash or not
# The `settings.restart-on-crash` setting in spigot.yml doesn't always work
# but also sometimes server might not return proper exit code,
# so it's best to keep both options enabled
# Accepted values: y/yes/true/n/no/false
readonly restart_on_crash='yes'
# The name of your server jar
readonly server_jar='paperclip.jar'
# What will be passed to `-Xms` and `-Xmx`
readonly heap_size='1G'
# JVM startup flags, one per line for better readability
# NOTE: -Xms and -Xmx are set separately
# These are mostly "Aikar flags"
# taken from: https://mcflags.emc.gs/
readonly jvm_flags=(
  -XX:+UseG1GC
  -XX:+ParallelRefProcEnabled
  -XX:MaxGCPauseMillis=200
  -XX:+UnlockExperimentalVMOptions
  -XX:+DisableExplicitGC
  -XX:+AlwaysPreTouch
  -XX:G1NewSizePercent=30
  -XX:G1MaxNewSizePercent=40
  -XX:G1HeapRegionSize=8M
  -XX:G1ReservePercent=20
  -XX:G1HeapWastePercent=5
  -XX:G1MixedGCCountTarget=4
  -XX:InitiatingHeapOccupancyPercent=15
  -XX:G1MixedGCLiveThresholdPercent=90
  -XX:G1RSetUpdatingPauseTimePercent=5
  -XX:SurvivorRatio=32
  -XX:+PerfDisableSharedMem
  -XX:MaxTenuringThreshold=1
  -Dusing.aikars.flags=https://mcflags.emc.gs
  -Daikars.new.flags=true
)
# Minecraft args you might want to start your server with
# Usually there isn't much to configure here:
readonly mc_args=(
  --nogui # Start the server without GUI
)
# END OF SETTINGS

should_restart_on_crash() {
  case "${restart_on_crash,,}" in
    y|yes|true) return 0;;
    n|no|false) return 1;;
    *)
      printf 'ERROR: Invalid value for "restart_on_crash" variable: %s\n' "${restart_on_crash}" >&2
      exit 1
      ;;
  esac
}

# The arguments that will be passed to java:
readonly java_args=(
  -Xms"${heap_size}" # Set heap min size
  -Xmx"${heap_size}" # Set heap max size
  "${jvm_flags[@]}" # Use jvm flags specified above
  -jar "${server_jar}" # Run the server
  "${mc_args[@]}" # And pass it these settings
)

# Remove restart flag, if it exists,
# so that we won't restart the server after first stop,
# unless restart script was called
rm "${restart_flag}" &>/dev/null || true

# Check if `restart_on_crash` has valid value
should_restart_on_crash || true

while :; do # Loop infinitely
  # Run server
  java "${java_args[@]}" || {
    # Oops, server didn't exit gracefully
    printf 'Detected server crash (exit code: %s)\n' "${?}" >&2
    # Check if we should restart on crash or not
    if should_restart_on_crash; then
      touch "${restart_flag}"
    fi
  }
  # Check if restart file exists or exit
  if [ -e "${restart_flag}" ]; then
    # The flag exists - try to remove it
    rm "${restart_flag}" || {
      # If we can't remove it (permissions?), then exit to avoid endless restart loop
      printf 'Error removing restart flag (exit code: %s) - cowardly exiting\n' "${?}" >&2
      exit 1
    }
  else
    break # Flag doesn't exist, so break out of the loop
  fi
  printf 'Restarting server in 10 seconds, press Ctrl+C to abort.\n' >&2
  sleep "${restart_delay}" || break # Exit if sleep is interrupted (for example Ctrl+C)
done

printf 'Server stopped\n' >&2

While this is a matter of personal opinion, if this script is intended to run on a Linux servers than it should incorporate the screen command to run the console in a accessible terminal because one might wish to perform other actions on the machine while the server is running. If you want some examples for automatic creation of detached screen's and sending of commands to those screens hmu on discord at GalacticB69#4729.

it should incorporate the screen command to run the console in a accessible terminal because one might wish to perform other actions on the machine while the server is running

This is not the purpose of this script. Yes, I agree that you such use that tool when running the server, but don't couple it tightly into the script.
We'd quickly run into issues of duplicating code for screen, tmux and other multiplexers or starting methods. Instead, you should run the script directly in screen/tmux. For example, imagine following workflow used for starting the server:

PC $ ssh myserver.example.com
SERVER $ cd myserver
SERVER $ tmux new -s myserver
TMUX $ ./start.sh

You can easily view logs, whether it restarted or not etc. Without unnecessary coupling of the script to one specific multiplexer/tool.

Does that make sense?

I made this work on Windows by simply making a run.bat with "bash start.sh" then changed the restart.sh to restart.bat and make it do

ECHO >> .restart_flag
exit

Then using run.bat to actually start the server.

@HexedHero I believe this won't work for most people as Windows doesn't ship with bash. Do you know how have you installed it? Git for Windows?

@Prof-Bloodstone That's true, I forgot about that. I have Git for Windows installed yea.

@Prof-Bloodstone

This is not the purpose of this script. Yes, I agree that you such use that tool when running the server, but don't couple it tightly into the script. We'd quickly run into issues of duplicating code for screen, tmux and other multiplexers or starting methods. Instead, you should run the script directly in screen/tmux.

Hi, I discovered your wonderful script, and I both agree and disagree
Yes, the multiplexing feature should not go in start.sh, as the single task of start.sh is to run the server and handle reloads
However, I think start.sh should NOT be run manually from the server folder, and support for a third external bash script is necessary to both avoid coupling with a specific multiplexer, and limit the risk of human error when starting the server

Here's how I integrated screen

1. At the start of start.sh, add a cd command to go to the server folder
   That will ensure that launching it from anywhere won't cause unexpected issues
   In my case

#!/bin/sh
cd /home/ubuntu/mcserver/
(the entire original start.sh)

2. Create a third script that, in my case, runs start.sh in screen.
   Said script will be setup-specific and depend of the admin's multiplexer, and as a result IMHO should not be located in the server folder
   Because start.sh can now be called from anywhere, that init.sh can even be placed directly in the user's home folder

#!/bin/sh
screen -dmS "Minecraft" bash /home/ubuntu/mcserver/start.sh
screen -ls

Oh, and a huge thank you for solving a problem that I had encountered nearly a decade ago and still had no way to solve... yesterday

Hi, I'm wondering how is determined the time at which the server will reboot ? It seems that it's around 2:50am but I'm not sure why and I would like to change it. Thanks !

1. At the start of start.sh, add a cd command to go to the server folder
   That will ensure that launching it from anywhere won't cause unexpected issues
   In my case

@laplongejunior If you put the start.sh in the directory that your server is in, you can replace this line with cd "$(dirname -- "${BASH_SOURCE[0]}")" - this automatically changes your working directory to the one your script is in!

Hi, I'm wondering how is determined the time at which the server will reboot ? It seems that it's around 2:50am but I'm not sure why and I would like to change it. Thanks !

@Plop This script doesn't do restarts. Restarts might be done by your server host, a plugin, a cronjob etc. Look in the logs what happens before the server starts to shut down - maybe a restart command is being sent?

I've been having some major problems with trying to get sh files in general. So I'm thinking I'm either missing something completely or there is something wrong with my server. I'm running an ubuntu server 22.04 lts. I went ahead and put the start.sh and restart.sh files into my minecraft server file. When I tried to run start.sh I got the pipefail error so I went ahead and followed the directions and ran it again and got this. /usr/bin/env: ‘bash\r’: No such file or directory . I don't really know what to do next so if somebody could please help me that would be great.

@Mint-101 that \r means you have CRLF ending from Windows - see the second Known issues :)

Thank you so much! I got everything working now. I was using filezilla with windows to edit the files but I didn't think making the files in windows would affect the actual .sh file.

@Mint-101 If you like to work on Windows, Notepad++ is able to show the newline characters and change them if needed

The original notepad bundled with Windows will, obviously, bring back the characters. Unless I'm mistaken, Filezilla doesn't bundle a text editor and instead opens the system editor by default

@Prof-Bloodstone, thanks for your wonderful script, it saved me a lot of time.

I rewrote this script to .bat, so Windows users can use it without any external software like Git.

The only difference from the bash script is single lined jvm_flags and mc_args, because I didn't find any good solution to do it multiline. If you know how to implement it - feel free to contribute.

If you want, you can put link to it in gist, or just copy and attach .bat files.
Link to Windows script

Thanks @ipiepiepie . The scripts look fine, though I'm not able to test them. I've updated the gist to both - include your solution, as well as link to your gist.

Thanks, it worked on Windows with .bat files. Is there any advantage if I run it with Bash/Git for Windows, which I've also installed?