Skip to content

Instantly share code, notes, and snippets.

@sfan5
Last active July 10, 2022 20:20
  • Star 9 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save sfan5/6351560 to your computer and use it in GitHub Desktop.
Collects media files from mods/games and puts them in a media directory, also creates an index.mth file in the MTHS format
#!/bin/bash
###### Options ######
MINETESTDIR=/home/username/minetest
GAMENAME=minetest_game
WORLDDIR=none
MEDIADIR=./media
LEGACY_SYMLINKS=0
# When settings this up be aware that the Minetest client
# will send a POST request to index.mth. You need to configure
# your web server to serve the file on POST requests for this to work.
#####################
die () {
echo "$1" >&2
exit 1
}
collect_from () {
echo "Processing media from: $1"
find -L "$1" -type f -name "*.png" -o -name "*.ogg" -o -name "*.x" -o -name "*.b3d" | while read f; do
basename "$f"
hash=`openssl dgst -sha1 <"$f" | cut -d " " -f 2`
cp "$f" $MEDIADIR/$hash
[ $LEGACY_SYMLINKS -eq 1 ] && ln -nsf $hash $MEDIADIR/`basename "$f"`
done
}
[ -d "$MINETESTDIR" ] || die "Specify a valid Minetest directory!"
which openssl &>/dev/null || die "OpenSSL not installed."
mkdir -p $MEDIADIR
[ ! $GAMENAME == none ] && collect_from "$MINETESTDIR/games/$GAMENAME/mods"
[ ! "$WORLDDIR" == none ] && collect_from "$WORLDDIR/worldmods"
[ -d "$MINETESTDIR/mods" ] && collect_from "$MINETESTDIR/mods"
printf "Creating index.mth... "
printf "MTHS\x00\x01" >$MEDIADIR/index.mth
find $MEDIADIR -type f -not -name index.mth | while read f; do
openssl dgst -binary -sha1 <$f >>$MEDIADIR/index.mth
done
echo "done"
@est31
Copy link

est31 commented Jul 5, 2016

What about putting it into the util or misc directory of the minetest repo?

@Cat5TV
Copy link

Cat5TV commented Aug 15, 2016

This is awesome, thanks! Wondering is there a way you could modify it to leave the file extensions intact? Naming as a hash is fine, but by removing the .png extension (for example) this breaks a proxy cache, which could be leveraged to hugely improve the performance of the given server.

For example, a .png file will be cached by the proxy based on the filename something**.png** - but naming it instead 1b2e0ad7a04c551be776c3c0ed851c30 without a file extension makes it so the proxy cache just passes through the file every time since it does not know it is actually a png... so every request results in a pull from the server, rather than the cache. That also means the user is accessing the file from the main server, rather than a CDN node.

Naming the file instead, 1b2e0ad7a04c551be776c3c0ed851c30**.png** would lead the proxy to cache it locally. In the case of a CDN (which is our case) that would mean the file would be served from a local server to the player and not pulled from our main server.

Would love to see this changed so we can leverage the proxy cache on our servers!!

Thanks!
RobbieF

@sfan5
Copy link
Author

sfan5 commented Aug 16, 2016

No, that's not possible. Minetest looks for the files with just the sha1 hash, adding file extensions would break it.

Can't you tell your CDN to forcefully cache everything in a single folder?

@Cat5TV
Copy link

Cat5TV commented Aug 16, 2016

Depends on which cache I put it on, but the one I would like to use requires extensions as that's how it determines the cache policy to use.

Is it possible to have the hashed file + the index.mth on different servers? Or does the mht file have to be located in the same folder as all the files? This is also a problem since the ideal cache server doesn't allow POST operations (static file serving only).

@sfan5
Copy link
Author

sfan5 commented Aug 16, 2016

Nope, both files need to be at the same path.

@Cat5TV
Copy link

Cat5TV commented Aug 16, 2016

K thanks sfan5. Not perfect, but still will help offload the game server itself.

@ShadowNinja
Copy link

I made this as an alternative: https://github.com/ShadowNinja/mt_media_collector

@taikedz
Copy link

taikedz commented Jan 16, 2017

I'd like to edit this a little and share - what license do you wish to detail please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment