Using Your Own Web Server For ROMs and BIOS Files For RetroArch

RetroArch Assets.md

Table of Contents

1. Purpose
2. Before you begin
3. Background - Understanding RetroArch behavior
4. Starting point - mimic the buildbot
5. Mirroring assets
6. Automatically creating .index and .index-dirs files
7. Proxying S3

Purpose

RetroArch's Online Updater is capable of downloading Core System Files (BIOS dumps, etc.) and Content (ROMs). The files available by default are ones that are freely distributable, mostly homebrew. It is possible to override the location the Online Updater uses to point at your own collection of ROMs. This is not a great way of managing your ROMs; most people will be better served by using your computer's file manager. However, for some devices (e.g. tvOS), copying ROMs to the device can be a cumbersome process, and the Online Updater is one more tool you can use to try to make it easier.

Sharing ROMs is illegal! This guide is to make managing your private ROM collection easier. Do not share your ROM collection with others.

Before you begin

This can get complicated, and there are lots of options for how you set up the server. I recommend you read the entire guide first and decide on what your goal is. With a little bit of proper prior planning, you can save yourself some time and wasted effort. There is not a "quick and easy" or "one size fits all" solution.

Background - Understanding RetroArch behavior

RetroArch's Online Updater has two base URLs that it uses, one for cores and one for assets, system files, and content. For our purposes, we are going to leave the URL for cores alone. The assets URL is kept in the config file under the key core_updater_buildbot_assets_url, and can be changed through the Updater Settings in the Online Updater. It defaults to http://buildbox.libretro.com/assets/. There are three subdirectories at that URL:

• frontend - This contains core info files, assets, controler profiles, etc. The files in here have names that are hard-coded in RetroArch.
• system - This contains the core system files (BIOS dumps). The only path that is hard-coded in RetroArch is .index, which is a text file with one filename per line.
• cores - This contains the content (ROMs). The only path that is hard-coded in RetroArch is .index-dirs, which is a text file with one directory name per line. Inside each directory the only path that is hard-coded is .index, which is a text file with one filename per line.

If the filename ends in .zip or .7z, RetroArch will unzip it. The "Automatically Extract Downloaded Archive" option only applies to the content from the cores directory, and defaults to On.

RetroArch fetches everything using the net_http client in libretro, which does not support most standard HTTP features, most notably, redirects. Also, most platforms do not include SSL support by default.

Starting point - mimic the buildbot

The easiest possible initial setup is to completely replace the buildbot infrastructure. Using the description of what's contained in the buildbot assets above, this should be fairly straightforward.

• frontend - This can just be a straight mirror of the buildbot, https://buildbot.libretro.com/assets/frontend/.
• system - Put all of your system files here, and create a .index file with a list of all of the files.
• cores - Create directories here of your choosing, and a .index-dirs file with a list of those directories. Inside each directory, put the ROM files, with a .index file with the list of files in that directory.

You can use pretty much any web server that you want, including just hosting all of these files directly on S3 (or similar). Setting up S3 to be a static file web server is possible but outside the scope of this document.

Mirroring assets

There are four strategies for how to build and maintain the frontend assets:

1. Do nothing. You aren't required to have the frontend assets if you never use that part of the Online Updater.

2. Fetch and forget. Just download them once and keep that version on your server forever. If you don't regularly add new ROMs to your collection, and don't use cheats and shaders and care about getting the latest ones, this is a very reasonable strategy.

3. Regularly update the mirror. Set up an automated task that periodically downloads the updated assets from the buildbot and uploads them to your server. You might choose to only update the packs you particularly care about. The zip files on the buildbot are updated nightly whether there are changes or not, so it can be hard to discern if something has changed that is worth downloading, so this might use bandwidth needlessly. The files are relatively small so this may not be a concern.

4. Proxy the buildbot. You cannot set up your web server to simply redirect to the buildbot, as RetroArch does not follow standard HTTP 3xx redirects. However, depending on which web server you are using, you could set it up to fetch the file from the buildbot and proxy it itself. With apache, you could create an .htaccess file to proxy them with the following rule:

   RewriteRule "^frontend/(.*)\.zip$" "https://buildbot.libretro.com/assets/frontend/$1.zip" [P]
   

   This obviously doesn't work if you're using S3 as your web server.

Automatically creating .index and .index-dirs files

If you regularly add new ROMs to your collection, you might prefer to have the .index and .index-dirs files generated automatically. This also does not work if you're using S3 as your web server, but most other web servers offer some type of server side scripting.

As an example using apache and PHP, you can modify the .htaccess file to watch for .index and .index-dirs requests, and change it to be handled by a PHP script:

RewriteRule "^(.*)/\.index$" "assets.php?files=$1" "[B= ?,L]"
RewriteRule "^(.*)/\.index-dirs$" "assets.php?dirs=$1" "[B= ?,L]"

You can then have a PHP script which just lists all of the files or directories:

<?
header('Content-Type: text/plain');
if (array_key_exists('files', $_GET)) {
    $path=realpath($_GET['files']);
    $type="f";
} else if (array_key_exists('dirs', $_GET)) {
    $path=realpath($_GET['dirs']);
    $type="d";
}
$root=dirname($_SERVER['SCRIPT_FILENAME']);
if (substr($path, 0, strlen($root)) == $root) {
   echo `find "{$path}" -mindepth 1 -maxdepth 1 -type {$type} -printf "%f\n" | sort`;
}
?>

Proxying S3

You might have decided that you want to use your own apache server so that it can build the .index files automatically, but your server provider does not give you enough storage for all of your ROMS, and want to put them on S3. In that case, you will want to set up your apache .htaccess file to proxy from your s3 server:

RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-s
RewriteRule "^(.+)" "https://mys3.server.com/path/$1" [P]

Then you will need to change the script so that instead of building the index by finding files and directories locally, it finds the files and directories on your s3 store. That, or mimic the s3 store with empty files.

You can do something like:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/retroarch

    <Directory /var/www/retroarch>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>

    # Rewrite rules for dynamically generating .index and .index-dirs
    RewriteEngine On

    # Rewrite rule for .index files
    RewriteRule "^/system/(.*)/\.index$" "/index.php?path=system/$1" [L]
    RewriteRule "^/cores/(.*)/\.index$" "/index.php?path=cores/$1" [L]
    RewriteRule "^/cores/\.index-dirs$" "/index.php?dirs=cores" [L]

    # Proxy frontend assets from the buildbot
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule "^/frontend/(.*)\.zip$" "https://buildbot.libretro.com/assets/frontend/$1.zip" [P]

    # Error handling
    ErrorLog ${APACHE_LOG_DIR}/retroarch_error.log
    CustomLog ${APACHE_LOG_DIR}/retroarch_access.log combined
</VirtualHost>

And in PHP for index.php:

<?php
header('Content-Type: text/plain');

if (isset($_GET['path'])) {
    $path = realpath($_GET['path']);
    if ($path) {
        $files = scandir($path);
        foreach ($files as $file) {
            if ($file !== '.' && $file !== '..' && is_file("$path/$file")) {
                echo $file . "\n";
            }
        }
    }
} elseif (isset($_GET['dirs'])) {
    $path = realpath($_GET['dirs']);
    if ($path) {
        $dirs = scandir($path);
        foreach ($dirs as $dir) {
            if ($dir !== '.' && $dir !== '..' && is_dir("$path/$dir")) {
                echo $dir . "\n";
            }
        }
    }
}
?>