leiradel/retromods.md

## retromods.md

      
    Raw
  

              retromods.md
            
          
    Retromods

What is a mod?

A mod is a set of code and assets that changes or augments the way a game is run. Examples of mods are:

Music and sound packs that replace the original game music and sound.
Graphics packs that replace the original game graphics.
Level packs that change or replace the original game levels.
Achievements that are awarded to the players when they complete tasks in the game.
ROM patches that otherwise change the way the game is presented or played.

In the Retromods context, a mod can also control the emulation in a level above of the game or system that emulates it. Examples of this kind of mod are:

Challenges that change the sequence of the game, i.e. present all Super Mario World bosses in sequence and ask the player to beat them below the given allocated time.
Meta-achievements across games and systems, i.e. an achievement awarded when the player beats all Metroid games in all platforms.

A mod is a ZIP file

ZIP files are not streamable, but we don't need to unpack them in advance; they will be used in their compressed form by employing an API that abstracts access to their content.
The mod identifier is an URL

A mod is identified by its URL, and it can reference other mods by it. This will avoid name clashes when different people are authoring mods. The URL must use the http scheme.
The HTTP server at the given host and port of the URL must accept GET and HEAD requests to the local path part of the URI. GET requests must return the mod's ZIP package in the HTTP response, while HEAD requests can be used to check for mod updates. Therefore, HTTP servers should return valid Last-Modified header fields.
The frontend is only required to understand the http and file URI schemes. The later can be used during mod development to identify mods in the user's local file system. When development finishes, the mod must be uploaded to a public repository in an unspecified way and receive a publicly accessible URL by that repository, which will be used as its identifier.
Mods can be stored in any form

Mods can be stored in the file system, as blobs in a database, or in any other way, but they must be accessible by their URL.
Games are identified by UUID

A game is uniquely identified by its UUID. Mods can ask the frontend to load and run a specific game by its UUID.
Libretro cores are identified by name

A libretro core is uniquely identified by its name, which is the name of its binary file minus the extension. Mods can ask the frontend to load and run a game using a specific core by using its name.
Although libretro cores are a central piece of Retromods, nothing prevents a stand-alone emulator from implementing this specification. It will, however, be limited to the mods that work with the underlying emulator technology.
Emulated systems are identified by an acronym

Each system that can be emulated also has an unique name to identify them. These names are shorts of the complete system name, i.e. snes for Super Nintendo Entertainment System. The complete list is TBD.
The scripting language is Lua

Lua is easy to embed and extend, and it's fast and lean.
There is no magic, no surprises

The ZIP file must contain a metadata.json file with the following fields: uri, name, version, description, authors, category, keywords, and dependencies:
{
  "uri": "http://retromods.org/mods/smwhd",
  "name": "High-definition Super Mario World",
  "version": "1.0.0",
  "description": "Super Mario World remake with the best music and graphics replacements.",
  "author": "Winston Churchill",
  "category": "music graphics",
  "keywords": "music pack replacement super mario world mozart graphics pack sprite replacement super mario world pablo picasso",
  "dependencies": [
    "http://retroparadise.net/super_mario_world_hd.zip",
    "http://emuaudio.com/packs/supermarioaudio"
  ],
  "supported-games": [
    "03539380-3b2f-11e8-94cd-eb2885799cc5"
  ],
  "supported-cores": [
    "snes9x_libretro"
  ],
  "supported-systems": [
    "snes"
  ]
}
The field supported-games, if present, contains a list of game UUIDs with which the mod can be used. If this field is missing, the mod is applicable to any game of the supported systems. The same thing is true with the supported-cores and supported-systems fields.
Besides metadata.json, the ZIP file must also contain a main.lua file with has the mod's functionality. Both metadata.json and main.lua must exist in the root folder of the mod ZIP.
When a mod is added to the frontend, the frontend must download it's dependencies if they aren't already available. If one of the dependencies of the mod fails to be added to the frontend for whatever reason (i.e. failed download, corrupted package), the mod must not be added.
Other than that, the only guarantees are those made by code.
Mods work via event handlers

All modding functionality is done via Lua code. main.lua must return a function that will be called when a mod is activated by the frontend. This function will receive all its arguments as fields in a table passed as the first parameter:
return function(args)
  -- args.vfs: the read-only virtual file system of the mod.
  
  -- args.persistent: a read-write virtual file system for data the mod wants to
  -- make persistent. This vfs should be used for user-related content, such as
  -- high scores.
  
  -- args.cache: a read-write virtual file system for persistent, but not
  -- user-related, content that the mod wants to have around. Unlike the persistent
  -- vfs, cache is never synchronized, and the mod must be able to recreate its data
  -- if needed.
  
  -- args.scratch: a read-write virtual file system for a temporary area that the
  -- mod can use at will, but which will be erased when the mod is unloaded.

  -- args.submods: a map from each of the sub mods' URL to their respective
  -- read-only virtual file system.
end
A reference to the args table or any of its fields can be saved by the mod to be used at any moment. It can also be changed in any way the mod sees fit.
When executed, this function must return a table with the events that it's interested in:

onUnload: runs when the mod has been deactivated by the frontend.
onCoreLoadedruns when a libretro core has been loaded by the frontend.
onCoreUnloaded runs when a libretro core has been unloaded by the frontend.
onContentLoad: runs when a game has been loaded, but emulation has not been started.
onContentUnload: runs when a game has been unloaded.
onEmulatedFrame runs once per frame, right after the emulated frame has ended.
onEmulationPause: runs when the user pauses the emulation.

Note: This list is not final and can be changed as the project matures.
Each field must be a function that will be executed when the event is fired. The events are only fired to the main mod, it's up to the main mod to propagate events to its submods.
Example main.lua:
local frontend = require 'frontend'
local json = require 'json'

return function(self)
  local source = assert(self.vfs:read('/metadata.json'):result())
  self.metadata = assert(json.read(source))
  
  for url, vfs in pairs(self.submods) do
    local main = assert(vfs:read('/main.lua'):result())
    
    -- this won't break the iterator
    self.submods[url] = {
      vfs = vfs,
      handlers = assert(frontend.run(main))
    }
  end
  
  function self:onFrame()
    for url, mod in pairs(self.submods) do
      mod.handlers:onFrame()
    end
  end
  
  return self
end
Persistent user data

Modd receive a persistent virtual file system in the persistent field of the argument to their main function. This file system's content is persisted between runs of the mod, and can be used to store content that cannot be lost such as high scores.
Although not required, frontends should use 3rd-party online services to keep this user data available and synchronized online. Which online service and how the frontends keep the userdata synchronized is not specified here.
Backends must implement a set of required services

Backends wanting to offer mods discovery functionality for frontends must implement a set of REST services.
Note: These services are not final and can be changed as the project matures.
Querying mods

Mods can be queried in many different ways:

By game: /mods/?games=${game-uuid-list}. If the list has more than one game UUID, they must be separated by commas.
By category: /mods/?categories=${category-name-list}.
By keywords: /mods/?keywords=${keyword-list}.
By list: /mods/?list=${list-name}. list-name can be one of:

most-downloaded: mods ordered from the most downloaded to the least downloaded.
most-starred: mods ordered from the most starred by the site users to the least starred.
editor-picks: mods ordered from the most starred by the site staff to the least starred.
curated: mods ordered from the most starred by a specific user to the least starred. This list requires &user={user-id} to be appended to the URL, where user-id uniquely identifies an user on the backend.


Backends are not required to implement all queries. Queries not implemented should return a 501 Not Implemented HTTP status code.
The result is paginated in pages of 25 mods each. While frontends are expected to check for this limit, they are not required to handle more than 25 elements in each page. In other words, frontends need not prepare to parse responses with infinite items.
To request a specific page add &page=${page-number} to the URL. Page numbers start at 1. Any list can be reversed by adding &reversed=true to the URI. Games, categories, keywords, and one list can be freely intermixed to provide advanced ways to filter the collection of mods.
Example of response:
{
  "total-results": 150,
  "page-number": 1,
  "first-element-in-page": 1,
  "last-element-in-page": 25,
  "results": [
    {
      "url": "http://emuaudio.com/packs/supermarioaudio",
      "name": "Music pack for Super Mario World",
      "version": "1.2.1",
      "description": "Complete music replacement with original game music performed live by Mozart.",
      "author": "Mozart",
      "category": "music",
      "keywords": "music pack replacement super mario world mozart",
      "dependencies": [],
    },
    {
      "url": "http://retroparadise.net/super_mario_world_hd.zip",
      "name": "Graphics pack for Super Mario World",
      "version": "3.4.0",
      "description": "Complete graphics replacement with pixel art made by Picasso.",
      "author": "Pablo Picasso",
      "category": "graphics",
      "keywords": "graphics pack sprite replacement super mario world pablo picasso",
      "dependencies": []
    },
    {
      "url": "http://retromods.org/mods/smwhd",
      "name": "High-definition Super Mario World",
      "version": "1.0.0",
      "description": "Super Mario World remake with the best music and graphics replacements.",
      "author": "Winston Churchill",
      "category": "music graphics",
      "keywords": "music pack replacement super mario world mozart graphics pack sprite replacement super mario world pablo picasso",
      "dependencies": [
        "http://retroparadise.net/super_mario_world_hd.zip",
        "http://emuaudio.com/packs/supermarioaudio"
      ]
    },
    Other results follow...
  ]
}
Additional fields are available in each result depending on the specific query:

If keywords is used, each result will have a keywords field containing a number from 0 to 1, which is the relevance of the result in relation to the other results in the same result list. A result with a value of X is more relevant than a result with a value of Y if X > Y.
If a list if used, each result will have a list field containing a number, which range and meaning depends on the list:

most-downloaded: the total number of downloads since genesis.
most-starred, editor-picks, and curated: a number from 0 to 1, where 0 means "no starts" and 1 means "all stars."


Querying games

Games are identified with their UUIDs. To return the details of a game, the /games/${game-uuid} can be used.
Queries that retrieve game lists are:

By hash: /games/hash=${hash-list}. The hash is calculated by a system-specific algorithm that runs over the game content.
By system: /games/?system=${system-name-list}.
By keywords: /games/?keywords=${keywords}.

Each game has at least an UUID, its name, and the system name:
{
  "uuid": "03539380-3b2f-11e8-94cd-eb2885799cc5",
  "name": "Super Mario World",
  "system": "snes"
}
The resulting lists must be formated in the same way as the lists of mods shown above. As with mod queries, each result may have additional fields depending on how the query was made.
The backend is free to return additional information about the games if present in their databases, i.e.

A description of the game.
The year it has been released.
The name of the recommended libretro core to play the game.
The UUIDs of games in the same series.
URLs that can be used to view or download game art, screenshots, and videos.

Some of those information may be specified in the future to have a specific key in the resulting JSON.
Frontends should not offer the download of copyrighted games, nor should they offer directions to places where they can be downloaded.
Libretro cores

The backend must provide services to query libretro cores to be used with games and mods.
To retrieve the list of libretro cores that support a given system, use the /cores/?system=${system-name-list}&platform={platform} URL.
To retrieve information for a specific version of a core, add &version=${version} to the URL, where version is in the core-specific version format. If version is not informed, the result list only includes the latest versions.
As an example, the resulting list of the query for the /cores/?system=atari2600&platform=linux%2Fx86_64 could be:
[
  {
    "name": "Stella",
    "downloads": [
      {
        "version": "3.9.3",
        "platform": "linux/x86_64",
        "url": "http://buildbot.libretro.com/nightly/linux/x86_64/latest/stella_libretro.so.zip"
      }
    ]
  }
]
Note: System and platform names are TBD.
Script bindings

Lua scripts will have bindings available to communicate with the emulated system, its components, the frontend, and the backend. The exact set of functionalities is TBD.
Types

Lua is extended with some types that represent things that cannot be represented natively or in a efficient way in the language.
bytes

bytes represent a block of bytes. It has the following methods:

get8(address): returns the byte value at address. Byte values go from 0 to 255.
get16le(address): returns the word value at address, in little-endian format. This is equivalent to get8(address + 1) << 8 | get8(address).
get32le(address): returns the double word value at address, in little-endian format. This is equivalent to get16le(address + 2) << 16 | get16le(address).
get16be(address): returns the word value at address, in big-endian format. This is equivalent to get8(address) << 8 | get8(address + 1).
get32be(address): returns the double word value at address, in big-endian format. This is equivalent to get16be(address) << 16 | get16be(address + 2).
get8bcd(address): returns the byte value at address, in BCD format. This is equivalent to (get8(address) >> 4) * 10 + (get8(address) & 15). No checks are made to see if the value is a valid BCD value.
get16bcdle(address): returns the value of the word value at address, in BCD little-endian format. This is equivalent to get8bcd(address + 1) * 100 + get8bcd(address).
get32bcdle(address): returns the value of the double word value at address, in BCD little-endian format. This is equivalent to get16bcdle(address + 2) * 10000 + get16bcdle(address).
get16bcdbe(address): returns the value of the word value at address, in BCD big-endian format. This is equivalent to get8bcd(address) * 100 + get8bcd(address + 1).
get32bcdle(address): returns the value of the double word value at address, in BCD big-endian format. This is equivalent to get16bcdle(address) * 10000 + get16bcdle(address + 2).
getFloatle(address): return the value of the 4-byte float at address, stored in little-endian.
getFloatbe(address): return the value of the 4-byte float at address, stored in big-endian.
size(): the total number of bytes in the object.
clone(): returns a clone of the object. The contents are copied, so the cloned instance is completely independent of the original object.
view(offset, size): creates a new byte with a view of the object. offset and size default to 0 and size(), and allow copies of pieces of the original object. Content is not copied, the view points to the same content as the original object. In other words, changes in the content of the original object are observable in its views.
concat(blocks...): this static method creates a new byte that is the concatenation of the given byte instances. Likewise the views, content is not copied, accesses go to the correct byte.
create(size, value, rw): this static method creates a new byte which has size bytes all set to value. If rw is true, writes to the block are persisted, otherwise the block is optmized but does not persist writes.

byte instance also have setX(address, value) methods with the same variations of the getX(address) ones.
localpath

Lua scripts are not allowed to access the local file system, for security reasons. However, it's necessary that they sometimes reference objects that exist in the local file system, like the file containing the response body of a the result of a HTTP GET. In these cases, the local path to the file system is represented by localpath, which is an opaque object that can only be decoded by a small, secure set of native functions.
localpath instances are never created from Lua, and don't have any methods. They can only be returned from and passed to native functions.
Frontend bindings

local frontend = require 'frontend'

run(sourceCode): runs the script and returns its return value to the caller.
saveState(): returns a snapshot of the game state as a byte, which can be used later to restart the game from the point the snapshot was taken.
loadState(state): restart the game at the point the snapshot was taken. Snapshots are game and core dependent, loading a snapshot from a game and/or system into another game and/or system may cause the frontend to crash.
patch(localPath, patchData): patches a content at localPath using the IPS, BPS, UPS, or PPF patch at path. Returns true on success, or nil plus an error message on error. Patches are not permanent; all modifications to a content are lost when the content is unloaded.

Async bindings

local async = require 'async'
Waitable objects

Asynchronous work will always return a waitable object when scheduled. This object can be used to wait for the completion of the associated work, and contains information about its completion.
To wait for the completion of work, use the wait() method of the associated waitable. wait() is a blocking call, and will put the script to sleep until the waitable finishes. wait() returns two results, the result of the asynchronous operation if it succeeds, or nil plus a string describing the error. It's legal to use assert(waitable:wait()) to synchronize on a waitable object, while checking for its success and returning its result at the same time.
To test whether a waitable has finished or not without blocking, use finished() method, which returns true if it has, and false otherwise. Waitable objects also have a result() method which is identical to wait(), use one or the other according to the semantics of your code.
Limits on the number of concurrent asynchronous work being performed may be limited depending on the system. Do not assume the system will always perform all asychronous work in parallel.
Async functions


waitAll(set, onFinished): waits for the completion of all waitable objects in the list, successful or not. For each waitable that is completed, the onFinished function will be called, if present, with the waitable object, and its value in the set.
waitAny(set): wait for the completion of any one of the waitable objects in the list, and return the one that was completed along with its value in the set.

On both functions the waitable object is removed from the set.
The mod can sleep for some time via the pause function:

pause(ms): returns a waitable that will only finish after ms milliseconds have passed.

Networking

local http = require 'http'

get(url, dicardResponse): starts downloading asynchronously via HTTP and returns an waitable object. When the download completes, the waitable object will have a status field in the result property with the HTTP response code, and a localPath field which contains the local path to the contents of the download if it was successful. If discardResponse is true, then the HTTP response will not be saved, and localPath will be nil.

File system

local filesystem = require 'filesystem'
File system objects cannot be created from Lua. They are returned by some native methods to let Lua scripts interact with some files and folders in the file system.
filesystem objects have the following methods.

read(path, offset, count): starts reading the contents of the entry path in the file system. The result of the waitable object if of the bytes type. offset defaults to 0, and be used to specify the position of the file to start reading. count defaults to make fileRead read all the entry content beginning at offset.
write(path, contents): starts writing contents, which can be a bytes user data or a string, into the file at path.

Hash

local hash = require 'hash'
Hashes can be computed by creating a hash object, and feeding strings or byte objects to its update method. The computed hash is returned by the final method as a byte instance. To return the hash as a hexadecimal number in a string, use the tostring method. Note that both final and tostring finishes the hash object, so it cannot be further used.

create(type): this static method creates a new hash object of the given type. The type must be a string with the name of a supported hash.
update(content): updates the hash object with a string or bytes object.
final(): returns the hash as a byte object.
tostring(): returns the hash as a hexadecimal number in a string.

The following hashes are supported: MD5, SHA-1, SHA-256, SHA-384, SHA-512, CRC-32, CRC-64, and Adler-32.
JSON

local json = require 'json'

unserialize(json): translates the JSON in the given string to a Lua table. Returns nil plus an error message in case of failure.
serialize(data): translates the Lua table in data to a string containing the equivalent JSON and returns it. Returns nil plus an error message in case of error, i.e. circular references.

Since Lua can't differentiate between a key with the nil value in a table, and a key which doens't exist in a table, json.null is used as the value of keys which are null in the JSON input. It should also be used to set keys in tables that should be null when serializing back to JSON.
Timer

local timer = require 'timer'
Mods can create timers to control the passing of time. Their precision is milliseconds, and they can count upwards or downwards.

create(): creates a timer that counts upwards.
create(ms): creates a timer with the time set to ms milliseconds, that counts downwards.

Timers have these methods:

start(): starts the counting.
stop(): stops the counting. Counting can be started again with a call to start.
millis(): returns the current millisecond.
second(): return the current second.
minute(): return the current minute.
hour(): return the current hour.
onTime(ms, func): executes func, passing the timer as its only argument, when the time reaches ms milliseconds. Note that this is not the amount of milliseconds passed, but an actual point in time inside the timer's range, i.e. onTime(0, func) will trigger when timer that counts down reaches 0. If the point in time has already passed, onTime will not trigger. Each call to onTime overrides the last one.

Links


TASVideeos Lua API