unify.md

GPM_VFS -- Virtualization of a Unified Filesystem

Unionfs is a filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems. It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system. Contents of directories which have the same path within the merged branches will be seen together in a single merged directory, within the new, virtual filesystem.

Vocabulary

• [W]rite - Any operation where data is written to disk.
• [R]ead - Any operation where data is read off of the disk.
• E[X]ecute - Any operation where a file is executed, loaded via LoadLibrary(), etc.
• VFS - Virtual File System
• VFSL - Virtual File System Layer
• Virtual Directory - A virtual directory "projected" upon the target process such that some directory X seems to contain files Y and Z, but are really fragmented across several different locations.

Introduction

To facilitate the state-oriented design specifications of GPM, a game-first virtual filesystem must be designed with immutability as a first-class citizen.

Virtual File System

The VFS itself is a simple user-mode hook, which redirects Win32 API calls into arbitrary code. With full access to the arguments and calling of the original function, it is possible to "jump" filesystem access around to locations of the client's choosing. For example, a request to open file dir/a.txt can be transparently redirected to some file dir2/dir3/b.txt. This "projected" directory structure is private -- only visible to the target game process.

Virtual Directory Layer

With just a VFS, we can easily redirect filesystem calls to arbitrary locations. However, it becomes difficult to maintain the immutable nature of GPM without a system to ensure state. Using concepts derived from UnionFS and Docker, it becomes possible to combine multiple directories together to form a single, projected merge of the two. For instance, directory dir1 and dir2 which each contain some number of files can be combined into dir3, which contains all children of dir1 and dir2.

VFSLs are used to ensure that the immutability of root-most files are maintained, while still remaining flexible enough to allow for file write operations. The two classes of layers -- immutable and mutable -- exchange files when necessary. The state of a layer is saved in a manner which allows for rollbacks easily at any point in the modding process. For instance, state s0 ran well, while s1 crashed the game; At any point the configuration can be set back to s0 with little downtime.

Runtime VFSL Linking

At runtime one or more VFSLs must be combined into a single projected virtual filesystem. To enable this, the GPM_VFS linker "trickles down" each layer, selecting files based on their layer load order.

 ----------------
| Root VFSL      |
 ----------------
 ----------------
| Package VFSL   |
 ----------------
 ----------------
| Overwrite VFSL |
 ----------------

For example, file dir1/test.txt within the Root VFSL has no collisions. At the end, this file will be projected onto the game as normal. However, a package is installed which replaces this file with package/otherFile.txt. The Root VFSL entry will be replaced as the linker steps down each layer. At the end, the file package/otherFile.txt will be project onto the game, not dir1/test.txt.

Package <package_name> adds file package_name/dir/config.ini with no collisions. This file, since it has no replacements, is added to the virtual projected directory.

Root VFSL

The root layer is the bottom-most structure in the virtual directory. Generally comprised of original game files, this layer is always read-only.

Package VFSL

Packages with the intent to modify, overwrite or delete game files are placed into this read-only layer. As with following layers, behavior is dictated by the copy-on-write rules described [here].

Overwrite VFSL

Files that have been modified are placed in one or more of these layers. However, only the newest layer can be active at any given time. Due to the inherent size associated with GPM packages, "archived" layers can be compressed until they are needed to rollback to a previous state.

Copy-On-Write Behavior

To ensure that files are handled in consistent and expected ways, a series of rules must be established to define the behavior of the COW filesystem.

Root Layer

The most important files reside within this layer, and as such, should remain immutable.

- W -> COW into OL.
- R -> R with VFS relay.
- X -> Rehook, adopt parent layers.

Examples:

• In many cases there are times where the game will modify it's own state. While this write is occuring from the parent, the COW rules listed above disallow this. For example, Skyrim will write save files to the user's home directory -- for this example, home/My Games/Skyrim/Saves/somesave.ess. This directory is not an inherent part of the VFS, but upon write we add it to the overwrites directory. Upon completion, the requested file home/My Games/Skyrim/Saves/somesave.ess will not exist, but to the game, it will.

Package Layer

Like the Root Layer, it must be assumed that these files are immutable and irreplaceable.

- W -> COW into OL.
- R -> R with VFS relay.
- X -> Rehook, adopt parent layers.

Examples:

• Game file game/a.txt is being replaced by package test's file test/replace.txt. Instead of manually overwriting files on disk, at runtime the gpm_vfs linker will replace all calls to dir/a.txt with test/replace.txt.
• If test/replace.txt is then written to, we must first copy it to the instance's overwrite layer and then allow for a file write at the new location.
• Say a package file runs a script which creates a file on disk at location game/plugin/file.ini. Following the W rule, this file write is redirected into the current overwrite layer. Once the write has completed, this new file can be atomically added into the virtual filesystem.

Overwrite Layers

File and directory manipulation events made upon an immutable layer will be copied (if valid) and written into one or more of these.

- W -> Copy and write into self.
- R -> Read with VFS relay.
- X -> Rehook, adopt parent layers.

Examples:

• Game file game/a.txt is modified by the parent process. Upon interception of the write call, we copy game/a.txt into the current instance's Overwrite Layer, creating a file <timestamp>/ol/game/a.txt. Due to the cascading nature of the virtual directory layer system, this new file is preferred to the old.
• Package with file package/file.ini is updated. The new package is linked to the profile, but the updated file itself is placed into an Overwrite Layer, effectively removing the old (without breaking immutability promises).