Myst III HD modding guide

myst3-hd-modding.md

Introduction

ResidualVM is gaining the ability to create mods for Myst III. It is possible to replace the original game assets with hi-resolution versions. This combined with the emergence of machine learning image upscaling tools enables the creation of a hi-resolution mod for Myst III.

I am releasing here a version of ResidualVM usable to create and play such a mod. However, graphics processing is not my forte. I need help from the community to figure out how to upscale the original assets to produce the best visuals and to put together a mod.

A HD mod has already been successfully completed for The Longuest Journey, another game supported by ResidualVM: https://tljhd.github.io/. Hopefully the the same feat can be replicated with the same quality for Myst III.

Example mod

I am providing here a proof of concept mod I've used to test the programming changes. This example mod is not meant to be pretty, nor to be used by regular players. It is not representative of the visual quality the final mod might have. It contains an upscaled version of most of the exterior of the J'nanin age and the English journals. It is meant to be used with the DVD version of the game.

Screenshots:

• http://www.framecompare.com/image-compare/screenshotcomparison/7YWW7NNX
• http://www.framecompare.com/image-compare/screenshotcomparison/9100JNNU#

Downloads:

• ResidualVM HD v5 (built from this branch https://github.com/bgK/residualvm/tree/stable):
  • Windows (i686): residualvm-myst3-hd-mod-windows-v6.zip
  • macOS (x86_64 + arm64): residualvm-myst3-hd-mod-macos-v6.zip
  • Linux (x86_64): residualvm-myst3-hd-mod-linux-v6.tar.xz
• Example mod: https://drive.google.com/uc?export=download&id=1RcxCRz8isJF-Lh3HVTlKzKxwX-64Xl_2

To try the mod, set ResidualVM up to play Myst III (https://github.com/residualvm/residualvm#4-running-myst-iii). Then uncompress the mod archive in the game folder, so the mods directory is at the same level as the Data and M3Data directories. Then run the game using the ResidualVM version from this post.

Modding guide

In this section I will succinctly explain how to create a simple mod. You can contact me for more details (by private message here or on the #myst IRC channel on libera.chat).

The first step is to configure ResidualVM for modding. Open the residualvm.ini file and perform the following changes (https://github.com/residualvm/residualvm#61-location-of-configuration-file):

• Add enable_external_assets=true in your existing myst3 target section. This will cause ResidualVM to prefer loading files outside of the game archives.
• Add debugflags=modding in the [residualvm] section. This will make ResidualVM to print the names of the external files it tries to open.

Next, extract the game archives to access the assets. To do so, open Myst III in ResidualVM, open the debug console using the ctrl-d keyboard shortcut and enter the dumpArchive LEISnodes.m3a command. This will create a dump folder containing the assets for the J'nanin exterior in the ResidualVM folder. There are a lot of files. You can use the log output of ResidualVM to match the files to their in-game use.

ResidualVM will now load the files from the dump directory instead of from the game archives if it can find them. It will look for files with the following extension in that order: .dds, .png, .jpg. This means it's possible to experiment very quickly. Modify the extracted files, go to the corresponding location in game and observe the changes.

Next the modded assets need to be prepared for packing. PNG files are fine to experiment with but are not suitable for distribution as they are bulky and slow to load. ResidualVM expects the images to be in DDS format to prepare the mod for distribution. It accepts uncompressed, BC1, BC2, BC3 and BC7 compressed DDS files.

For the example mod, I've used the following process:

• JPEG artifact removal using https://github.com/ilyakurdyukov/jpeg-quantsmooth
• Upscaling using ESRGAN with the 4xBox model (https://drive.google.com/file/d/1KToK9mOz05wgxeMaWj9XFLOE4cnvo40D/view?usp=sharing)
• Downscaling to 2048px using ImageMagick so the files are not too large
• DDS BC1 conversion using https://github.com/richgel999/bc7enc

Next the prepared assets can be packed into a mod archive. Open Myst III in ResidualVM and enter the modArchive LEISnodes.m3a command. This will pack the modified files from the dump directory into a LEISnodes.m3a.patch file created in the ResidualVM folder. This command takes a long time to run as it compresses some of the files. You can disable the compression to experiment quickly using the modArchive LEISnodes.m3a false command.

Finally, the mod is ready to be tested and distributed. Place the mod archive in a directory corresponding to the mod name in the mods directory in the game directory. Set enable_external_assets to false in residualvm.ini so the files are loaded from the mod archive instead of the dump folder.

The same process can be used to mod the following asset types:

• Bink videos (The example mod replaces two of the Squee videos)
• Cursors (Bitmap files in RSRC.m3r)
• Inventory and font used in the load and save menus (.data files in RSRC.m3r, open as raw with The Gimp)

Note that there are some limitations:

• The mod archives are limited to 4 GB. Generate multiple archives to work around the limitation.
• Some of the videos will not display at the correct scale when they are loaded from the dump directory. They should play correctly once packed.
• I have probably left behind a number of bugs :) Report them to me so I can fix them.

Hey @bgK, thanks for your detailed guide! I would love to try and build a HD mod for Myst III. Because the ResidualVM project has been archived and Myst III is supported by ScummVM now, I tried to adapt your guide to ScummVM, but I ran into a few issues.

Here's what I got so far: I can open the debug console (with ctrl+alt+d) and export textures as jpg with dumpArchive. I cannot make ScummVM load the exported textures by setting enable_external_assets=true in scummvm.ini and the modArchive command is not available in ScummVM. Is that a known limitation of ScummVM or am I just missing something? Would it be better to write the mode with ResidualVM, even though the project won't be updated anymore?

Hi, at the moment, only the ResidualVM build linked from this post supports modding.

ok, I see. Thanks for updating the links in this document, I tried to build your fork on my machine, but I failed (not a C++ dev 😅), then I tried to set it up with GitHub Actions but I got stuck with an issue where the compiler fails to find zlib, so I'm glad that I can use your binaries. Now I can load custom textures from the dump folder.

Hey there @bgK! I'm coming back to this after a little while. I was a little discouraged with my efforts a few years ago to upscale the pre-rendered scenes due to nasty seams appearing where the six upscaled images making up the panorama met. However, I now have renewed interest since discovering MYSTERAssetExplorer, which allows me to extract the entire panorama as one image (so that when it's upscaled, the seams don't appear). I'll let you know how my playing around gets on once I have some nodes done :)

I'm currently trying to discuss the ability to upscale these scenes in the way I'd hoped with the maker of this YouTube video. I do not know whether (A) he has simply upscaled the spherical projections that MYSTERAssetExplorer can produce, or (B) actually done some manipulations on the cubic projections that avoids the seams.

If it's the former, then things will not be so simple, and I may write a little program which takes the six cube faces which either your extractor or MYSTERAssetExplorer produces, tile them in an extended manner such that the six primary pieces that need upscaling have copies of their required neighbours adjacent to them (to help match pixels and avoid seams), upscale, and then cut out those six pieces.

@bgK Hang on... your example upscales don't have any seams... how did you manage that? Is it something do to with the 4xBox model?

Hi, I am testing your build on linux and everything is upscaled. RealSR-ncnn-vulkan for images and Realesrgan-ncnn-vulkan for videos.
Fullscreen videos are converted to 30fps with ifrnet-ncnn-vulkan and overlays with alphamasks are 15fps because looping points.

Everything works really fine but there is couple minor issues. When Edanna ends and fullscreen video changes to overlay video
in the bird's nest, residual engine crashes. When loading that node from console it works and can exit from level.

Also in Narayan where you have to enter symbols to two panels with circular areas including four circles each all lefmoust circles do not
lit when clicking but white glow appears when puzzle is solved.

Removing mods doesn't help but scummvm works fine with same content.

In any case, the results are so good that I'm posting a video to youtube.

Hi, @MsSepeteus,

Thank you for the bug reports, I have updated the gist with new builds of ResidualVM that should fix the issues you noticed.

I look forward to seeing your work on the upscale. Please send me a link to the video!

Do you plan on publishing your mod on a platform such as moddb.com?

Hi, @bgK,

Here is a link: https://www.youtube.com/watch?v=eYesCjwj0Iw

There is a little quality drop after google's reencoding, but it's tolerable.

Maby I publishing it for testing. Fullscreen videos, overlay videos and images are in separate contents if editing is needed.

This is so impressive, thanks for sharing @MsSepeteus! It's a pity that the original overlay videos are of such low quality, because there the imperfections are notiicable if the characters are far away, but the landscapes are just gorgeous. Far superiour to what I have been experimenting with 2 years ago. If you decide to publish it, I would love to give it a try!

Hi, @klamann
I can upload it to Google drive, but it's 21GiB so it takes time.
I'll put the link here when it's ready.

Indeed, this looks very good! Awesome work @MsSepeteus. I'm eager to try it.
Let me know if I can be of assistance with more fixes or improvements to the engine.

Fantastic work @MsSepeteus! And nice work on the quick patch @bgK - awesome to suddenly see this sort of progress :)
I would love to ask though... how did you avoid the seams at the edges of the pre-rendered cubes made up of the six images for each node? I was using MYSTERAssetExplorer to first produce a single-image panorama, but I was wondering if there was a better approach.

hi, @bgK

You mentioned that you can fiddle the engine a bit more if needed, so if possible proper 30fps support would be nice.
And even better support for standart video formats would be nice also. Now that bink part is a most annoying phase in the whole thing.

Thank you for your great work with the engine !

it works! Thank you so much for sharing @MsSepeteus

hi, @bgK

I got the whole game tested with mods and didn't notice any issues. I woul'd say it's fully playable with mods.
Playlist: https://www.youtube.com/playlist?list=PL_VkzB24mavMDveb2hg8Iuyhrla6brOyD

Link to the mod if anyone is interested:
https://drive.google.com/drive/folders/19IyRy1E42udUgCAFYSvxGM0l-MOkuC7g?usp=sharing

The mod contains two sets of images and two sets of videos, all in separate contents if editing is needed.

Voltaic images: RealESRGAN_x4plus by pytorch
Tomahna J'nanin Amateria Edanna Narayan images: RealSR-ncnn-vulkan DF2K_JPEG

Amazing stuff @MsSepeteus :)

Thanks, @tomdodd4598

Did you test the mod, did you notice any bugs ?

@MsSepeteus haven't noticed any so far!

@MsSepeteus tremendous work!! Just finished playing it from start to finish and noted no issues, excerpt for a little one: when you first enter J'nanin main tusk and go at the nodes where the pages of the notebook are, two of the three visors in the room are open, not closed. I checked on the o.v. and the visors are closed.
Also I'm working on an Italian version of it, but I haven't found a way to extract videos from the mods to swap the audio. Any idea?
Thanks!

Thanks, @Amarok87

I didn't notice the whole visors thing, even though it was in my video too.
However, I compared the jpg dump and dds content inside LEOS.nodes, and they were identical.

I don't know proper way how to extract videos from patch nodes, but just copy that node to data dir and dump it.
eg. TOHOnodes.m3a to TOHOnodes.m3a.bak and TOHOnodes.m3a.patch to TOHOnodes.m3a
Restore Data dir before playing.

Thanks @MsSepeteus, it worked!

I noticed another thing: in TOHOnodes there's a video file named TOHO-11002 which isn't present in your mod files, so the game shows the low res version of it.

Working now on the italian audio files. I was wondering if adding all the audio streams to the files will work with the localization. I'll let you know.

Hi, @Amarok87

Thank you for pointing out that missing video. Can you inform me if you find more?
I'll update that mod archive later.

The main thing is @bgK's excellent HD-build, which makes things easy.

May I ask, are you going to use RAD tools for multitrack audio or do you have better muxer for bink files?

Hi @MsSepeteus

unfortunately I didn't find any other muxer for Bink videos, so I'm going through RAD tools. Good thing is that there are only 20-something videos to modify.
Adding all of the other languages to the video files is pretty easy, but I'm having an issue putting them in the dump folder: ResidualVM seems to ignore these Bink files. I also swapped the english audio with another language without doing multiaudio files but the issue persists. Modded images works fine instead. Did you encountered the same problem?

Thanks

Hi @bgK

Could you please check sometime when you have time that visors thing. It's present only in nodes leos 6 and leos 7.

Thank you in advance.

Hi @Amarok87

According to the tutorial residual engine loads only images not .bik files from dump dir. I never tried load anything from dump dir.
So rename dump dir and create new one with modded nodes and run "modarchive ####nodes.m3a false".(false disables packer)
Residual engine complains if there is mixed nodes or name errors.
Maby it's better to rename mods dir and create new one for testing.

I got gfpgan working with pytorch rocm, so now realesrgan can run with --face_enhance.
gfpgan can restore faces if you can recognize whose face are in picture but most of footage is totally out of quality.
Anyway, gfpgan looks promising and I'll remake some videos and update mods archive later.

Finally I managed to merge all the localized audio to the upscaled videos, everything seems to work smoothly. I updated the related *.patch files both in bink_1080p and bink_overlays.
I also upscaled italian journal entries and the rolling credits.