During my time working with mods in Caves of Qud, I noticed a number of places where it could be improved, especially in the Script Mod side of things. This document intends to aggregate these improvements and their justifications for quick reference. These don't necessarily lay out any complete plan for a revamp to the modding system, but the ideas may serve as a good resource for creating such a plan.
Each proposal below aims to improve a problem presented under its heading, but they are not usually a mutually exclusive "choice". It is often possible to implement more than one of the proposals under a heading and get different kinds of benefits.
The concept of having the game compile mod sources from source C# scripts provided by the mod is an interesting one. It makes it very easy to inspect the mod's code before approving it for use. However, the C# Compiler Service that comes with the game is rife with bugs. Here are a few that I've found:
Generic methods are completely broken. They seem to always fail with an InvalidProgramException: Method body is empty error, even with very simple methods.
Take this snippet as an example, which successfully compiles but throws the aforementioned exception if called.
/// <summary>
/// Inserts an element in-between each element of this `source`.
/// </summary>
public static IEnumerable<T> Interleave<T>(this IEnumerable<T> source, T inBetween) {
using (var enumerator = source.GetEnumerator()) {
// If the enumerable was empty, emit nothing.
if (!enumerator.MoveNext()) yield break;
// Yield the first element.
yield return enumerator.Current;
// Now we begin the interleaving.
while (enumerator.MoveNext()) {
yield return inBetween;
yield return enumerator.Current;
}
}
}By extension generic classes also seem to have problems. Exactly in what way they have issues seems to be difficult to pin down, but in my experimenting, I would get strange exceptions thrown, like a System.NullReferenceException at System.String.memset. This was an internal call, not being directly invoked by the code I had written; the exception's call stack didn't even include a method from my code. Perhaps from compiler-generated code?
I've also managed to completely crash the compiler by nesting an enum in a static class. It just completely kills the game with an exception raised from the compiler's native code.
The other big compiler bug I found, which is less problematic, is that it outputs bad IL code for exception filters, a feature introduced in C# 6: catch (Exception ex) when (ex.InnerException != null) { }. This is easy to work around, but it still demonstrates how full-of-holes the current C# compiler is.
Needless to say, catering to this special-needs compiler has been frustrating and these are just a few of the problems I stepped on myself; there are almost certainly more. Having generics, an old and well-understood feature of C# since its second-edition, being so poorly supported suggests to me that this compiler is just way too shoddy to safely continue using.
There are a few Unity Store assets that utilize the Roslyn compiler to perform runtime compilation of C# scripts instead of Mono's Compiler Service, like this one for $20.
The Roslyn compiler has taken over as the primary compiler for Unity and is the reference compiler for all of C# now. It's capable of supporting newer features provided by C# 7 and even experimental features being trialed for C# 8 (though I wouldn't recommend enabling experimental features for runtime compilation; I'm just sayin' that it's the primary focus of current C# development and maintenance). This would provide modders with the most current and best tools to do the job.
From my experience, Mono is capable of executing the IL code that it emits just fine. As C#'s reference compiler, you can be more assured that the compiler is doing the correct thing when it compiles code.
Mono itself has moved toward Roslyn as well with their csc compiler, so perhaps there is a clean migration path between the two libraries that would not require much work to support. Depends on how much the underlying API changed, I suppose.
It could just be that the library the code compiler relies on is just hilariously out-of-date... And there may be some truth to this considering the assembly's copyright field says "2001 - 2009 Novell, Inc." I don't believe this assembly actually dates back to 2009 because it seems to have C# 6 support, which came out in 2012, but it raises an eyebrow.
It's possible all these issues are fixed in newer versions. If there is a drop-in update that fixes things, it would probably be a lot less painful than adapting the ModManager to use a totally different asset.
The solution that many other games have gone with is to just permit mods to package an assembly DLL that the game will load for the mod. This essentially allows modders to build their mod using whatever toolsets and CIL-compatible languages they wish, removing the responsibility of code-compilation of the mod from the game.
While this does make it harder to inspect the code before approving the mod, this hasn't really led to many security concerns when it was utilized by other games, like RimWorld. It also makes developing large-scale mods a little nicer since a full IDE that organizes and compiles such a project can manage the project, its references, and the build process.
This would have a few additional challenges to managing the mods, since searching for types referenced in blueprints would need to be a little more complex, but I don't think it would be too rough of a change, especially if some of the suggestions in the "Isolating Types between Mods" section are utilized.
A mod's config.json can configure an assembly: string property, which indicates the relative path to the main assembly the mod provides and that the game should load. If the assembly property is defined in the mod's config.json, runtime compilation of C# files is then skipped for this mod altogether.
You could optionally also have a references: bool property that tells Caves of Qud to tell the Mono Runtime to search the mod directory for additional assemblies that the mod will require using the AppDomain.AssemblyResolve event before loading the mod's assembly.
One thing that I've found myself wanting is some way to tell the game to execute some code after the mod has been loaded, but before the game begins to initialize the rest of the game's data, like the blueprints.
This is the trick Harmony Injector uses, but it's a pretty dumb one... And by dumb, I mean like a dumb-fire missile: you don't know what is being initialized and have no way to control it.
private static void RunStaticConstructors()
{
foreach (var type in ModManager.modAssembly.GetTypes())
RuntimeHelpers.RunClassConstructor(type.TypeHandle);
}While running type constructors is usually alright to do, it is making assumptions about the mod's code: all static constructors contain mod initialization code. This can lead to some wasted time constructing types that have little purpose until the game is actually under way.
After the mod assembly has been loaded, a quick search for concrete classes that extend some abstract class, like Qud.ModEntry, can be performed. The ModManager can then instantiate that class and store the instance in ModManager somewhere. This class's constructor can then perform any pre-game initialization that it needs.
Additionally, this class can provide some nice services to mods like hooks that get called when a new game is started or a save file is loaded or saved. It could perhaps even provide an interface to an IGamestateSingleton that is specially designed for storing a mod's global state, if it has any; data which can be discarded if the mod is ever deactivated and the types that were serialized no longer exist. This would help prevent save-game corruption when the player simply decides to stop using a mod.
If programming a whole new class just for mods is too daunting but you still want modders to have control over what gets run as a mod's initializer, a special attribute that can be attached to a class or method could be provided instead.
After the game loads the mod assembly, it searches for classes and static methods that have this attribute applied and, if one is found, it is invoked:
namespace Qud
{
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class ModInitializerAttribute : Attribute { }
}
namespace XRL
{
using ModInitializer = Qud.ModInitializerAttribute;
public static class ModManager
{
/* Rest of `ModManager` here... */
private static void InitMods()
{
// Supports only simple, concrete classes.
// No nesting or generics allowed.
var nonGenericClasses = modAssembly.GetTypes()
.Where(t => t.IsClass && !t.IsAbstract && !t.IsNested)
.Where(t => !t.ContainsGenericParameters);
var classInitializers = nonGenericClasses
.Where(HasModInitializer);
foreach (var type in classInitializers)
RuntimeHelpers.RunClassConstructor(type.TypeHandle);
var methodInitializers = nonGenericClasses
.SelectMany(t => t.GetMethods(BindingFlags.Public | BindingFlags.Static))
.Where(HasModInitializer)
.Where(m => !m.ContainsGenericParameters)
.Where(m => m.GetParameters().Length == 0);
foreach (var method in methodInitializers)
method.Invoke(null, new object[0]);
}
private static bool HasModInitializer(MemberInfo m) =>
m.GetCustomAttributes(typeof(ModInitializer), false).Length > 0;
}
}And some example usages:
namespace TestMods
{
using Qud;
[ModInitializer]
public class ModOne
{
static ModOne()
{
// This static constructor is invoked due
// to the attribute on `ModOne`.
}
}
public class ModTwo
{
[ModInitializer]
public static void DoInit()
{
// This static method is invoked due to
// the attribute on `ModTwo.DoInit()`.
}
}
}It's a quite simple and quick-to-implement solution that makes no assumptions about the mod's code. Just need to add some feedback when the attribute is used incorrectly, and you'd be done!
One of the scary things that Caves of Qud currently does is that it can end up with multiple mod assemblies in memory. When a new mod assembly is compiled and replaces another, the original mod assembly is not unloaded, even though the reference to the assembly that the game was holding will have fallen out of scope of the program.
It is still there and, depending on how a mod is coded, it could still be doing things!
This has the potential to cause some scary undefined behavior. It's entirely possible that a mod from the old assembly may have registered delegates on events or spawned its own thread, meaning it could still be responding to and manipulating the game's state all while a second instance of the mod is trying to do the same.
Note: This issue only applies to the runtime-compiled mod assembly. If pre-compiled assemblies were loaded for modding instead, this isn't an issue; the Mono runtime will only load a specific assembly once, even if Assembly.LoadFrom is called multiple times to load its DLL file.
If Caves of Qud has already compiled the mod assembly once and it finds it needs to do it again during a single run of the application, the best thing to do is automatically restart the game and compile the mod assembly in a fresh process.
This is generally how most other games handle this problem and what is enforced in the Harmony Injector mod. This mod allows other mods to apply runtime patches to the game's code; you really, really only want to allow this patching to occur once, and a new mod assembly would attempt to patch the game again. So, to prevent this, a patch is applied to ModManager.BuildScriptMods early on to restart the game if it is called to compile another mod assembly.
Right now, mods introduce or override components for blueprints by creating a namespace in their code for the kind-of-component Qud is going to look in to find it, IE: XRL.World.Parts.Skills.* for components related to skills.
Obviously, the point of this is to keep things simple for Caves of Qud to find the correct type that a blueprint requires. Right now, it just looks for the name from the blueprint in the namespace that is designated for that the kind-of-component, first in the mod assembly, then in the main game assembly. A type from the mod assembly is favored over a type in the main assembly, effectively overriding it.
Unfortunately, it makes it impossible to extend pre-existing components using traditional inheritance.
namespace XRL.World.Parts
{
public class AmmoGrenade : XRL.World.Parts.AmmoGrenade
{
// Additional implementation would go here...
}
}The intention is this class extends the pre-existing AmmoGrenade class that came with Qud, but C# interprets this as the class just extending itself; a circular dependency. Unsurpringly, this does not compile.
Even using global::XRL.World.Parts.AmmoGrenade or a using alias to try and differentiate between the two identically named types cannot resolve this error. The new keyword is no hope here either, as it can't be applied to classes. You just can't use inheritance with this design.
The only thing that would work is to utilize extern aliasing, so that namespaces from differing assemblies can be distinguished. This kind of alias can only be established through compiler configuration, though. The CSharpCompiler.CodeCompiler class that Qud ships with does not support this feature, though it can be modified to do so without too much effort, assuming its source code can be modified.
However, leveraging extern aliases would also probably break many mods already in the wild.
This issue is usually worked-around in three ways, all with their own drawbacks:
- By changing the name of the specific component referenced by every blueprint. But, imagine how many edits that might end up being for some of the more common components... And this only works for blueprints that are known to the modder at the time they're developing it, mainly the blueprints in the base game. Any blueprints added by other mods that reference the original name of the component can't be adjusted to use the new component so easily.
- By disassembling the original component's class and copying the code for the entire class, essentially re-implementing it with whatever other changes the modder wished to make. This obviously gives up traditional inheritance and this kind of modification can only be done once by one mod. Not to mention, if the original implementation is changed via a game update, the mod will still be using the old implementation until it too is updated to reflect the changes, which may cause the game to break in undefined ways in the interim.
- By patching the component's methods with something like Harmony, however this doesn't allow the modder to add new members to the component.
Beyond this, using namespaces in this way can result in name collisions that cause the mod assembly to fail to compile if two different mods happen to use the same fully-qualified name for a component.
An attribute can be used to tell Caves of Qud what kind-of-component a component's class is supposed to belong to, meaning it no longer has to live in a specific namespace. Along with that, it can indicate a specific Type it is intended to override within the blueprints. Caves of Qud can use this information to properly map components the way the modder intended, including being able to target components from other mods and extend them too.
Here we setup an enumeration for the different kinds-of-components and the attribute:
namespace XRL
{
public enum ComponentKinds
{
// For override-only components...
Override,
// Then an entry for each kind-of-component...
Part, BaseMutation,
CustomChargen, Skill
// ...and the rest would follow.
}
public static class ComponentKindExtensions
{
// Something to easily map component-kinds to their
// original namespaces.
public static string AsNamespace(this ComponentKinds kind)
{
switch (kind)
{
case ComponentKinds.Part: return "XRL.World.Parts";
case ComponentKinds.BaseMutation: return "XRL.World.Parts.Mutation";
case ComponentKinds.CustomChargen: return "XRL.CharacterCreation";
case ComponentKinds.Skill: return "XRL.World.Parts.Skills";
// ...and the rest.
default: return null;
}
}
}
}
namespace Qud
{
using ComponentKinds = XRL.ComponentKinds;
[AttributeUsage(AttributeTargets.Class)]
public class ModComponentAttribute : Attribute
{
// Constructor for newly introduced components.
public ModComponentAttribute(ComponentKinds kind)
{ this.kind = kind; }
// Constructor for override-only components.
public ModComponentAttribute(Type overrides)
: this(ComponentKinds.Override)
{ this.overridden = overrides; }
public readonly ComponentKinds kind;
public readonly Type overridden;
}
}Then we need to modify XRL.ModManager.ResolveType to make use of these new additions:
namespace XRL
{
using ModComponent = Qud.ModComponentAttribute;
public static class ModManager
{
/* Rest of `ModManager` here... */
public static Type ResolveType(ComponentKinds kind, string typeID)
{
// Find a component that matches the request.
var type = EnumerateComponents(kind, typeID).FirstOrDefault();
if (type == null) return null;
// Then start mapping through the overrides.
var newType = default(Type);
while (TryResolveOverride(type, out newType)) type = newType;
return type;
}
private static IEnumerable<Type> EnumerateComponents(ComponentKinds kind, string typeID)
{
// Doing some pattern-matching with `goto` jumps...
// This all works much nicer in C# 7, but this is a
// decent approximation in C# 6.
var fullName = $"{kind.AsNamespace()}.{typeID}";
if (!bCompiled) goto caseQudAssembly;
if (modAssembly == null) goto caseQudAssembly;
var attr = default(ModComponent);
foreach (var type in modAssembly.GetTypes())
{
// Matching by name...
if (type.Name != typeID) goto caseSkip;
if (type.FullName == fullName) goto caseMatched;
// Matching by attribute...
if (!IsSimpleClass(type)) goto caseSkip;
if (!Match_ModComponent(type, out attr)) goto caseSkip;
if (attr.kind == kind) goto caseMatched;
caseSkip:
continue;
caseMatched:
yield return type;
}
caseQudAssembly:
var type = Type.GetType(fullName);
if (type != null) yield return type;
}
private static bool TryResolveOverride(Type curType, out Type newType)
{
var attr = default(ModComponent);
foreach (var t in modAssembly.GetTypes())
{
if (!IsSimpleClass(t)) continue;
if (!Match_ModComponent(t, out attr)) continue;
if (attr.overridden != curType) continue;
newType = attr.overridden;
return true;
}
newType = null;
return false;
}
private static bool IsSimpleClass(Type t)
{
if (!t.IsClass || t.IsAbstract || t.IsNested) return false;
if (t.ContainsGenericParameters) return false;
return true;
}
private static bool Match_ModComponent(Type t, out ModComponent attr)
{
attr = t.GetCustomAttributes(typeof(ModComponent), false)
.Cast<ModComponent>()
.FirstOrDefault();
return attr != null;
}
}
}And now, with all that in place, we can make use of the attributes in a mod:
namespace AmmoMod
{
using XRL;
using Qud;
// Introducing a new component.
// It would be referenced in blueprints with
// "AmmoIckslugCanister" as a `<part />`,
// of course.
[ModComponent(ComponentKinds.Part)]
public class AmmoIckslugCanister : XRL.World.IPart
{
// Icky things go here...
}
// Extending and overriding an existing component.
// All references to "AmmoGrenade" in blueprints
// are automatically redirected to this class.
[ModComponent(typeof(XRL.World.Parts.AmmoGrenade))]
public class MyCoolAmmoGrenade : XRL.World.Parts.AmmoGrenade
{
public override bool FireEvent(XRL.World.Event E)
{
// Additional coolness goes here...
return base.FireEvent(E);
}
public int coolnessFactor = 9001;
}
}The game goes through the following process when the blueprint parser comes across <part Name="AmmoGrenade" /> in the blueprints:
- The blueprint parser asks the
ModManagerto resolve aComponentKinds.Partcalled "AmmoGrenade". - The
ModManagerresolves the vanillaXRL.World.Parts.AmmoGrenadeclass as its first candidate, but then looks through the overrides. - It sees that
AmmoMod.MyCoolAmmoGrenadehas registered an override forXRL.World.Parts.AmmoGrenade. The current candidate is dropped andAmmoMod.MyCoolAmmoGrenadebecomes the new candidate. - Since there are no overrides registered for
AmmoMod.MyCoolAmmoGrenade, this type is the one that is returned.
With the components now living in their own namespaces, mods can easily find and extend types on other mods. For example, if someone else wanted to make my AmmoIckslugCanister even ickier...
namespace TerribleIckiness
{
using Qud;
// Mods can now mod mods!
// All references to "AmmoIckslugCanister" in
// the blueprints would now map to this class.
[ModComponent(typeof(AmmoMod.AmmoIckslugCanister))]
public class AmmoIckslugCanisterWithEvenMoreSlime : AmmoMod.AmmoIckslugCanister
{
// Pour in the sliiiiiime!
}
}And this modification would require no changes to the blueprints supplied by AmmoMod.
Obviously, the example implementation here would benefit a lot from adding features like caching, but this is the gist. It might be a bit of a refactor, a small chore to centralize all those namespace strings scattered across the code-base into the ComponentExtensions.AsNamespace extension method, but I think it'd be a big gain and reduce the difficulty of getting into Caves of Qud script mods quite a bit. They can just use what they know about C# inheritance and not have to learn the workarounds.
Plus, it's backwards compatible! No existing mods should break, unless they were calling the previous signature of XRL.ModManager.ResolveType, but you can always just make an overload with the original signature to preserve the current public API.
The only issue that I see with it is the case where two different mods provide overrides for the same component, but I think the default rule to follow would be to log a warning about the conflict in the debug log and then proceed by respecting the load-ordering of mods to resolve the conflict.
Here's just a bunch of smaller proposals that would make the whole modding experience more friendly. I'm sure some of these are already on the dev's todo list, but may as well list them out here anyways.
Currently, if the mods fail to compile, a player goes blissfully unaware that all their mods are disabled until they're in game and notice or they open the build log.
How about just a nice little popup...? Or maybe an angry, bouncing red icon on the main menu...? Anything that informs the player the mod assembly failed to build and to check the logs to find out why.
Just a convenience to both mod developers and mod users; easy access to the reason why the mods didn't compile and potentially the knowledge of which mod was to blame.
Depending on the user's config, script mods may require approval, but it takes some effort to actually find where they're installed to inspect them to see that they're safe. Adding a way to open the mod's folder would be helpful and make it easier to see that BitcoinMiner.cs is not included in a mod's folder.
If any one mod fails to build, all mods fail to build. This could be helped a bit by building each mod into its own assembly.
It's trivial to just loop through each ModInfo in ModManager.mods and run the C# compiler on each mod's source code individually; the Mono runtime doesn't much care if mods come in one assembly or twelve as long as they link properly. If one mod fails to compile, it will only break those mods that were reliant on the mod that failed.
After that, it's just fixing the type resolution methods in ModManager. A foreach loop over the mod assemblies is all that would take, but you'd probably want to do it in reverse order, in order to respect mod load-order. A mod loaded later should override a type from a mod loaded earlier.
And then the game would be right back to working as it does now, just with mods spread across many assemblies.
Caves of Qud recently added a new logging system in global::Logger which separates "build" logs from "game" logs, and spares us the (Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 48) noise that the UnityEngine.Debug logger has.
Great idea! But for whatever reason (and I really tried to understand why), the new logging system doesn't register calls into it from the mod assembly. Modders like to make use of debug logging while they're developing their mod, so giving modders the ability to use such a clean, noise-free logger would really be appreciated.
Popups are currently only designed to run from within the game-thread, but have no safe-guards checking for this, which means attempting to summon one from the UI-thread deadlocks the game as it waits for keyboard input that can't be detected due to deadlock.
And since the game-thread does not pump its ThreadTaskQueue while in the main menu, a task queued on it requesting the game-thread to display a popup goes ignored.
This is another feature that Harmony Injector was able to hack in for itself, with some effort (all of the classes that control the popups are marked internal), but it would be nice not to need a hack.