⚔️ Salutations fellow legends! ⚔️
Wow! Time has passed quickly since our last blog post, however, much progress has been made! Today, we're going to go over the technical aspects of the core engine for Legend of Worlds and what has been accomplished so far. The core of the engine has been completed, and now we are at the point where we can focus on gameplay and world creation logic!
Legend of Worlds is a cross-platform, cross-play, 2D online sandbox multiplayer experience where you can join, play, create and share player created worlds. This blog post covers the open-source game engine I've created in order to build this game.
In the last post, we discussed the benefits of Rust and WebAssembly in our game engine architecture. Rust gave us many benefits, but one issue it gave us was long compile times, depending on the more code and libraries we jammed into one module. This was getting to be a problem, as a Rust project starts off on my machine as a few seconds to compile, and can quickly grow to a few minutes! This happens especially when you include multimedia or graphics libraries, such as SDL or WGPU. This is not a problem for gameplay scripts because those will be hot-reloaded, but changes to the core engine are a pain to make due to dealing with long recompile times to see the changes.
The solution is ideally polyglot (multiple programming languages other than Rust), even though the core engine will always be in Rust, we can quickly prototype changes to the engine from other languages, such as TypeScript! Then we can easily port the new engine code to performant Rust right after.
So the solution was to break everything up into WebAssembly modules, and optionally dynamically linked libraries on native targets. What we mean by "everything" is the entity component system, rendering, input, audio, gameplay logic, user scripts, etc. However, this comes with one problem, how do we make sure these modules are all communicating with each other and working together?
Unfortunately, there is no official solution for this yet which is ready. The official solution would be the WebAssembly Component Model, but even if it and Rust implementations of it such as wit-bindgen were ready (which they aren't as of time of writing), it won't really work for game dev use cases. This is because of performance issues with the "shared-nothing dynamic linking" model that the WASM Component Model proposes.
This is due to performance issues with serializing data (roughly, converting it into binary code and then converting it back to a memory format understood by the language), so I had to create my own solution.
My first attempt at a solution looked something more like the WASM Component Model, simply because I thought it might be the only viable way to make the engine compatible with browsers at the time, and the WASM Component Model wasn't ready.
I tried to give each module an entity component system so that when it queried the main ECS module which was the source of truth, it didn't have to wait for the new serialized data, such as a renderer if an entity position hadn't changed, sending a query to another WASM module, getting the data back, and deserializing the data on every frame. This, of course, makes things absurdly convoluted to manage and is not very performant. One way I tried to get around performance, was to use multiplayer netcode synchronization techniques such as lag compensation, entity interpolation, and reconciliation, except all on the client-side as well rather than just the client and the server.
That way, even if communicating between the renderer and gameplay modules had a large amount of latency, techniques would be used to smooth it out and make it unnoticeable to the end-user. However, this made the architecture even more complex in a way that was becoming difficult to manage.
But I found another way...
Through use of memory access patterns such as using pointer + offset data to grab individual primitive values like numbers from the struct fields of components, I was able to get rid of serialization and map/access the data directly. So now we can share memory directly between WASM components, and directly map access to the data values rather than deserializing a set of values every time.
Not only this, we can either do this from Rust or any language that compiles to WebAssembly. This means zero overhead scripting from basically any language!
The solution is open source and can be seen here (as I've been accepted into the Flecs hub organization): https://github.com/flecs-hub/flecs-polyglot
The full open-source game engine will be hosted here when it's cleaned up and ready for public consumption/contribution: https://github.com/toxoidengine/toxoid
So what entity component system should we use for our engine? As you may have noticed from the libraries and flow charts, we're using one with a name.
Introducing Flecs, "a fast and lightweight Entity Component System that lets you build games and simulations with millions of entities". Not only that, but it has support for reflection and metadata which is good for scripting. Not that we even needed it, because we achieved maximum performance through another method anyways, with no reflection overhead!
Flecs is written in C, which makes it maximally portable to any platform/target. While Rust also has very good cross-platform support, Rust entity component systems also tend to use specific features (TypeId, compile time features such as macros and generics, etc.) that make scripting impossible. This C library is very flexible and can be used from Rust, and now thanks to my contributions, potentially from any other language with a Rust core and with hot-reload! Flecs is already sitting at almost 4k stars on GitHub. Shout out to the Flecs creator and any of the members of the community who have been supportive of my efforts, especially those who have also helped me develop the polyglot library.
More updates on gameplay, rendering, collaborative world editor, etc. soon.
Stay tuned, legends!
- Rou