Skip to content

Instantly share code, notes, and snippets.

@PgBiel
Last active January 31, 2024 03:02
Show Gist options
  • Save PgBiel/ffa695a479ef4466cb24755db983950b to your computer and use it in GitHub Desktop.
Save PgBiel/ffa695a479ef4466cb24755db983950b to your computer and use it in GitHub Desktop.
Gdext WebAssembly Experiment #1

Everything below was tested with Godot 4.1.1, gdext at commit master@639aeb495f8f43fe68a326f6abe70a8ce4694ce3, Rust v1.72.0 (when using the stable channel), Rust v1.74.0-nightly (5ae769f06 2023-09-26, when using the nightly channel) and emsdk v3.1.28, in a Debian 12 podman container.

  1. Setup: So far, to reach the furthest progress I could find in gdext WebAssembly export, it seems you need at least to:

    1. Use the Emscripten flags --no-entry -sSIDE_MODULE=2 -sUSE_PTHREADS=1 (last one appears to be superseded by -pthread on recent versions);
      • To apply those flags, one can add to the env var RUSTFLAGS -C link-arg=emscripten-flag-here for each emscripten flag.
      • You can also avoid having to use RUSTFLAGS by adding the flags to project/rust/.cargo/config.toml in the format
      [target.wasm32-unknown-emscripten]
      rustflags = [
          "-Clink-arg=--no-entry",
          "-Clink-arg=-sSIDE_MODULE=2",
          "-Clink-arg=-sUSE_PTHREADS=1"
      ]
      • NOTE: To enable some debug symbols and checks, I had to add the Emscripten flags -O1 -g -sASSERTIONS=2 -sSAFE_HEAP=1 -sSTACK_OVERFLOW_CHECK=2 -sDEMANGLE_SUPPORT=1
      • UPDATE (2023-10-11): The flags -sSTACK_SIZE=2000MB -sINITIAL_MEMORY=3999MB, while also excluding -sASSERTIONS=2 -sSTACK_OVERFLOW_CHECK=2, may also be required to prevent certain stack overflow errors.
        • Additionally, using emscripten 3.1.28 is currently required if building Godot from source.
    2. Use the rustc flag target-feature=+atomics,+bulk-memory,+mutable-globals (required for emscripten to allow enabling shared memory)
      • Can be added in RUSTFLAGS as -C target-feature=+atomics,+bulk-memory,+mutable-globals
        • Or as a line in 'rustflags' in the project's local .cargo/config.toml file as shown above
      • Due to this, it appears that rebuilding the Rust standard library with these rustc features is necessary. For this, one will have to use nightly cargo and append the following to ~/.cargo/config.toml:
      rustflags = [
          "-C", "target-feature=+atomics,+bulk-memory,+mutable-globals",
      ]
      
      [unstable]
      build-std = ["panic_abort", "std"]
    3. Build with cargo +nightly build --target wasm32-unknown-emscripten (plus RUSTFLAGS set or whichever other modifiers are used).
    • Make sure to add emscripten target support with rustup target add wasm32-unknown-emscripten
    • NOTE: Don't forget to build normally with cargo build as well (if you run cargo clean) to make sure you can load the project in the Godot editor and perform the Web export.
    • UPDATE (2023-10-11): If you're using an emscripten version older than 3.1.40 and you get an error indicating some library can't be found, e.g. -lc-mt-debug, you may have to apply the changes to emcc.py from this emscripten 3.1.40 commit into your local /opt/emsdk/upstream/emscripten/emcc.py file: https://github.com/emscripten-core/emscripten/commit/b76fdf445f0a8dfdebfd0f67749bfbc20665d397
    • Additionally, you may have to patch gdext as follows, to ensure a particularly problematic function doesn't run on WASM:
      diff --git a/godot-ffi/src/compat/compat_4_1.rs b/godot-ffi/src/compat/compat_4_1.rs
      index e258e58..679a31a 100644
      --- a/godot-ffi/src/compat/compat_4_1.rs
      +++ b/godot-ffi/src/compat/compat_4_1.rs
      @@ -49,6 +49,10 @@ impl BindingCompat for sys::GDExtensionInterfaceGetProcAddress {
               // first fields have values version_major=4 and version_minor=0. This might be deep in UB territory, but the alternative is
               // to not be able to detect Godot 4.0.x at all, and run into UB anyway.
      
      +        if cfg!(target_arch = "wasm32") {
      +            return;
      +        }
      +
               let get_proc_address = self.expect("get_proc_address unexpectedly null");
               let data_ptr = get_proc_address as *const LegacyLayout; // crowbar it via `as` cast
    1. Add the following line to ExtensionName.gdextension:
    web.debug.wasm32 = "res://path/to/rust/target/wasm32-unknown-emscripten/debug/crate_name.wasm"
    
    1. Enable "Enable support for GDExtensions" in the Web Export dialog in the Godot editor.
    2. UPDATE (2023-10-11): Compile a web template from source with Godot v4.2-dev6 (commit 57a6813bb8bc2417ddef1058d422a91f0c9f753c) and select the resulting bin/godot.web.template_debug.wasm32.dlink.zip file on "Debug" under "Export template" when exporting to the Web.
      • To do this, first ensure you have the correct tools to compile Godot from source (see https://docs.godotengine.org/en/stable/contributing/development/compiling/compiling_for_web.html#doc-compiling-for-web).
      • Then, git clone https://github.com/godotengine/godot, cd godot and git checkout --detach 57a6813bb8bc2417ddef1058d422a91f0c9f753c to go to that commit.
      • Finally, run scons platform=web production=yes dlink_enabled=yes debug_symbols=yes target=template_debug to build the web export template.
      • It will generate the aforementioned zip which you may select while exporting to the web in your editor.
  2. Results: Godot exports successfully; however, this seems to be producing different errors on game startup (when loading the page), depending on the project setup.

  • In the Dodge the Creeps example from the gdext repo (commit master@639aeb495f8f43fe68a326f6abe70a8ce4694ce3), I'd get index.js:51071 Aborted(stack overflow (Attempt to set SP to 0x568490, with stack limits [0x684d0 - 0x9daf60])) (without enabling debug it's just "memory access out of bounds")
  • In a simple Hello Gdext example, created manually by following the gdext docs, I'd get WebAssembly.instantiate(): Compiling function #700:"godot_ffi::gen::table_scene_classes::ClassScene..." failed: local count too large @+416567 (or, without debug flags, just wasm validation error: at offset 416568: too many locals) - which seems to have been the latest progress reported in the thread regarding Webassembly in the gdext Discord server.
  • Regarding this difference, I found out that adding default-features = false to the gdext dependency on Cargo.toml would result in the Stack overflow error, while enabling it back would result in the 'local count' error. So this seems to be the cause in the difference between the two projects (the Dodge the Creeps project disables default-features).
  • NOTE: On Firefox, I'd often get the (unhelpful, without any form of stack trace) error WebAssembly module validated with warning: failed to allocate executable memory for module instead, but this seemed to be a one-time thing (after a computer restart, this doesn't seem to be happening anymore, and the errors match Chromium's as the two listed above - although that's weird as I had plenty of memory available (over 40 GB) in my computer).

UPDATE 1 (2023-09-30): The Stack overflow error also appears with default features enabled when compiling with --release (cargo +nightly build --release --target wasm32-unknown-emscripten) and changing the necessary file paths in .gdextension. That is, --release seems to make the 'local count too large' error disappear. (See also yewstack/yew#478)

UPDATE 2 (2023-10-01): The error changes to "index out of bounds" on Firefox when using -O0 with -sSTACK_SIZE=50MB and 4 GB of initial memory, in some internal emscripten WASM function named "cull_zombies" (image here).

UPDATE 3 (2023-10-11): Now using Godot 4.2-dev6 (57a6813bb8bc2417ddef1058d422a91f0c9f753c) compiled from source (see new instructions above), with Emscripten 3.1.28 (the only version which works at that commit). As a result, got simply a stack overflow at first, which was solved by adding the flags -sSTACK_SIZE=2000MB -sINITIAL_MEMORY=3999MB -sALLOW_MEMORY_GROWTH=1 and removing assertions and stack overflow checks. Then, hit the error native code called abort() - gdext was panicking at ensure_static_runtime_compatibility under godot-ffi/src/compat/compat_4_1.rs (apparently due to UB). Had thus to apply a patch to return early at that function (see https://github.com/PgBiel/gdext/commit/77932bd0996be9531c754d581379ed46ebf1fd68). With that, I got the error Assertion failed: bad function pointer type - dynCall function not found for sig 'jji'. Adding -sEMULATE_FUNCTION_POINTER_CASTS=1 changed the error to Cannot convert 491636 to BigInt. Using -sWASM_BIGINT=1 did not change the error.

UPDATE 4 (2023-10-14): Attempting the same procedure from Update 3, however using Godot 4.2-beta1 (b1371806ad3907c009458ea939bd4b810f9deb21) compiled from source with Emscripten 3.1.39 (the latest version which works at that commit), and disabling debug assertions and stack overflow checks, the game seems to simply freeze instead of throwing the bad function pointer type error. However, re-enabling stack overflow checks does still result in a stack overflow error.

Below are the full trial and error steps I took, in a quite verbose manner. I summarized my results in gdext-wasm-min-report.md above.

This was done in a Debian 12 podman container.

  1. Installed Godot 4.1.1 by downloading the binary from the Godot website
  2. Cloned dodge-the-creeps example from gdext (master@639aeb495f8f43fe68a326f6abe70a8ce4694ce3) (Godot failed to open it though due to invalid relative paths)
  3. Modified 'DodgeTheCreeps.gdextension' paths to point to res://../rust/
  4. Modified rust/Cargo.toml to point to git dependency (to the gdext repository) instead of path dependency
  5. Set renderer to "Compatibility" (worked)
  6. cd rust && cargo build
    • Rust 1.72.0 (stable toolchain on rustup)
  7. Exported it to Linux/desktop - worked
  8. Tried to export to web - failed (no web declared in .gdextension file)
    • (Enabled support for GDExtensions)
  9. Tried to cargo build --target wasm32-wasi or cargo build --target wasm32-unknown-emscripten - failed
    • Here had emscripten installed through apt install emscripten
  10. rustup target add wasm32-wasi and rustup target add wasm32-unknown-emscripten - OK
  11. wasi target compiled normally
  12. emscripten target failed: missing main symbol
  13. Compiling with RUSTFLAGS="-C link-args=--no-entry" cargo build --target wasm32-unknown-emscripten worked
  14. Added line web.debug.wasm32 = "res://../rust/target/wasm32-unknown-emscripten/debug/dodge_the_creeps.wasm" to DodgeTheCreeps.gdextension
  15. Web export succeeded
  16. Tried to open index.html from Firefox and Chromium - failed (CORS header problem)
  17. Downloaded serve.py from godot repo at platform/web/serve.py (https://github.com/godotengine/godot/blob/master/platform/web/serve.py)
  18. Ran python3 /path/to/serve.py -r $(realpath .) from within the folder with web export files
  19. Opened localhost:8060 in Firefox and enabled JavaScript
  20. Tried to load game - failed: need the dylink section to be first
    • Likely emscripten version incompability (3.1.6 is installed, Godot uses newer versions)
  21. Switched to emsdk version 3.1.14 to test
  22. Rebuilt with new emsdk version (cd rust && cargo clean && RUSTFLAGS="..." cargo build ... as before)
  23. Re-exported to web
  24. Ran python server again
  25. Tried to load game again on Firefox - failed: "WebAssembly module validated with warning: failed to allocate executable memory for module" "uncaught exception: out of memory"
  26. Switched to emsdk version 3.1.45 (seems to be the correct version for Godot 4.1) by repeating steps 20-24 (but with 3.1.45)
  27. Tried to load game again on Firefox - failed with the same out of memory error
    • On Ungoogled Chromium: "need the dylink section to be first"
  28. Rebuilt with cargo clean && RUSTFLAGS="-C link-arg=--no-entry -C link-arg=-s -C link-arg=SIDE_MODULE=2" cargo build --target wasm32-unknown-emscripten
    • Adding '-s SIDE_MODULE=2' to emscripten options as per results from a search
  29. Re-exported to web etc.
  30. Tried to load game again on Firefox - failed with the same out of memory error
  31. Tried to load game again on Ungoogled Chromium - new error: "WebAssembly.instantiate(): Import #71 module="env" function="memory": mismatch in shared state of memory, declared = 0, imported = 1"
  32. Recompiling with RUSTFLAGS="-C link-arg=--no-entry -C link-arg=-sSIDE_MODULE=2 -C link-arg=-sUSE_PTHREADS=1" cargo build --target wasm32-unknown-emscripten
    • USE_PHTREADS=1 enables shared memory
    • Compilation failed: wasm-ld: error: --shared-memory is disallowed by /.../dodge-the-creeps/rust/target/wasm32-unknown-emscripten/debug/deps/dodge_the_creeps.(...).rcgu.o because it was not compiled with 'atomics' or 'bulk-memory' features.
      • Apparently this means I need to recompile the Rust standard library
  33. Installed nightly Rust toolchain to enable unstable features with rustup toolchain install nightly
    • Version 1.74.0-nightly (5ae769f06 2023-09-26)
  34. Wrote to ~/.cargo/config.toml:
rustflags = [
    "-C", "target-feature=+atomics,+bulk-memory,+mutable-globals",
]

[unstable]
build-std = ["panic_abort", "std"]
  1. Tried to recompile as in 31., failed with the same error
    • Apparently the program itself needs the extra flags
  2. Recompiled with RUSTFLAGS="-C link-arg=--no-entry -C link-arg=-sSIDE_MODULE=2 -C link-arg=-sUSE_PTHREADS=1 -C target-feature=+atomics,+bulk-memory,+mutable-globals" cargo +nightly build --target wasm32-unknown-emscripten
    • Successful
  3. Tried to load game again on Firefox - failed with out of memory error
  4. Tried to load game again on Ungoogled Chromium - new error: "memory access out of bounds"
  5. Tried to recompile adding -O3 (-C link-arg=-O3) to 35., no luck (same errors in Firefox and UC)
  6. Tried to recompile adding -C link-arg=-sINITIAL_MEMORY=4294967296 -C link-arg=-sMAXIMUM_MEMORY=4294967296 -C link-arg=-sALLOW_MEMORY_GROWTH=1 to 38., no luck
  7. Tried to recompile adding -C link-arg=-sEMULATE_FUNCTION_POINTER_CASTS=1 to 39., no luck
  8. Recompiled with debug symbols, using RUSTFLAGS="-C link-arg=-O1 -C link-arg=-sASSERTIONS=2 -C link-arg=-sSAFE_HEAP=1 -C link-arg=-sSTACK_OVERFLOW_CHECK=2 -C link-arg=-sDEMANGLE_SUPPORT=1 -C link-arg=-g -C link-arg=--no-entry -C link-arg=-sINITIAL_MEMORY=4294967296 -C link-arg=-sMAXIMUM_MEMORY=4294967296 -C link-arg=-sALLOW_MEMORY_GROWTH=1 -C link-arg=-sEMULATE_FUNCTION_POINTER_CASTS=1 -C link-arg=-sSIDE_MODULE=2 -C link-arg=-sUSE_PTHREADS=1 -C target-feature=+atomics,+bulk-memory,+mutable-globals" cargo +nightly build --target wasm32-unknown-emscripten
  9. Chromium error changed to stack overflow:
index.js:51071 Aborted(stack overflow (Attempt to set SP to 0x568490, with stack limits [0x684d0 - 0x9daf60]))
onPrintError	@	index.js:51071
abort	@	index.js:796
___handle_stack_overflow	@	index.js:19144
stubs.<computed>	@	index.js:4606
$em_task_queue_create	@	dodge_the_creeps.wasm-066c0442:0x22347c
$_emscripten_thread_mailbox_init	@	dodge_the_creeps.wasm-066c0442:0x2252b0
$__emscripten_init_main_thread	@	dodge_the_creeps.wasm-066c0442:0x2233b7
$__wasm_call_ctors	@	dodge_the_creeps.wasm-066c0442:0x40ed
postInstantiation	@	index.js:4631
(anonymous)	@	index.js:4653
Promise.then (async)
loadModule	@	index.js:4652
(anonymous)	@	index.js:4666
Promise.then (async)
loadWebAssemblyModule	@	index.js:4665
(anonymous)	@	index.js:4736
Promise.then (async)
getLibModule	@	index.js:4735
loadDynamicLibrary	@	index.js:4748
(anonymous)	@	index.js:51386
(anonymous)	@	index.js:51385
Promise.then (async)
start	@	index.js:51367
(anonymous)	@	index.js:51430
Promise.then (async)
startGame	@	index.js:51429
(anonymous)	@	(index):224
(anonymous)	@	(index):244
  1. Tried to increase stack size with -C link-arg=-sSTACK_SIZE=8388608 - same error as above (the addresses remain the same as well)
  2. Made a hello_gdext project (using the same gdext commit at 2.) like the docs suggest (stopping at the first play test, where the Player just rotates)
    • godot/ and rust/ structure
  3. Added the same link args as 41. to rust/.cargo/config.toml, like so:
[target.wasm32-unknown-emscripten]
rustflags = [
  "-Clink-arg=--no-entry",
  "-Clink-arg=-O1",
  "-Clink-arg=-sASSERTIONS=2",
  "-Clink-arg=-sSAFE_HEAP=1",
  "-Clink-arg=-sSTACK_OVERFLOW_CHECK=2",
  "-Clink-arg=-sDEMANGLE_SUPPORT=1",
  "-Clink-arg=-g",
  "-Clink-arg=-sSTACK_SIZE=8388608",
  "-Clink-arg=-sINITIAL_MEMORY=4294967296",
  "-Clink-arg=-sMAXIMUM_MEMORY=4294967296",
  "-Clink-arg=-sALLOW_MEMORY_GROWTH=1",
  "-Clink-arg=-sEMULATE_FUNCTION_POINTER_CASTS=1",
  "-Clink-arg=-sSIDE_MODULE=2",
  "-Clink-arg=-sUSE_PTHREADS=1",
  "-Ctarget-feature=+atomics,+bulk-memory,+mutable-globals"
]
  1. Compiled with cargo +nightly build --target wasm32-unknown-emscripten (with 33. kept)
  2. In Firefox, still out of memory error; in Ungoogled Chromium, new error:
WebAssembly.instantiate(): Compiling function #700:"godot_ffi::gen::table_scene_classes::ClassScene..." failed: local count too large @+416567
displayFailureNotice @ (index):212
Promise.then (async)
(anonymous) @ (index):239
(anonymous) @ (index):244
  1. Interesting result. Noticed that the previous project (Dodge the Creeps) had default-features = false on gdext.
  2. Tried to disable default features for this project (Hello Gdext) as well.
  3. The result then became the same as for the Dodge The Creeps project: stack overflow error instead of "local count too large".
  4. Also, Firefox had the same errors as Chromium now, indicating that the "out of memory" error was somehow sporadic.
  • It seems that restarting Firefox fixes that problem whenever it appears.
  1. After seeing this issue, I attempted compiling in release mode instead (cargo +nightly build --release --target wasm32-unknown-emscripten).
  2. After running the exported game in release mode (Debug mode disabled in Godot as well), the 'local count too large' error was replaced by the aforementioned stack overflow error, even with default features not disabled!
  3. I disabled default-features again and tested some suggestions from this emscripten issue, including using -O0, removing debug assertions, setting stack size to 50MB (and enough initial memory), and got an "index out of bounds" error on Firefox, in some "cull_zombies" internal emscripten function, at the line "i32.load offset=56" (image here).
  4. (Building Godot from source after new WebAssembly info) Cloned Godot with git clone https://github.com/godotengine/godot.
  5. Checked out to the commit for 4.2-dev6 with git checkout --detach 57a6813bb8bc2417ddef1058d422a91f0c9f753c.
  6. Switched to emscripten 3.1.28 following the steps from step 20 (replacing 3.1.14 with 3.1.28), as per info from godotengine/godot#82865.
  7. Compiled Godot web export templates with scons platform=web production=yes dlink_enabled=yes debug_symbols=yes target=template_debug
  8. Moved the resulting bin/godot.web.template_debug.wasm32.dlink.zip to somewhere accessible by the Godot editor.
  9. (Optional) Compiled the editor for that version with scons platform=linuxbsd target=editor (you can also just download the editor from https://github.com/godotengine/godot-builds/releases/tag/4.2-dev6).
  10. Recompiled Hello gdext with emscripten 3.1.28 activated as per step 46.
  1. From the editor, opened Hello gdext and exported it while selecting, on the Export Template > Debug line, the .zip file generated in step 60. (Ensure 'Export with Debug' is checked when selecting the destination.)
  2. Used serve.py from step 17 to host the exported game and open it on the browser.
  3. Got a stack overflow error.
  4. Added emscripten flags (cf. step 45) -sSTACK_SIZE=2000MB -sINITIAL_MEMORY=3999MB -sALLOW_MEMORY_GROWTH=1 and removed ASSERTIONS and STACK_OVERFLOW_CHECK flags, and recompiled the Rust extension and re-exported with the same web export template.
  5. Got native code called abort(). Seems like Rust code panicked.
  6. Through debugging with breakpoints (use emscripten flags -g -sDEMANGLE_SUPPORT=1), tracked the problem down to this function: https://github.com/godot-rust/gdext/blob/3491d7bc91c9711dd4139b3db104aa37bb419bd1/godot-ffi/src/compat/compat_4_1.rs#L28 (ensure_static_runtime_compatibility).
  7. The function only exists to ensure we're not using 4.0, so we can just return from it early on WASM, by adding this patch:
diff --git a/godot-ffi/src/compat/compat_4_1.rs b/godot-ffi/src/compat/compat_4_1.rs
index e258e58..679a31a 100644
--- a/godot-ffi/src/compat/compat_4_1.rs
+++ b/godot-ffi/src/compat/compat_4_1.rs
@@ -49,6 +49,10 @@ impl BindingCompat for sys::GDExtensionInterfaceGetProcAddress {
         // first fields have values version_major=4 and version_minor=0. This might be deep in UB territory, but the alternative is
         // to not be able to detect Godot 4.0.x at all, and run into UB anyway.
 
+        if cfg!(target_arch = "wasm32") {
+            return;
+        }
+
         let get_proc_address = self.expect("get_proc_address unexpectedly null");
         let data_ptr = get_proc_address as *const LegacyLayout; // crowbar it via `as` cast

The patch is (currently) available as its own branch here: https://github.com/PgBiel/gdext/tree/wasm-test

  1. Recompiling with said patch applied to gdext (and default-features = false as before), the error changes to Assertion failed: bad function pointer type - dynCall function not found for sig 'jji'
    • Apparently this is due to a function pointer cast that is UB and isn't properly supported on WASM.
  2. Compiling with -sEMULATE_FUNCTION_POINTER_CASTS=1 results in the error Cannot convert 491636 to BigInt. Using -sWASM_BIGINT=1 doesn't change that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment