Skip to content

Instantly share code, notes, and snippets.

@surma
Last active March 8, 2024 12:06
Show Gist options
  • Star 67 You must be signed in to star a gist
  • Fork 14 You must be signed in to fork a gist
  • Save surma/b2705b6cca29357ebea1c9e6e15684cc to your computer and use it in GitHub Desktop.
Save surma/b2705b6cca29357ebea1c9e6e15684cc to your computer and use it in GitHub Desktop.
webpack-emscripten-wasm

Minimal example making webpack and wasm/Emscripten work together.

Build instructions:

  • Clone this gist
  • npm install
  • npm start
  • Open http://localhost:8080
  • Look at console

Note: Docker is required to build this project.

I filed a bug with webpack to make this integration easier.

More questions? Hit me up on Twitter.

License Apache-2.0

<3 Surma

/**
* Copyright 2018 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include <emscripten.h>
EMSCRIPTEN_KEEPALIVE
int fib(int n) {
int i, t, a = 0, b = 1;
for (i = 0; i < n; i++) {
t = a + b;
a = b;
b = t;
}
return b;
}
<!doctype html>
<script src="./dist/bundle.js"></script>
/**
* Copyright 2018 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import fibonacci from './fibonacci.js';
import fibonacciModule from './fibonacci.wasm';
// Since webpack will change the name and potentially the path of the
// `.wasm` file, we have to provide a `locateFile()` hook to redirect
// to the appropriate URL.
// More details: https://kripken.github.io/emscripten-site/docs/api_reference/module.html
const module = fibonacci({
locateFile(path) {
if(path.endsWith('.wasm')) {
return fibonacciModule;
}
return path;
}
});
module.onRuntimeInitialized = () => {
console.log(module._fib(12));
};
{
"name": "lol",
"scripts": {
"build:codec": "docker run --rm -v $(pwd):/src trzeci/emscripten emcc -O3 -s WASM=1 -s EXTRA_EXPORTED_RUNTIME_METHODS='[\"cwrap\"]' -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s 'EXPORT_NAME=\"fibonacci\"' -o ./fibonacci.js fibonacci.c",
"build:bundle": "webpack",
"build": "npm run build:codec && npm run build:bundle",
"serve": "http-server",
"start": "npm run build && npm run serve"
},
"devDependencies": {
"exports-loader": "^0.7.0",
"file-loader": "^1.1.11",
"http-server": "^0.11.1",
"webpack": "^4.8.3",
"webpack-cli": "^2.1.3"
}
}
/**
* Copyright 2018 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const webpack = require("webpack");
const path = require("path");
module.exports = {
mode: "development",
context: path.resolve(__dirname, "."),
entry: "./index.js",
output: {
path: path.resolve(__dirname, "dist"),
filename: "bundle.js"
},
// This is necessary due to the fact that emscripten puts both Node and web
// code into one file. The node part uses Node’s `fs` module to load the wasm
// file.
// Issue: https://github.com/kripken/emscripten/issues/6542.
browser: {
"fs": false
},
module: {
rules: [
// Emscripten JS files define a global. With `exports-loader` we can
// load these files correctly (provided the global’s name is the same
// as the file name).
{
test: /fibonacci\.js$/,
loader: "exports-loader"
},
// wasm files should not be processed but just be emitted and we want
// to have their public URL.
{
test: /fibonacci\.wasm$/,
type: "javascript/auto",
loader: "file-loader",
options: {
publicPath: "dist/"
}
}
]
},
};
@ascorbic
Copy link

@Quirksmode: You need to change the browser options to:

node: {
	fs: "empty"
}

@elv-peter
Copy link

$ npm run build:bundle

> lol@ build:bundle /p/e/3rdparty/webpack-emscripten-wasm
> webpack

/p/e/3rdparty/webpack-emscripten-wasm/node_modules/webpack-cli/bin/config-yargs.js:89
				describe: optionsSchema.definitions.output.properties.path.description,
				                                           ^

TypeError: Cannot read property 'properties' of undefined
    at module.exports (/p/e/3rdparty/webpack-emscripten-wasm/node_modules/webpack-cli/bin/config-yargs.js:89:48)
    at /p/e/3rdparty/webpack-emscripten-wasm/node_modules/webpack-cli/bin/webpack.js:60:27
    at Object.<anonymous> (/p/e/3rdparty/webpack-emscripten-wasm/node_modules/webpack-cli/bin/webpack.js:515:3)
    at Module._compile (internal/modules/cjs/loader.js:707:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
    at Module.load (internal/modules/cjs/loader.js:605:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
    at Function.Module._load (internal/modules/cjs/loader.js:536:3)
    at Module.require (internal/modules/cjs/loader.js:643:17)
    at require (internal/modules/cjs/helpers.js:22:18)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! lol@ build:bundle: `webpack`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the lol@ build:bundle script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

@elv-peter
Copy link

elv-peter commented Nov 14, 2018

It worked after the configuration change above and updating webpack-cli to 3.1.2

And "exports-loader" doesn't seem to be necessary. I think it also doesn't do anything if you inspect (e.g. by adding console.log to) the loader output.

Also, Emscripten has a "-s ENVIRONMENT=web" option

@shayc
Copy link

shayc commented Nov 26, 2018

@elv-peter thank you, I've been suffering so much trying to integrate. Managed to build with your help! May you win a billion internet points.

@kevinchar93
Copy link

kevinchar93 commented Nov 28, 2018

Stumbled upon this thread while looking for answers, appears that there has now been change made to Emscripten to stop it outputting node.js related code in any output .js file.

Adding -s ENVIRONMENT="web" to my build line resolved the issues I was seeing.
Hope thins helps!

See PR emscripten-core/emscripten#6565

@kevinchar93
Copy link

Came back to add another update, used this technique to successfully load a .wasm file from a location other than the same directory as the .js file that Emscripten generates to load said .wasm file.

My issues was effectively the same as this question:
https://stackoverflow.com/questions/46332699/how-can-i-load-wasm-files-stored-in-a-subdirectory

Some differences I experienced / gotchas:

  • if the publicPath for the file-loader config is incorrect fetch requests for the .wasm file will fail (404) causing the streaming compilation to fail (expected magic word 00 61 73 6d) as it tired to load the response, so double check that this is correct.
  • I didn't have to use the exports-loader , not sure if this will cause issues later, hopefully not!

Many Thanks!

@GreLI
Copy link

GreLI commented Jun 7, 2019

Compiling with option -s "ENVIRONMENT='web'" solves the fs problem. (There is also, FILESYSTEM=0 option.)

@tzvc
Copy link

tzvc commented Jan 7, 2020

Hey guys,

I'm trying to load a WASM module from a react project created with create-react-app and I'm having trouble finding the correct publicPath cause I keep getting

wasm streaming compile failed: CompileError: WebAssembly.instantiateStreaming(): expected magic word 00 61 73 6d, found 6d 6f 64 75 @+0

my glue code and my wasm sources are located in /src/video/ so I have the following webpack rule:

 // override default create-react-app config
  config.module.rules.push({
    test: /video\/rPPG_asm\.wasm$/,
    type: 'javascript/auto',
    loader: 'file-loader',
    options: {
      publicPath: 'public/'
    }
  });

When I look at the chrome network console however, a call is made by my glue code at

http://localhost:3000/static/media/rPPG_asm.9df80b34.wasm

Any Ideas ?

@tzvc
Copy link

tzvc commented Jan 27, 2020

Following my previous comment, I digged in a bit more:

First, I think it's important to mention that I'm working with a webpack config created by create-react-app that I override using https://github.com/timarney/react-app-rewired

With this webpack override config

  config.module.rules.push({
    test: /rPPG_asm\.wasm$/,
    type: 'javascript/auto', // ← !!
    loader: 'file-loader',
    options: {
      publicPath: 'public/static/wasm/'
    }
  });

I get two files are generated in the build:

/public/static/media/rPPG_asm.6074f4a1.wasm which contains

module.exports = "public/static/wasm/dad54818a8426f637a10e1e56c6659b5.wasm";

and /public/static/wasm/dad54818a8426f637a10e1e56c6659b5.wasm

which contains the actual wasm

Thing is, the path provided in the LocateFile function is this one /public/static/media/rPPG_asm.6074f4a1.wasm so the glue code tries to load that as valid WASM hence the magic number error.

What am I missing here ? :)

@skandonkumar
Copy link

skandonkumar commented Mar 4, 2020

EMScripten locateFile() not reading .WASM file after bundling with WebPack #10637

I have converted my C library to WASM and wrapper JS using EMScripten. I also wrote my own JS library. I want to integrate both the libraries. I used Webpack to bundle my JS library including my Emscripten wrapper JS and added the WASM file using "file-loader" of WebPack. Initially, the Emscripten generated JS wrapper reads the WASM file using locateFile() method. But after bundling, the locate file is unable to locate the WASM file. Can someone please help me?

@programer-s
Copy link

programer-s commented Nov 16, 2020

Hello,
I can't understand how this code is supposed to work:

locateFile(path) {
    if(path.endsWith('.wasm')) {
      return fibonacciModule;
    }
    return path;
  }

fibonacciModule is not string, and emscripten doesn't find the wasm module.
Please help to fix this issue.

@skandonkumar
Copy link

Hello,
I can't understand out how this code is supposed to work:

locateFile(path) {
    if(path.endsWith('.wasm')) {
      return fibonacciModule;
    }
    return path;
  }

fibonacciModule is not string, and emscripten doesn't find the wasm module.
Please help to fix this issue.

locateFile(path) is a hook for the Emscripten generated Javascript file to find the .WASM file. By default without the hook, the JS file looks for WASM file in the root directory. Your code looks correct to look for .wasm file or just return the path specified in the locateFile() hook.

@programer-s
Copy link

Thank you for your answer! This code is not mine. (from the example above)
fibonacciModule is javascript object, and late I get an error in the next code (Emscripten generated Javascript file):

function hasPrefix(str, prefix) {
  return String.prototype.startsWith ?
      str.startsWith(prefix) :
      str.indexOf(prefix) === 0;
}

startsWith is not a function, because str is not a string type. (fibonacciModule)

If I think correctly then the locateFile function returns a string type and fibonacciModule is not the string type.

@Modjular
Copy link

Modjular commented Feb 5, 2021

@surma Should this work in a Web Worker? I've been told webpack now supports wasm out-of-the-box
https://github.com/webpack/webpack/tree/master/examples/wasm-simple but I haven't had any luck, with this method or anything else on google.

I'm not sure if I'm using outdated methods, or I shouldn't even be attempting this in a worker. Can you shine some light on this?

@surma
Copy link
Author

surma commented Feb 5, 2021

I haven’t looked into Wasm + Webpack or Workers + Webpack with Webpack 5. So I don’t really know.

@igormcoelho
Copy link

This original gist helped me years ago, but it stopped working (for me...). Some advices that worked here:

  • updates on package.json:
    • "build:codec": "docker run --rm -v $(pwd):/src emscripten/emsdk emcc -O3 -s WASM=1 -s EXPORTED_RUNTIME_METHODS='[\"cwrap\"]' -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s 'EXPORT_NAME=\"fibonacci\"' -o ./fibonacci.js fibonacci.c"
      • Because of warning: emcc: warning: EXTRA_EXPORTED_RUNTIME_METHODS is deprecated, please use EXPORTED_RUNTIME_METHODS instead [-Wdeprecated]
      • Also because image trzeci/emscripten seems to be officially replaced by emscripten/emsdk
    • fixed devDependencies because some packages were old, specially http-server (webpack is still v4 and couldn't manage to make it work on v5):
      • "exports-loader": "^0.7.0",
      • "file-loader": "^1.1.11",
      • "http-server": "^13.0.2",
      • "webpack": "^4.8.3",
      • "webpack-cli": "^3.1.2"
  • updates on webpack.config.js:
    • Added: library to module.exports.output:
      • library: 'TestLib'
  • updates on index.js:
    • need to process async promise to get module working
    • did not manage to use onRuntimeInitialized as in the original example

index.js

 import fibonacci from './fibonacci.js';
 import fibonacciModule from './fibonacci.wasm';
 
 export var loadedModule = null;  // storing it on TestLib module

const module = fibonacci({
   locateFile(path) {
     if(path.endsWith('.wasm')) {
       return fibonacciModule;
     }
     return path;
   }
 }).then(instance => {
    console.log("use it!");
    console.log("_fib -> "+instance._fib(5));
    loadedModule = instance;
  });

On the browser, TestLib.loadedModule._fib(5) correctly invokes the function fib.

index.html

<!doctype html>
<script src="./dist/bundle.js"></script>

webpack.config.js

 const webpack = require("webpack");
 const path = require("path");

 module.exports = {
   mode: "development",
   context: path.resolve(__dirname, "."),
   entry: "./index.js",
   output: {
     path: path.resolve(__dirname, "dist"),
     filename: "bundle.js",
     library: 'TestLib',
     globalObject: 'this' // https://webpack.js.org/configuration/output/
   },
   node: {
    fs: "empty"
    },
   module: {
     rules: [
       {
         test: /fibonacci\.js$/,
         loader: "exports-loader"
       },
       {
         test: /fibonacci\.wasm$/,
         type: "javascript/auto",
         loader: "file-loader",
         options: {
           publicPath: "dist/"
         }
       }
     ]
   },
 };

packages.json:

{
    "name": "lol",
    "scripts": {
      "build:codec": "docker run --rm -v $(pwd):/src emscripten/emsdk  emcc -O3  -s WASM=1  -s ASSERTIONS=1 -s EXPORTED_RUNTIME_METHODS='[\"cwrap\"]' -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s 'EXPORT_NAME=\"fibonacci\"'  -o ./fibonacci.js fibonacci.c",
      "build:bundle": "webpack",
      "build": "npm run build:codec && npm run build:bundle",
      "serve": "http-server",
      "start": "npm run build && npm run serve"
    },
    "devDependencies": {
      "exports-loader": "^0.7.0",
      "file-loader": "^1.1.11",
      "http-server": "^13.0.2",
      "webpack": "^4.8.3",
      "webpack-cli": "^3.1.2"
    }
  }

fibbonacci.c (original file)

#include <emscripten.h>

EMSCRIPTEN_KEEPALIVE
int fib(int n) {
  int i, t, a = 0, b = 1;
  for (i = 0; i < n; i++) {
    t = a + b;
    a = b;
    b = t;
  }
  return b;
}

@dhdaines
Copy link

dhdaines commented May 6, 2022

No longer works with Webpack 5 - instead of the node configuration, you need something like:

module.exports = {
    resolve: {
        fallback: {
            crypto: false,
            fs: false,
            path: false
        }
    }
};

For the wasm files, one solution is just to use copy-webpack-plugin, which is annoying since it makes it non-transparent for people to use your module, but then again, they're already adding the magic resolve incantation...

const CopyPlugin = require("copy-webpack-plugin");
module.exports = {
    plugins: [
        new CopyPlugin({
          patterns: [
              { from: "node_modules/MYPACKAGE/MYPACKAGE.wasm*",
                to: "[name][ext]" },
          ],
      }),
    ],
};

@9oelM
Copy link

9oelM commented May 10, 2022

Hi guys, I felt extremely lost while I was digging into this issue, so after I got everything working, I made a example repo containing detailed walkthroughs:

Hope it helps someone.

@acransac
Copy link

With Webpack 5, I think you can remove the locateFile hook, the file-loader dependency and simplify

    import fibonacciModule from './fibonacci.wasm';

to

    import './fibonacci.wasm';

provided that you change the .wasm rule in the Webpack config to:

    {
        test: /fibonacci\.wasm$/,
        type: "asset/resource",
        generator: {
            filename: "[name].wasm"
        }
    }

It might help with issues locating the wasm asset when loading the page.

@Spencer-Sherk
Copy link

Hi guys, I felt extremely lost while I was digging into this issue, so after I got everything working, I made a example repo containing detailed walkthroughs:

Hope it helps someone.

Thanks! You just saved me a ton of time with this. Confirmed still working as of 2/2/2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment