jimmywarting/cjs-in-esm.md

## cjs-in-esm.md

      
    Raw
  

              cjs-in-esm.md
            
          
    This is a response to https://gist.github.com/joepie91/bca2fda868c1e8b2c2caf76af7dfcad3?permalink_comment_id=4767875#gistcomment-4767875

if you are so sure about using 'import' from common js then send some code samples - maybe it will help someone else.

Sure @cybafelo, i can give you 3 example (from easy to more adv)
i'm going to create a basic hello world example that uses express (cjs) and node-fetch (esm-only)
here is what my package.json looks like:
{
  "type": "commonjs",
  "dependencies": {
    "express": "^4.18.2",
    "node-fetch": "^3.3.2"
  }
}
To be explicit i have set the type to commonjs to make no confusion that it's definitely a cjs application
Option 1 - using dynamic import() in cjs

the fetch api is just a function that returns a promise, so it could be easy to just defer the module and only load it when it's absolutely necessary by calling a proxy function that first import the module and then passes the argument along to fetch when it has been imported.
const express = require('express')
const app = express()
const port = 3003

// wrap esm-only node-fetch with an lazy dynamic import that is imported on first use
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));

app.get('/*', async (req, res) => {
  const response = await fetch('https://httpbin.org/' + req.url, {
    headers: req.headers,
    method: req.method,
    body: req.body
  })
  response.body.pipe(res)
})

app.listen(port, () => {
  console.log(`Example app listening on http://0.0.0.0:${port}`)
})
Option 2 - bootstrap the application and load all modules before your own code starts to execute

This requires that you have one single bootstrap-ing .mjs file that will load first and formost, when it have loaded all modules only then will it load your own application,
So the only thing you have to change is something like
- "start": "node app.js"
+ "start": "node bootstrap.mjs" 
// bootstrap.mjs

import { readFileSync } from 'node:fs'

const pkg = JSON.parse(readFileSync('./package.json', 'utf8'))

const imports = Object.keys(pkg.dependencies).map(async dep => {
  console.log(`import ${dep} from '${dep}';`)
  const module = await import(dep)
  return [dep, module]
})

globalThis.dependencies = Object.fromEntries(await Promise.all(imports))
console.log('loaded all dependencies')
console.log('')
console.log('starting the application itself')

import('./app.js')
// app.js

const express = globalThis.dependencies.express.default
// const express = require('express') - this works also.
const app = express()
const port = 3003

// `bootstrap.mjs` will have already imported the dependencies
const {
  default: fetch,
  Response,
  Request,
  Blob,
  fileFromSync,
} = globalThis.dependencies['node-fetch']

const file = fileFromSync('./package.json')
// console.log(file.name, file.size)

app.get('/*', async (req, res) => {
  const response = await fetch('https://httpbin.org/' + req.url, {
    headers: req.headers,
    method: req.method,
    body: req.body
  })
  response.body.pipe(res)
})

app.listen(port, () => {
  console.log(`Example app listening on http://0.0.0.0:${port}`)
})
i know what you might think... pooluting global context with dependencies and using const express = globalThis.dependencies.express.default instead of require('express'). it's ugly, but that is beside the point.
i could come up with a nicer way to design this, eg by manipulating require and pre populate it with things. so that require('node-fetch') have been cached with an already loaded esm-only package and did not read from any files when the app.js tries to load node-fetch
the solution i wanted to show you is that you can have a simple esm file that first imports all the esm-only modules first and formost (asynchronously) and once that's completed it would start the application as normally.
this is by no means a good solution for library developers that wants to publish something on npm.
then option 1 would be better.
option 3 - trying to have both esm and cjs dual runtime (like bun.js)

This solution is a bit more trickier to get up and running...
it involves

defaulting to esm (by setting "type": "module" in package.json)
register a custom import module loader
prepend own scoped variables like __dirname, __filename and require() on top of every file. (at runtime)

// register.js

import { register } from 'node:module'
register(import.meta.resolve('./hook.js'))
// hook.js

import { fileURLToPath } from 'node:url'
import { createRequire } from 'node:module'
import path from 'node:path'

async function load(url, context, nextLoad) {
  if (url === import.meta.url ||
    !url.startsWith('file://') ||
    url.includes('node_modules')
  ) return nextLoad(url, context);

  const { source } = await nextLoad(url, { ...context, format: 'module' });

  const header = `let { exports, require, module, __filename, __dirname } = await import('${import.meta.url}').then(_ => _._(import.meta));\n`
  const sourceCode = header + source.toString()

  return {
    format: 'module',
    shortCircuit: true,
    source: sourceCode,
  }
}

function _(meta) {
  // this will try to mimic nodejs module system somewhat
  //
  // (function(exports, require, module, __filename, __dirname) {
  //   // Module code actually lives in here
  // })

  return {
    require: createRequire(meta.url),
    __filename: path.dirname(fileURLToPath(meta.url)),
    __dirname: fileURLToPath(meta.url),
    module: {
      exports: {},
    }
  }
}

export {
  load,
  _,
}
// app.js

import fetch, { fileFromSync } from 'node-fetch'
const express = require('express')
const app = express()
const port = 3003

console.log(fileFromSync)
console.log(__dirname)
console.log(__filename)

app.get('/*', async (req, res) => {
  const response = await fetch('https://httpbin.org/' + req.url, {
    headers: req.headers,
    method: req.method,
    body: req.body
  })
  response.body.pipe(res)
})

app.listen(port, () => {
  console.log(`Example app listening on http://0.0.0.0:${port}`)
})
node --import ./register.js ./app.js 
And love and behold: now your app can use both cjs and esm in a way that you like.
it's more or less like trying to add this at top of every file (but is being done automatically for you at runtime)
import {fileURLToPath} from 'node:url'
import path from 'node:path'
import { createRequire } from 'node:module'

const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(fileURLToPath(import.meta.url))
const require = createRequire(import.meta.url)
One con is that anything you require() won't execute in a ESM runtime so those file that have been required can't use any ESM syntax. (or top level await)
the problem i had was that the ES import hook loader would not get triggered when loading cjs files
one way to override this could be to change require into async import()
require(requirePath) {
  import(meta.resolve(requirePath))
},
this had some pro & con in itself. the good thing where that it would run in a ESM runtime, and have both support for cjs and esm. (cuz the hook.js was running when calling our own require()
the con where that now require() now also became async... so returning anything became an issue.
if there had been a importSync() then this would be golden.. i tried looking into atomic wait and deasync but did not found a solution that did work.
anyway... i think this is a good solution to have while migrating to ESM. you kind of have to convert all the files like a file tree (starting from the root file) so that you can take your time converting things in a slow and steady pace
the goal i had was to try and have support for both esm and cjs at runtime (rather than using something like a compiler/transpiler that transform all import/export to & from cjs & esm to one format)
most ppl have tried having support for esm in a cjs runtime (i wanted to try the other way around by having cjs support in esm runtime)
i had some plan of appending some kind of footer to all files that took what had been assigned to module.exports and some how do a export default module.exports or something...
but did not want to spend any more time on this...