Skip to content

Instantly share code, notes, and snippets.

@greim
Last active June 8, 2022 17:21
Show Gist options
  • Save greim/e8e1bd3b1ed08a7505ce to your computer and use it in GitHub Desktop.
Save greim/e8e1bd3b1ed08a7505ce to your computer and use it in GitHub Desktop.

How you can help reduce node_modules bloat

This recent reddit thread reveals discontent among the web development community about the sheer volume of stuff in a typical node_modules dir. 140MB in this case!

Is it a design flaw in npm?

Opinions in the thread varied from "I'm surprised npm even works" to "everything is fine". I'm not going to offer an opinion, just these two observations:

  1. node_modules dirs typically do contain lots of stuff that doesn't need to be there.
  2. The latest version mitigates overall size by flattening the dependency tree, but some of the bloat is beyond npm's control.

The beyond npm's control part is where we come in, and it has to do with how npm modules are developed and published. Specifically, two common oversights on the part of module owners contribute to node_modules bloat:

First: faulty package.json configurations

Let's review the difference between dependencies and devDependencies inside package.json. The former is what the module needs to work as advertized. The latter is what people need to contribute to its development.

If you npm install thing, everything in its dependencies gets downloaded into your node_modules dir, while everything in its devDependencies is ignored. After all, if you wanted to contribute, you wouldn't npm install thing, you'd git clone thing and then npm install.

Unfortunately, module owners sometimes forget to pay attention to this, so things like gulp or mocha find their way into dependencies, forcing everyone in the world to download them on install.

Fortunately, "Move X to devDependencies" is an easy pull request. Better yet, it fixes a small problem permanently, for everyone. It's also a great way to get into OSS development and building rapport with module owners!

Second: missing or incomplete .npmignore files

By default, everything in your project gets uploaded to npm, and subsequently downloaded to users' machines on install. npm blacklists .git dirs and a few other obvious things, but it can't read you mind.

You have to tell it to ignore your tests/ dir and/or *.test.js files, for example, or that 3MB of test data, or the PhantomJS binary. All of that stuff becomes dead weight in node_modules dirs, planet-wide.

A simple .npmignore file in the project root—which works just like .gitignore—can fix this, but module owners sometimes forget, or don't know it's an option. I didn't know it either until someone pointed it out to me.

# npmignore file
tests/
*.test.js
phantomjs/

This is another opportunity for a simple PR which will forever fix the issue for everyone on the planet.

Finally: be nice to OSS developers!

I'm emphasizing PRs instead of telling module owners to fix their stuff, because OSS is a community effort. Yes, owners should fix their stuff, but there's a worrying lack of gratitude in the wider community toward OSS developers. These are people who create useful things in their spare time, and then give it away for free.

I'd personally like to keep developing and using OSS for the foreseeable future, but that means being respectful of others' time and contributions, and striving to be helpful in whatever ways I can. Ultimately it means assuming some responsibility for the things I don't like.

@jikkujose
Copy link

We need to do something about this collectively. Perhaps make a checklist & send it to every module owner. Collaboratively check the top 1000 modules. Think it will be trivially possible considering node land has significant community strength. Perhaps another solution to this is a new standard for the packages (we should be able to generate most of the following):

  • Create a branch 'releases'
  • It should contain a zip of:
    • The final (minified if possible) files only
    • Plain text file with only the required dependencies
  • Tag the branch with the version of the package

This will allow npm to clone the 'releases at specified version' with --depth 1 flag, downloading only the very minimal requirement. Perhaps overall this may reduce the file size significantly.

Currently with this size; its practically impossible to use npm in developing countries with spotty internet. Hope we can do something about this.

@Hum4n01d
Copy link

I totally agree, thanks @greim

@ignotus0001
Copy link

ignotus0001 commented Nov 11, 2017

Thanks for this. I'd forgotten about .npmignore.

I'm researching the topic of node_modules bloat because my team is doing some AWS development using Lambda functions in a node.js environment.

Have you checked out modclean by any chance? Looks like something that can take pruning to the next level.

Also, npm and yarn have dedupe commands, non?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment