What's wrong with the npm workflow?

npm-woes.md

Sara Chipps started this awesome thread about bundler vs. npm. You should check it out cause it's highly entertaining. https://twitter.com/SaraJChipps/status/484727602049859584

But I do want to get more info on the workflow problems people experience with npm. It has always been the simplest thing to me. But I'm also fully aware that perceptions are relative. Most ruby folks are totally fine with bundler even though it gives me constant headaches. But the bundler issues are well documented. This seems to be less true for npm. I want to gather some info about people's experiences. I promise I'll try to refrain from the canned "you're doing it wrong" response.

Also, we've already established that npm is network hungry and could do a better job of maintaining an offline cache. https://twitter.com/ag_dubs/status/484733343108706305 Nobody's arguing that. But I'm less concerned about feature requests here. I'm interested in where npm breaks for you.

Here's Ryan Florence. https://twitter.com/ryanflorence/status/484730296156176385

`rm -rf node_modules && npm cache clean && npm install && npm dedupe` feels almost as reassuring as just `bundle install`

I want to hear about the step before this where something's wrong and a simple npm install doesn't fix it. Messing around with the cache suggests there's something very strange in people's workflows. And messing around with dedupe suggests that the dependency tree that people are trying to achieve is very different from mine. Trashing your whole node_modules folder shouldn't be required. But it's usually an easier step than trying to debug whatever weird state you got yourself into. I believe this is because npm installs don't usually fail and you can just refresh your whole dep tree fairly easily (network permitting).

But these are just my own perceptions. So lets hear it. Why do you hate npm?

I haven't yet found an actual error that needed npm cache clean that wasn't "I switched registries and one was horribly broken".

remove-and-start-over sounds like a feature if you want it -- npm ls and npm outdated should be ways to be more fine-grained about it. It shouldn't be normal workflow, though.

npm dedupe sounds cargo-culted to me. I believe that a fresh npm install should yield no changes in that case.

The only things I ever do are rm -rf node_modules; npm install and npm install.

Now caching could be vastly improved, and I believe is THE hotbutton item holding up further client development -- what's there is entangled, ugly, and frustrating.

As a counterpoint to bundler being well documented, I feel like it's two hours of voodoo to make it work every time, and I don't know what makes it go. I feel like I'm missing something foundational that if it was just explained, I might not hate it. Haven't found it in any docs, though. I dread every new ruby app I have to mess with.

I have occasionally needed to clean my cache, but it was almost always from connectivity or proxy issues. The main thing that I have been fighting is the scenario where you want to bundle a dependency and check it in to git. A lot of devs put node_modules in their .gitignore and check that in, so if you bundle and that happens, the package can be incomplete (which isn't npm's fault at all, npm just doesn't recursively check already installed dependencies on npm install). I could be wrong, but i thought I had read that it was being worked on to recursively check dependencies...

The one scenario (bug) that I can recall off the top of my head where npm cache clean is required is when you install a git url of a module that has fork modifications, and then you revert that and go back to the npm registry semver version, then the cache seems to retain the git contents and it won't pick up the newer version.

I just now ran into this when cloning a branch of Ember, which uses npm to manage builds:

npm WARN unmet dependency /Users/tomdale/Code/ember.js/node_modules/ember-cli/node_modules/broccoli-sane-watcher requires rsvp@'~3.0.8' but will load
npm WARN unmet dependency /Users/tomdale/Code/ember.js/node_modules/rsvp,
npm WARN unmet dependency which is version 3.0.6

(I had previously run npm install in the directory, and ran it again after checking out the new branch.)

Deleting node_modules and re-running the command worked fine with no warnings. This is much different than my experience with Bundler, where running bundle install multiple times is idempotent.

So - is it just the npm cache clean bit that's most annoying?

What would be the ideal user flow?

(Since we can actually do something about this, I'd like to know how best to start the conversation with relevant parties ;-))

@rockbot what's your understanding of @tomdale's issue? I admit I very rarely get this kind of warning. Is this part of the "peer dependencies" topic? I admit I have avoided that whole thing because it always sounded like a bad idea to me.

In Tom's example, I would prefer to have 2 versions of rsvp installed. On at the root and one under broccoli-sane-watcher. I believe the problem is that if deps for broccoli-sane-watcher are not installed yet, but you do npm install from the root of your project, npm recurses up the dir tree and finds rsvp in the project root. That version doesn't meet the requirements for the broccoli module. But if you run this program and the broccoli module calls require('rsvp') it will get that version, but it's own version is not installed.

That makes sense based on how npm module loading works. What I don't understand is how you end up with broccoli-sane-watcher installed but no rsvp underneath it. Is broccoli checked into version control?

FYI, here is the bug in npm that will bite you in the ass if you are checking in deps. npm/npm#1341

I submitted it a long time ago and it has a billion +1s. People usually check in their top level deps without second level deps. But when you deploy and run npm install, it will not recurse into them and install their 2nd level deps. npm update will do this. We changed our deploy scripts to run that and haven't been bitten by this problem again. But it should really be the default behavior of npm install as well.

@evanlucas

A lot of devs put node_modules in their .gitignore and check that in, so if you bundle and that happens, the package can be incomplete (which isn't npm's fault at all, npm just doesn't recursively check already installed dependencies on npm install).

Two errors in that statement.

1. if you npm install something, and it has a .gitignore file, then one of two things are done with it. First, if there is not a .npmignore file, then it is renamed to .npmignore. (Empirically, people find it surprising when a gitignore'd file is published.) Second, if there already is a .npmignore file, then the .gitignore file is omitted from the package. So, unless you're using a very old version of npm, it should be fine.
2. When you install a package, and then move things around with npm, and now that packages dependencies are no longer available, then that is npm's fault. It should not allow broken situations without complaining a lot about it in the process, and telling you how to fix it.

In other words, yes, it should be (more) idempotent.

Re npm/npm#1341, the dicey thing there is that we're now saying that "bundledDependencies" is no longer "sacrosanct". Currently, bundling a dependency means that you are
taking 100% responsibility for its contents. npm will not touch it. This makes it possible to float patches, etc.

I'm fine with changing that contract a little bit. But it's a significant version change, and requires a lot of the same work involved with idempotent installs, flatter folder structures, and dedupe-on-install. In all cases, what we need is to do the refactoring that's already in progress to split the install command up into multiple stages.

Thanks for bringing this up. It's useful info :)

I am new to NPM but I coming from other development environments I might be missing something as I do not like some of the workflow have-tos. The simplest example is:

• I want to work with modules, actually creating most of my functionalities in modules.
• Some modules depend on other modules. Of course, this is done using devDependencies or dependencies in the package.json.

The biggest hurdle I am facing is:

• Even though there is the option to reference local modules using the file: mechanism, there is no way (or I haven't been able to find) to use environment variables that would be very helpful for sharing code among different developers that just define their OUR_FRAMEWORK_PATH kind of variable to point to where those modules are.
• Having to remove and re-install modules when they change is a mess. And npm link does not help much either as it is also a mess both in linux based systems (having to change the prefix or work as root? WTF).
  Then again, it might be my inexperience with the platform but I think not being able to easily share modules with other developers without having to do anything (without using a registry or publishing them) and not having an easy way to work with an evolving/under development module is quite surprising/frustrating.