sigh_modules.md

So here's (part of) the problem:

• When you use require in Node, Node will look at the closest node_modules and if it can’t find it there look in the parent dir for a node_modules install that has it, until it reaches /.

• When you ‘link’ npm/yarn do nothing else but create a symlink in your node_modules to the package that presumable lives outside of the tree your app lives in (eg it’s a sibling).

• Now when your linked package tries to require something and it can’t find it in its own node_modules where should it look? The (real) parent dir isn’t the app, but that’s the default behaviour anyways, so it will fail

• “But how could that package not be in the linked package’s node_modules?!” you ask, which is a very valid question.

  Seeing as you’re working on the linked package, you presumably have ran yarn install. However, now this package’s node_modules contains versions of packages that might normally (unlinked) have been hoisted up to the app’s node_modules.

  This becomes problematic when code in packages expect to be able to compare things. For instance, consider package “foo” to export a singleton object X, it reasonably has some code that compares objects to X, but all of a sudden that comparison fails when you would expect it to succeed. This is because now you have 2 versions of X in the runtime. 💥

  This is exacerbated in the case of a linked package, because it might have devDependencies that also exist in the app’s deps, which leads to even more occasions for breakage.

  The reason one might observe these problems to only occur ‘sometimes’ when linking, is probably because none of these preconditions were met. Lucky you, for the time being.

• One way to sometimes deal with this issue is to tell Node to preserve symlinks when looking in its parent directory. This is what the --preserve-symlinks option is for. When using this, you would delete node_modules from the linked package and Node should be able to find all the dependencies in the app’s node_modules. (Funnily enough there’s a separate option for the ‘main’ script, because why not?)

  However, this approach is rather naive and only works if your package has only dependencies that could be hoisted when integrated in the app. Because if your package does have dependencies that cannot be hoisted, you cannot just delete the package’s node_modules dir entirely, oh no, now you need to find out which deps those are and only leave those in the package’s node_modules. Fun times indeed.

• TL;DR, this is why we can’t have fun things and each time you link a package a kitten may die, but you may not even notice, Schrödinger something. The reason something like ‘yarn workspaces’ fix this, is because it moves the app and packages you would otherwise link into a shared parent and hoists all dependencies there. Fixed, because the naive traversing of parents will now always end up in the same parent dir.

  To conclude, it is all basically pretty much file-system semantics, which can be difficult enough to grasp, but all the abstractions may make it seem more magical than that, so it’s good to understand how fundamentally this is all just about traversing directories.

More reading:

• nodejs/node#3402
• nodejs/node#5950
• nodejs/node#6537
• nodejs/node#9673