yarn-link-design.md

High-level design

The link step essentially consist in three steps:

• Tree exposition
• Tree serialization
• Package builds

The first step is when Yarn communicates to the various linkers to let them know which packages in the dependency tree belong to them, and what are their dependencies.

The second step is when Yarn orders linkers to persist their state on the filesystem. We need to do this now, before running the builds, because the builds will likely need to run yarn install - which requires the linker states to have been serialized.

The third step is when Yarn runs the build scripts that have been requested by the linkers. It first checks whether the dependency tree for each of the packages that must be rebuilt changed, then sorts them so that the first rebuilt packages are those that other packages depend upon.

Cross-tree linking

The cross-tree linking is based on an assumption that every host language has a "PnP-like" linker, where folders are mapped to a set of dependencies. Because linkers don't know (and shouldn't know) what's the language of the guest dependent, they don't need to know their locator; only their path matters.

For example, in the case of a JS package X depending on a Python package Y, the Python linker would be told that Y will have to be made available to something located in <location advertised by the JS linker for X>. It will then, in its own "PnP map", store this information so that when X does yarn python ./my-script.py from the advertised location, then my-script.py will be able to access Y.

Problem: this implies that the package locations are known before the tree serialization step comes, which isn't true for some linkers such as the node_modules linker. For this reason those linkers cannot support cross-tree linking.

Multiple views of a same package

In the case of a node_modules it becomes even more tricky because a single package may have to be stored multiple times on the filesystem if it cannot be hoisted further. The plot thickens when on top of that it has build scripts; now those build scripts need to be run in every folder when the package will be stored.