Skip to content

Instantly share code, notes, and snippets.

@othiym23
Last active February 25, 2016 15:20
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save othiym23/b3d4fe1ded82d0d684cf to your computer and use it in GitHub Desktop.
Save othiym23/b3d4fe1ded82d0d684cf to your computer and use it in GitHub Desktop.

About a year and a half ago, I wrote a "periodic update" of the npm CLI road map. Apparently the period is 19 months! Of the 7 high-level items on the list, we've since addressed 3, which I'm going to call "not bad", given that at least one of the things we shipped, which turned out to be npm@3, is probably the most substantial rewrite of npm's core installer since the very early days of the CLI.

improve npm's tests

npm's developers need to trust its tests. They're the single most important signal that a new version of npm is not going to break users' workflows when a new release is pushed out. Unfortunately, we don't, and once we realized that, it became clear that we could no longer put off working on the test suite until we do trust the tests.

Here's the requirements we identified:

  1. The tests need to be always passing, not just locally, but in Travis, so we have a way of gauging if your patches are safe to merge.
  2. npm's tests are written in a completely different style than the tap tests that have been getting active development, and nobody currently on the team knew what they were testing.
  3. The suite doesn't work on Windows, thus making it very hard for us to figure out how good npm's Windows support is in practice.
  4. We don't have a Windows CI environment set up, so even if the test suite did work on Windows, we don't have automation that will run it for us.
  5. Even though @isaacs and @bcoe put a lot of work into tap and nyc to make them gather coverage information for npm's style of child-process based integration tests, we haven't turned coverage gathering on.
  6. An awful lot of the tests are written as integration tests, which is slow and results in imprecise coverage metrics.
  7. A decent number of the tests, all of which get run by default, rely on internet access to the primary npm registry. This makes it difficult to run npm's test suite as part of Node's own tests, which makes it harder for us to know if prospective changes to Node are going to have knock-on effects on npm.

Fixing this will make life easier for contributors, increase the confidence of both the npm and Node.js teams that we're shipping a quality product, and help us identify many areas for improvement in the CLI. After spending about two months on this project, we're about half way done, and my guess is we'll have reach our current goal – having all the tests passing on Windows, within a Windows CI environment – by the end of March.

solidify the npm@3 installer and fix serious known issues

While we've been working on the test suite, we've maintaining a lightweight version of our existing release process. Over the last year, the level of contribution from the community has gone up sharply, and we want to get your changes integrated and released as quickly as we can. At the same time, we're a small team, and there's only so many different kinds of things that we can do at a time.

As a result, we've been accumulating a list of known issues in the new npm@3 installer that lead to serious failures – either crashes in the install process, or things that result in inconsistent installs. These pretty obviously need to be fixed sooner rather than later, so as soon as we have the test suite where we want it, we're going to switch to addressing all of the issues marked big-bug.

Some or all of these bugs may have been fixed since they were filed, so if you have some spare time and feel like pitching in, verifying whether the older, less recently touched issues still exist, and coming up with repro cases for them, would be very helpful. So would fixing them!

improve CLI performance

npm@3 is slower than it should be. It doesn't have to be, though! The team believes that it can be at least as fast as older versions of npm for all of its most important use cases. Because the installer is such a finely-balanced set of technical tradeoffs designed to balance a large set of edge cases (what do you do when you have bundled dependencies in a package you're installing that don't match the semver declared in the package's package.json? what happens when there are both bundled dependencies and a shrinkwrap?), it's difficult to optimize the installer without making fairly wide-ranging architectural changes.

Given that there's an ecosystem of hundreds of thousands of packages and users, and several hundred versions of npm (almost all of which somebody, somewhere, is still using), we have to move very carefully. Even so, we believe that with some careful work on how caching works and how the CLI interacts with the registry, we can and will make npm much, much faster.

rewrite npm@3's network client / rewrite bits of npm's cache

Once of the most persistent problems that users are having with npm, as installs get larger and more complicated, is that npm is capable of generating enough network traffic to completely lock up many personal firewalls and home broadband connections. A significant portion of npm's support issues are from people grappling with how greedy for bandwidth the CLI is. To remedy this, we intend to build a real download manager into npm, and also continue the work of improving how npm's cache works.

The download manager is intended to make more effective use of available bandwidth by limiting the number of simultaneous connections to specific sites, and to prioritize traffic effectively to make sure that, for example, hosted Git dependencies don't block registry downloads, and vice versa. We also want to simplify the overcomplicated story for proxy configuration, and maybe experiment with proxy autodiscovery on Windows, which would tremendously simplify the process of getting up and running on Windows for new users. Eventually, we may also experiment with bundling package delivery using something like SPDY or HTTP/2, but that will require changes from the registry side that may or may not be in scope for this chunk of work. I've put together a simple specification for what I want the download manager to look like, for those who are interested.

npm's cache is a more or less invisible part of the CLI that is also hugely important. Every package that gets installed into your application has passed through your cache, and without it installs would take orders of magnitude longer. However, it could be doing more. Specifically, we're still working on making the cache content-addressable, which would allow shrinkwrapped installs to pull packages from the cache. A content-addressable cache also supports future work to decrease how often the CLI needs to check whether packages have been updatd on the registry, and makes it easier to support completely disconnected / offline operation for npm.

make npm shrinkwrap a first-class feature within npm

Fun fact: npm shrinkwrap was originally contributed by Dave Pacheco to solve a release management problem at Joyent. Like so many of npm's key features, it was bolted on to the side to solve a specific need. Given the importance of the lockfile concept to many people's understanding of package managers, this may come as a surprise. But it's true!

It's time for shrinkwrap to grow up and take its place as a first-class part of the npm user experience. We've been making steady improvements to it across the npm@2 and npm@3 release cycles, but we want to make working with shrinkwrapped applications as seamless and unsurprising as possible. I'm still working on writing down the concrete proposal for this, so keep a watch out, but I can say that even though npm is probably never going to rely on a lockfile by default, we do want to make it so that the process of working with the lockfile once it's created as familiar and painless as using Gemfile.lock is for Ruby users.

smaller features, some of them probably breaking

I expect the above to keep the team busy for most of 2016.

1. improve npm's support for git

While npm has always supported installing dependencies from Git, working with packages hosted on Git servers isn't nearly as convenient as working with the registry. This is at least somewhat unavoidable, as the registry is explicitly designed for the CLI's use. All the same, we've identified a bunch of smaller changes we can make to npm to make it more reliable with Git dependencies, and to feel more familiar to Bower users, who are used to relying on these features.

2. fix prepublish-on-install

A long time ago, Isaac solved an ongoing problem – integrating transpilation from CoffeeScript into the package lifecycle – with what seemed like a clever hack. That hack was to invoke the lifecycle script for prepublish when npm install was run in the root of a package. That way, any transpiled code would be prepared for use when you downloaded and installed the package to work on it, as well as when you published it to the registry. Unfortunately, while this did address that use case, it was also really confusing to a substantial minority of npm users. A few months ago, I put together a proposal outlining how to deal with the issue, and to do it properly, we're going to need to make at least one breaking change, and have a multi-semver-major release deprecation cycle to completely iron out all the kinks. The work is small, but the impact is significant enough that we need to be careful about rolling it out.

3. improve outdated and update

npm outdated and npm update are underused and poorly understood. In part, this is because they work differently than they do in other package managers due to npm's reliance upon SemVer (this is why, for instance, npm doesn't update outside the SemVer range declared in package.json). In part, though, this is because they're internally inconsistent, and don't always return useful results for linked and hosted Git dependencies.

4. refinements to package scripts

There's been a lot of talk recently about using npm package scripts in lieu of dedicated task runners like Gulp. My belief is that we've semi-inadvertently lucked into a good balance between usefulness and constraint with package scripts (which is another way of saying that there are still many situations in which Gulp and similar tools are going to be a better and simpler solution than using npm alone). We want to keep honing that balance, while adding a few more features to make it easier to write scripts that take full advantage of Node and the host platform, regardless of whether it's Windows or Unix.

5. document npm's internal architecture

Package managers are complicated things. I think one of the biggest impediments to potential CLI contributors is understanding what, how, and why the installer works its magic. Also, it's hard for people to build their own npm registry clients when a lot of the core details about things like bundledDependencies are largely undocumented. To remedy this, we're going to produce human-readable documentation to capture the important algorithms and design considerations for the more complicated bits of npm, like the installer and its cache. (Longer-term, we'd like to have a comprehensive API specification for the protocol used between the CLI and the registry, but that's a cross-company project, given the number of teams involved.)

6. continue to work on improving npm's user experience and diagnostics

This is a great, and easy, way to get involved with improving npm! In fact, the community has already contributed an enormous number of really useful tweaks – some small, some impressively large – to npm's usability and error-reporting. There are still a lot of cases where npm's error handling is frustratingly vague, and we'd also like to do a more comprehensive audit of its user experience (with user testing, and other sources of more objective metrics). npm's user base continues to move further towards newer developers, and npm was written to be an expert's tool. The nice thing about improving the user experience for newcomers is that, if it's done properly, it benefits all users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment