Reactiflux discussion: handling package dependencies with offline mirrors

reactiflux-package-deps-offline-mirrors.md

[1:11 PM] jswannabe: Question regarding npm. I had a debate last week with someone at work regarding npm update. I don't like running it since the latest updates from 3rd party libs/modules that my webapp might get might break. There is another dev who's on my side. However, there is another dev that likes running npm update. Our boss likes it too and says that we should always get the latest changes from 3rd parties. For me, I'd rather run npm update on a test branch first and if it's been proven not to break our app, that's the time we'll introduce the new version. It's been an ongoing debate for months now, LMAO! How are you folks doing it at work?(edited)
[1:14 PM] acemarke: My personal suggestion would be that you don't just randomly update libs
[1:14 PM] acemarke: I would agree it's better to do it on a test branch first
[1:14 PM] mruzekw: If they respect semver, there shouldn't be breaking changes? I could be wrong
[1:15 PM] acemarke: that's the theory, but in practice, things could always go wrong
[1:15 PM] mruzekw: Yeah, I made that mistake just a couple weeks ago at work >.<
[1:15 PM] acemarke: then again, I completely don't understand the idea of pulling down deps from a network at build time
[1:15 PM] acemarke: which is one of the reasons I like Yarn's "offline mirror" feature
[1:15 PM] mruzekw: You'd rather check in deps?
[1:15 PM] acemarke: yes
[1:16 PM] acemarke: check in the exact package tarballs you depend on
[1:16 PM] acemarke: reproducible builds, no left-pad incidents
[1:16 PM] mruzekw: left-pad?
[1:16 PM] acemarke: oh dear
[1:16 PM] acemarke: please go google that :)
[1:16 PM] mruzekw: lol k
[1:17 PM] mruzekw: Huh I see. Never ran into that
[1:18 PM] vcarl: i didn't either, some of us just got lucky
[1:19 PM] acemarke: and it happened again recently with some other package
[1:19 PM] mruzekw: Why checkin deps when you can check in the build?
[1:19 PM] acemarke: ...
[1:19 PM] acemarke: ???
[1:20 PM] mruzekw: It's an honest question.
[1:20 PM] acemarke: what do you mean by "check in the build"?
[1:20 PM] mruzekw: You build it...and you check it in to git
[1:20 PM] acemarke: uh
[1:20 PM] mruzekw: Along with the src of course
[1:21 PM] acemarke: that's a different topic, and no, that also seems like a bad idea
[1:21 PM] acemarke: let's back up
[1:21 PM] mruzekw: blinks
[1:21 PM] acemarke: first issue: dealing with deps
[1:21 PM] acemarke: it's apparently a common practice to simply declare dependencies in package.json
[1:21 PM] acemarke: and then have your build process do an npm install in a Jenkins build slave or something
[1:21 PM] acemarke: however, to me this has multiple issues
[1:22 PM] acemarke: first, it's likely that you could get newer versions of libs than the ones your devs have been using
[1:22 PM] acemarke: unless you've completely locked the versions
[1:22 PM] acemarke: which means that there might be a change that sneaks in, which could break things
[1:22 PM] acemarke: now as you said, in theory semver would make that unlikely
[1:22 PM] acemarke: but then we get into all kinds of lovely arguments over exactly how to apply semver
[1:23 PM] acemarke: then, there's network issues
[1:23 PM] acemarke: what happens if the network is down at build time?
[1:23 PM] acemarke: oh, and what if the new semver-compatible package is itself broken, like just happened a couple weeks ago?
[1:23 PM] vcarl: > you could get newer versions of libs
(one of the selling points of yarn)
[1:24 PM] acemarke: don't remember what package it was, but I know Henry Zhu (Babel maintainer) referenced it in his recent talk
[1:24 PM] acemarke: eh, fine, hang on
[1:26 PM] acemarke: http://henryzoo.com/maintainer-heal-thyself/assets/player/KeynoteDHTMLPlayer.html#26
[1:26 PM] acemarke: beautifier/js-beautify#1247
GitHub
Installing js-beautify fails · Issue #1247 · beautify-web/js-beautify
Description I am trying to install a library, which is dependant on this one, and starting today (I've tried the last week the last time) I get the following error when installing my dependencies: ...
[1:26 PM] acemarke: that thread gets really ugly really fast
[1:27 PM] acemarke: anyway, point is that installing non-locked versions, over a network, is risky
[1:27 PM] acemarke: so, possible solutions?
[1:27 PM] acemarke: 1) Check in your node_modules
[1:27 PM] acemarke: this is a bad idea for several reasons
[1:27 PM] acemarke: we all know that node_modules can get huge
[1:27 PM] acemarke: hundreds of thousands of files
[1:28 PM] acemarke: a lot of which isn't even entirely relevant to building or deploying the app
[1:28 PM] acemarke: (yes, NPM package maintainers should do a better job trimming junk out of packages)
[1:28 PM] acemarke: also, packages with native dependencies build them into platform-specific binaries
[1:28 PM] acemarke: like, say, node-sass
[1:28 PM] acemarke: which depends on libsass
[1:29 PM] acemarke: if you install node-sass on a Linux machine, it will compile the C code into a Linux-specific binary
[1:29 PM] acemarke: and if you commit that node_modules folder and someone on a Mac or Windows tries to use it, it won't work
[1:29 PM] acemarke: 2) Stand up an internal NPM package server/proxy
[1:29 PM] acemarke: this is a viable option, and may be more useful for larger teams / companies
[1:29 PM] acemarke: But, it's also probably overkill for most teams
[1:30 PM] acemarke: 3) Actually commit just the package tarballs themselves
[1:30 PM] acemarke: This, to me, is the best bang-for-the-buck option for most teams
[1:30 PM] acemarke: In combination with a package lockfile, you get the exact same dependencies being installed every time
[1:30 PM] acemarke: the tarballs themselves are compressed and relatively small
[1:30 PM] acemarke: say, 1000 tarballs at roughly 20MB
[1:31 PM] acemarke: rather than 50K files at 100-200MB depending on what tools / deps you're pulling in
[1:31 PM] acemarke: you can now do installations without having to make network requests, so it's faster, and resilient to network issues
[1:32 PM] acemarke: because the tarballs themselves are platform-independent, devs on different platforms can clone and install the same repo
[1:32 PM] acemarke: so to me, that's by far the best option