Skip to content

Instantly share code, notes, and snippets.

@hermo

hermo/README.md Secret

Created May 15, 2019 11:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hermo/45ff25dc6f6d1eca1795035279388023 to your computer and use it in GitHub Desktop.
Save hermo/45ff25dc6f6d1eca1795035279388023 to your computer and use it in GitHub Desktop.
NPM 6.9.0 breaks package-lock.json when updating packages with dependencies to several versions of a package

I recently broke our build after doing some seemingly safe NPM operations and decided to look into what happened.

Software used

$ uname -mrsv
Darwin 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64

$ node --version
v8.16.0

$ npm --versions
{ 'npm-deps-bug': '1.0.0',
  npm: '6.9.0',
  ares: '1.10.1-DEV',
  cldr: '32.0',
  http_parser: '2.8.0',
  icu: '60.1',
  modules: '57',
  napi: '4',
  nghttp2: '1.33.0',
  node: '8.16.0',
  openssl: '1.0.2r',
  tz: '2017c',
  unicode: '10.0',
  uv: '1.23.2',
  v8: '6.2.414.77',
  zlib: '1.2.11' }

Steps to reproduce

Create a new package with npm init and follow these steps.

  1. Add Hapi as a dependency: npm i -S hapi@17.3.1. Hapi has a dependency on catbox-memory@3.x.x
  2. Add catbox-memory@4 as a project dependency: npm i -S catbox-memory@4.0.1
  3. Time has passed, update newer version of hapi: npm i -S hapi@17.8.5
  4. Prepare to publish by running npm ci
  5. Hapi is now broken because it depends on catbox-memory@3: node -e 'require("hapi")' fails with the following error:
module.js:550
    throw err;
    ^

Error: Cannot find module 'big-time'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/npm-deps-bug/node_modules/hapi/node_modules/catbox-memory/lib/index.js:5:17)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)

Investigating the issue

In package-lock.json, hapi 17.8.5 depends on catbox-memory@3.x.x, which is OK. However the entry for catbox-memory@3.1.4 has no dependencies at all.

Running npm view catbox-memory@3.1.4 dependencies shows that it should require: { 'big-time': '2.x.x', boom: '7.x.x', hoek: '6.x.x' }

Running npm ls | grep ERR also detects that the deps for catbox-memory@3.1.4 are not ok:

npm ERR! missing: big-time@2.0.1, required by catbox-memory@3.1.4
npm ERR! missing: isemail@3.x.x, required by joi@14.0.4

There's also something wrong with 'joi' dependency 'isemail' but I will disregard that for this report.

Cause of issue

The issue is related to the fact that catbox-memory@4 is a direct dependency of the app while catbox-memory@3 is a dependency of hapi. When updating hapi and it's deps NPM seems to be looking at the deps for catbox-memory@4 instead of what is requested by hapi (catbox-memory@3.x.x).

Skipping step 2 (npm i -S catbox-memory@4.0.1`) will prevent the issue from appearing. This is easy to verify by editing test.sh and commenting out code for step 2.

Workaround

Running npm i will detect the errors and fix package-lock.json.

Expected result

NPM should detect missing dependencies when updating dependencies and keep package-lock.json in order without having to run npm i to fix things.

Test script

Running test.sh from this Gist in an empty directory will repeat the steps above and stores package.json/package-lock.json for each step.

Sample output from test.sh is also presented in output.log.

Add clean package.json
-- PHASE 1: Simulate development over time by adding/updating deps --
STEP 1: npm i --loglevel error -S hapi@17.3.1
+ hapi@17.3.1
added 56 packages from 5 contributors and audited 124 packages in 1.916s
found 0 vulnerabilities
SUCCESS: require('hapi')
SUCCESS: require('catbox-memory') (version not present, using 3.1.4 from root) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 3.1.4) in project root
STEP 2: npm i --loglevel error -S catbox-memory@4.0.1
+ catbox-memory@4.0.1
added 2 packages, updated 1 package and audited 128 packages in 0.917s
found 0 vulnerabilities
SUCCESS: require('hapi')
SUCCESS: require('catbox-memory') (version 3.1.4) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 4.0.1) in project root
STEP 3: npm i --loglevel error -S hapi@17.8.5
+ hapi@17.8.5
added 28 packages from 4 contributors, removed 52 packages, updated 1 package and audited 23 packages in 0.904s
found 0 vulnerabilities
WARN: Missing deps reported by 'npm ls'
npm ERR! missing: isemail@3.x.x, required by joi@14.0.4
SUCCESS: require('hapi')
SUCCESS: require('catbox-memory') (version 3.1.4) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 4.0.1) in project root
-- PHASE 2: Run 'npm ci' using package-lock.json files from previous steps --
Cleanup: node_modules
Cleanup: package.json
Cleanup: package-lock.json
Add clean package.json
STEP 1: npm ci after 'npm i --loglevel error -S hapi@17.3.1'
added 56 packages in 0.323s
SUCCESS: require('hapi')
SUCCESS: require('catbox-memory') (version not present, using 3.1.4 from root) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 3.1.4) in project root
STEP 2: npm ci after 'npm i --loglevel error -S catbox-memory@4.0.1'
npm WARN prepare removing existing node_modules/ before installation
added 58 packages in 0.395s
SUCCESS: require('hapi')
SUCCESS: require('catbox-memory') (version 3.1.4) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 4.0.1) in project root
STEP 3: npm ci after 'npm i --loglevel error -S hapi@17.8.5'
npm WARN prepare removing existing node_modules/ before installation
added 33 packages in 0.276s
WARN: Missing deps reported by 'npm ls'
npm ERR! missing: big-time@2.0.1, required by catbox-memory@3.1.4
npm ERR! missing: isemail@3.x.x, required by joi@14.0.4
module.js:550
throw err;
^
Error: Cannot find module 'big-time'
at Function.Module._resolveFilename (module.js:548:15)
at Function.Module._load (module.js:475:25)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/npm-deps-bug/node_modules/hapi/node_modules/catbox-memory/lib/index.js:5:17)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
FAIL: require('hapi')
module.js:550
throw err;
^
Error: Cannot find module 'big-time'
at Function.Module._resolveFilename (module.js:548:15)
at Function.Module._load (module.js:475:25)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/npm-deps-bug/node_modules/hapi/node_modules/catbox-memory/lib/index.js:5:17)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
FAIL: require('catbox-memory') (version 3.1.4) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 4.0.1) in project root
STEP 4: Fix deps by running 'npm i'
added 3 packages from 2 contributors and audited 133 packages in 0.625s
found 0 vulnerabilities
SUCCESS: require('hapi')
SUCCESS: require('catbox-memory') (version 3.1.4) in node_modules/hapi
SUCCESS: require('catbox-memory') (version 4.0.1) in project root
Diffs between broken and working package-lock.json
137c137,142
< "integrity": "sha512-1tDnll066au0HXBSDHS/YQ34MQ2omBsmnA9g/jseyq/M3m7UPrajVtPDZK/rXgikSC1dfjo9Pa+kQ1qcyG2d3g=="
---
> "integrity": "sha512-1tDnll066au0HXBSDHS/YQ34MQ2omBsmnA9g/jseyq/M3m7UPrajVtPDZK/rXgikSC1dfjo9Pa+kQ1qcyG2d3g==",
> "requires": {
> "big-time": "2.x.x",
> "boom": "7.x.x",
> "hoek": "6.x.x"
> }
186a192
> "isemail": "3.x.x",
309a316,328
> },
> "isemail": {
> "version": "3.2.0",
> "resolved": "https://registry.npmjs.org/isemail/-/isemail-3.2.0.tgz",
> "integrity": "sha512-zKqkK+O+dGqevc93KNsbZ/TqTUFd46MwWjYOoMrjIMZ51eU7DtQG3Wmd9SQQT7i7RVnuTPEiYEWHU3MSbxC1Tg==",
> "requires": {
> "punycode": "2.x.x"
> }
> },
> "punycode": {
> "version": "2.1.1",
> "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.1.1.tgz",
> "integrity": "sha512-XRsRjdf+j5ml+y/6GKHPZbrF/8p2Yga0JPtdqTIY2Xe5ohJPD9saDJJLPvp9+NSBprVvevdXZybnj2cv8OEd0A=="
#!/usr/bin/env bash
set -eu -o pipefail
project_root="$PWD"
cleanup() {
if [ -d node_modules ]; then
echo "Cleanup: node_modules"
rm -rf node_modules
fi
if [ -f package.json ]; then
echo "Cleanup: package.json"
rm package.json
fi
if [ -f package-lock.json ]; then
echo "Cleanup: package-lock.json"
rm package-lock.json
fi
echo "Add clean package.json"
cat << EOF > package.json
{
"name": "npm-deps-bug",
"version": "1.0.0",
"description": "",
"main": "index.js",
"author": "Mirko Grönroos <mirko@verkkokauppa.com>",
"license": "ISC",
"repository": "none",
"dependencies": { }
}
EOF
}
test_catbox() {
catbox_version_root=$(node -e 'console.log(require("catbox-memory/package").version)')
cd "$project_root/node_modules/hapi"
if [ -f "node_modules/catbox-memory/package.json" ]; then
catbox_version_hapi=$(node -e 'console.log(require("catbox-memory/package").version)')
else
catbox_version_hapi="not present, using $catbox_version_root from root"
fi
if node -e 'require("catbox-memory")'; then
echo "SUCCESS: require('catbox-memory') (version $catbox_version_hapi) in node_modules/hapi"
else
echo "FAIL: require('catbox-memory') (version $catbox_version_hapi) in node_modules/hapi"
fi
cd "$project_root"
if node -e 'require("catbox-memory")'; then
echo "SUCCESS: require('catbox-memory') (version $catbox_version_root) in project root"
else
echo "FAIL: require('catbox-memory') (version $catbox_version_root) in project root"
fi
}
test_hapi() {
if node -e 'require("hapi")'; then
echo "SUCCESS: require('hapi')"
else
echo "FAIL: require('hapi')"
fi
}
list_missing_deps() {
local missing
missing="$(npm ls 2>&1 | grep ERR || true)"
if [ "$missing" != "" ]; then
echo -e "WARN: Missing deps reported by 'npm ls'\n$missing\n"
fi
}
# Phase 1: Simulate normal development
cleanup
# Clean up steps
if [ -d steps ]; then
echo "Cleanup: steps"
rm -rf steps
fi
echo -e "\n-- PHASE 1: Simulate development over time by adding/updating deps --"
# Install some version of Hapi.
# Using --loglevel error, because Hapi recently moved to @hapi/hapi and
# the deprecation notices cause noise irrelevant to this issue.
# It depends on catbox-memory@3, which depends on big-time.
echo -e "\nSTEP 1: npm i --loglevel error -S hapi@17.3.1"
npm i --loglevel error -S hapi@17.3.1
list_missing_deps
# Store package.json and package-lock.json for phase 2
mkdir -p steps/1
cp package*.json steps/1
# require catbox-memory in node_modules/hapi (should work)
# and project root (is not a direct dependency, works by chance)
test_hapi
test_catbox
# Install catbox-memory@4. It does NOT depend on big-time.
echo -e "\nSTEP 2: npm i --loglevel error -S catbox-memory@4.0.1"
npm i --loglevel error -S catbox-memory@4.0.1
list_missing_deps
# Store package.json and package-lock.json for phase 2
mkdir -p steps/2
cp package*.json steps/2
# require catbox-memory in node_modules/hapi (should work)
# and project root (should work, direct dep)
test_hapi
test_catbox
# Update newer Hapi version
# It still depends on catbox-memory@3, which depends on big-time.
echo -e "\nSTEP 3: npm i --loglevel error -S hapi@17.8.5"
npm i --loglevel error -S hapi@17.8.5
list_missing_deps
# Store package.json and package-lock.json for phase 2
mkdir -p steps/3
cp package*.json steps/3
# require catbox-memory in node_modules/hapi (should work)
# and project root (should work, direct dep)
test_hapi
test_catbox
# Phase 2: Run NPM CI after each step
echo -e "\n-- PHASE 2: Run 'npm ci' using package-lock.json files from previous steps --"
cleanup
echo -e "\nSTEP 1: npm ci after 'npm i --loglevel error -S hapi@17.3.1'"
cp steps/1/package*.json .
npm ci
list_missing_deps
# require catbox-memory in node_modules/hapi (should work)
# and project root (is not a direct dependency, works by chance)
test_hapi
test_catbox
echo -e "\nSTEP 2: npm ci after 'npm i --loglevel error -S catbox-memory@4.0.1'"
cp steps/2/package*.json .
npm ci
list_missing_deps
# require catbox-memory in node_modules/hapi (should work)
# and project root (should work, direct dep)
test_hapi
test_catbox
echo -e "\nSTEP 3: npm ci after 'npm i --loglevel error -S hapi@17.8.5'"
cp steps/3/package*.json .
npm ci
# npm ls shows that big-time (and some other deps) are missing
list_missing_deps
# This step fails when trying to require catbox-memory in node_modules_hapi
# because big-time is no longer marked as a dependency for catbox-memory
test_hapi
test_catbox
mkdir -p steps/4
cp package*.json steps/4
echo -e "\nSTEP 4: Fix deps by running 'npm i'"
# Fix package-lock.json by running 'npm i'
npm i
# require catbox-memory in node_modules/hapi (should work)
# and project root (should work, direct dep)
# The require test succeeds again, because running 'npm i' detected and fixed the invalid deps
test_hapi
test_catbox
# Show diff of package*.json
echo -e "\nDiffs between broken and working package-lock.json"
# No difference
diff steps/4/package.json package.json
# big-time and other deps have reappeared as deps for catbox-memory@3
diff steps/4/package-lock.json package-lock.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment