There are lots of ideas on how to best get long term caching into webpack. Here is yet another one.
tl;dr;
- Use NamedModuleIds
- Use NamedChunkIds
- Sprinkle a little bit of magic
But First things first. What stops a default webpack build from being long term cacheable?
Let's just take the walk. Lets set up a small app with webpack an run into all those problems, then try to fix them.
This is out webpack config:
// webpack.config.js
const path = require('path');
const webpack = require('webpack');
module.exports = {
entry: {
main: './src/foo',
},
output: {
path: path.join(__dirname, 'dist'),
filename: '[name].[hash].js',
},
};
and this is what we find in foo.js
// foo.js
import preact from 'preact';
console.log(preact.toString());
Building this gives us a webpack output along these lines:
Asset Size Chunks Chunk Names
main.db3022283e4b37cce06b.js 23.6 kB 0 [emitted] main
[0] ./~/preact/dist/preact.js 20.5 kB {0} [built]
[1] ./src/foo.js 61 bytes {0} [built]
so far so good.
The first thing we may want to do is pull preact out of our main entry file, as it will hopefully change less often as the rest of our app, therefore we add a the CommonsChunkPlugin to our webpack config.
// webpack.config.js
...
plugins: [
new webpack.optimize.CommonsChunkPlugin({
name: ['vendor']
})
]
...
We build out app again and we get an output similar to that:
Asset Size Chunks Chunk Names
main.423221edd7eef26d646e.js 506 bytes 0 [emitted] main
vendor.423221edd7eef26d646e.js 26.7 kB 1 [emitted] vendor
[0] ./~/preact/dist/preact.js 20.5 kB {1} [built]
[1] ./src/foo.js 61 bytes {0} [built]
[2] multi preact 28 bytes {1} [built]
Don't fall asleep. Bear with the fun is about to begin.
You may have noticed, we encountered our first issue here. The main and the vendor chunk are the same. Any change to the main file would now also invalidate our vendor chunk.
To fix this we must switch from hash to chunkhash in our filename:
// webpack.config.js
...
output: {
path: path.join(__dirname, 'dist'),
filename: '[name].[chunkhash].js',
},
...
Running our build again we now see two different hashes.
Asset Size Chunks Chunk Names
main.edc22f71759cbe5336ae.js 528 bytes 0 [emitted] main
vendor.27f1230219fd2a606a54.js 26.7 kB 1 [emitted] vendor
[0] ./~/preact/dist/preact.js 20.5 kB {1} [built]
[1] ./src/foo.js 83 bytes {0} [built]
[2] multi preact 28 bytes {1} [built]
So far so good.
Changing anything in the main chunk should now keep vendor chunk untouched.
Lets add a new bogus line to it:
// foo.js
...
console.log(preact.toString());
console.log("hello world");
Running the build again however shows us that everything is lost, yet again:
Asset Size Chunks Chunk Names
main.91022729b32987083f0d.js 506 bytes 0 [emitted] main
vendor.0da51f051fcf235d7027.js 26.7 kB 1 [emitted] vendor
[0] ./~/preact/dist/preact.js 20.5 kB {1} [built]
[1] ./src/foo.js 61 bytes {0} [built]
[2] multi preact 28 bytes {1} [built]
But why? The problem here is in the detail:
As webpack behaves extracting preact out of the main-chunk also extracts the runtime of webpack into it.
The runtime is the part of webpack that resolves modules at runtime and handles async loading and more. Looking into it, we see a reference to our main chunk in it:
// somewhere in the vendor.0da51f051fcf235d7027.js
...
chunkId + "." + {"0":"91022729b32987083f0d"}[chunkId]
...
Luckily we can fix this.
If you add a CommonsChunkPlugin with the name of a chunk that does not exist as the name of an entry-point webpack will extract the runtime, create a chunk by that name and put the runtime in there.
Sounds magic? Well yeah I guess?
// webpack.config.js
...
plugins: [
...
new webpack.optimize.CommonsChunkPlugin({
name: ['runtime']
})
]
Swinging our magic wand and running our build again now yields this:
Asset Size Chunks Chunk Names
vendor.634878b098e5c599febd.js 20.7 kB 0 [emitted] vendor
main.d59c6ff3126e3943c563.js 538 bytes 1 [emitted] main
runtime.25ce0c546aab71f8eac5.js 5.92 kB 2 [emitted] runtime
[0] ./~/preact/dist/preact.js 20.5 kB {0} [built]
[1] ./src/foo.js 93 bytes {1} [built]
[2] multi preact 28 bytes {0} [built]
Changing something in the main chunk now, will only change the runtime and the main chunk, the vendor chunk will remain untouched.
However the story does not end. As our project grows we add more dependencies:
// foo.js
import bar from './bar';
...
We run our build once again expecting only our build we only expect the main and vendor chunk to change. But you guessed it, thats not what happens:
Asset Size Chunks Chunk Names
main.cec87b856171489c2719.js 811 bytes 0 [emitted] main
vendor.73db375ed475c718163f.js 20.7 kB 1 [emitted] vendor
runtime.93b41beba42ebff23af0.js 5.92 kB 2 [emitted] runtime
[0] ./~/preact/dist/preact.js 20.5 kB {1} [built]
[1] ./src/bar.js 23 bytes {0} [built]
[2] ./src/foo.js 118 bytes {0} [built]
[3] multi preact 28 bytes {1} [built]
Even though nothing in the vendor chunk changed, its hash changed yet again. The reason is once again a detail. Every chunk gets a numerical chunk id. These are given in order, and as the order can change with every new import added, the chunks ids may change with them.
Enter NamedChunksPlugin. This is a recent add to the webpack source (2.4) and allows to have names rather than numbers for our chunks:
// weback.config.js
...
plugins: [
new webpack.NamedChunksPlugin(),
...
...
This will use the unique chunk name instead of its id to identify a chunk.
We can run our build again with and without the addition of bar.js and see the vendor chunk hash stays the same.
Well, except it doesn't. Looking into the two vendor chunks we see something like this:
// old vendor build
...
/***/ }),
/* 1 */,
/* 2 */
/***/ (function(module, exports, __webpack_require__) {
...
// vendor build with new import
...
/***/ }),
/* 1 */,
/* 2 */
/* 3 */
/***/ (function(module, exports, __webpack_require__) {
...
For some reason webpack adds the ids of all the modules that exist to our vendor chunk. Lucky enough there is yet another solution.
Enter NamedModulesPlugin.
// webpack.config.js
...
plugins: [
new NamedModulesPlugin(),
...
...
It does very much the same as our the chunk equivalent. Instead of using ids for our modules it uses a unique path to them.
Our builds now keep the vendor hash always the same:
without bar.js:
Asset Size Chunks Chunk Names
main.5f15e6808c8037c8bdbc.js 628 bytes main [emitted] main
runtime.72ef2fc7d9df236c7f1c.js 5.94 kB runtime [emitted] runtime
vendor.73c86187abcdf9fd7b18.js 20.7 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[0] multi preact 28 bytes {vendor} [built]
[./src/foo.js] ./src/foo.js 121 bytes {main} [built]
with bar.js:
Asset Size Chunks Chunk Names
main.47b747115cd1c2c24b93.js 901 bytes main [emitted] main
runtime.8d94ab27ee79b53aa9a2.js 5.94 kB runtime [emitted] runtime
vendor.73c86187abcdf9fd7b18.js 20.7 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/bar.js] ./src/bar.js 23 bytes {main} [built]
[0] multi preact 28 bytes {vendor} [built]
[./src/foo.js] ./src/foo.js 118 bytes {main} [built]
Is that it? Please tell me thats it, its boring as hell and everything changes all the time.
Well, guess what? haha.
As our app grows it gets heavy. To prevent loading all the codes at once we break it up using some async split points. First we add one
// foo.js
import('./async-bar').then( a => console.log(a));
...
Asset Size Chunks Chunk Names
0.9110a255e8cbd547adc7.js 311 bytes 0 [emitted]
...
and shortly after another
// foo.js
import('./async-bar').then( a => console.log(a));
import('./async-baz').then( a => console.log(a));
...
Asset Size Chunks Chunk Names
0.612f1fa751a3287cf615.js 311 bytes 0 [emitted]
1.9606a5eadde08d70c763.js 311 bytes 1 [emitted]
...
WTF WEBPACK. Why is my async chunk suddenly named differently? And why do they have numbered ids again? I thought cache invalidation was hard, but you invalidate everything all the time :(.
Well turns out that per default the NamedChunkPlugin only handles chunks that are have a name. That is not the case for our async chunks. Stupid lazy OSS devs, pff.
Lets fix this. The NamedChunksPlugin accepts one parameter, this is a function that gets the chunk and must return an id for it. Lets change our plugin to something like this:
// webpack.config.js
plugins: [
...
new webpack.NamedChunksPlugin((chunk) => {
if (chunk.name) {
return chunk.name;
}
return chunk.modules.map(m => path.relative(m.context, m.request)).join("_");
}),
]
Running our build again, we can now add as many async chunks as we want. Previously added ones will stay untouched:
Asset Size Chunks Chunk Names
async-bar.js.06896d922ee7bb8af159.js 324 bytes async-bar.js [emitted]
async-baz.js.fff77d118cec24487e5d.js 324 bytes async-baz.js [emitted]
main.16bb304a9926477df558.js 1.18 kB main [emitted] main
runtime.8f75ffddb5ac5820d4a9.js 6.01 kB runtime [emitted] runtime
vendor.73c86187abcdf9fd7b18.js 20.7 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/async-bar.js] ./src/async-bar.js 41 bytes {async-bar.js} [built]
[./src/async-baz.js] ./src/async-baz.js 41 bytes {async-baz.js} [built]
[./src/bar.js] ./src/bar.js 23 bytes {main} [built]
[0] multi preact 28 bytes {vendor} [built]
[./src/foo.js] ./src/foo.js 218 bytes {main} [built]
Sidenote: Feel free to change the aesthetics of these async chunk names to whatever. Me too lazy.
Okeeeeh is that it now?
Well how about NO
There is this once thing that wont die and we have to support it. Also for some reason we still use it somewhere in our app. And as we dont want to load it twice we want to get it from the global context. But being good devs we also want to make the dependency explizit. So we define our jQuery as an external dependency.
// webpack.config.js
...
externals: {
jquery: 'jQuery'
},
...
What could pissibly go wrong.
before:
Asset Size Chunks Chunk Names
async-bar.js.06896d922ee7bb8af159.js 324 bytes async-bar.js [emitted]
async-baz.js.fff77d118cec24487e5d.js 324 bytes async-baz.js [emitted]
main.e4704e563c227ee88d16.js 1.21 kB main [emitted] main
runtime.6add1d75139c2dd504b1.js 6.01 kB runtime [emitted] runtime
vendor.73c86187abcdf9fd7b18.js 20.7 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/async-bar.js] ./src/async-bar.js 41 bytes {async-bar.js} [built]
[./src/async-baz.js] ./src/async-baz.js 41 bytes {async-baz.js} [built]
[./src/bar.js] ./src/bar.js 23 bytes {main} [built]
[0] multi preact 28 bytes {vendor} [built]
[./src/foo.js] ./src/foo.js 250 bytes {main} [built]
after:
async-bar.js.06896d922ee7bb8af159.js 324 bytes async-bar.js [emitted]
async-baz.js.fff77d118cec24487e5d.js 324 bytes async-baz.js [emitted]
main.f8b6bbf7315113f5b167.js 1.48 kB main [emitted] main
runtime.5544c7f42d217300ca94.js 6.01 kB runtime [emitted] runtime
vendor.3614a776703bbb63977c.js 20.7 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/async-bar.js] ./src/async-bar.js 41 bytes {async-bar.js} [built]
[./src/async-baz.js] ./src/async-baz.js 41 bytes {async-baz.js} [built]
[./src/bar.js] ./src/bar.js 23 bytes {main} [built]
[1] multi preact 28 bytes {vendor} [built]
[./src/foo.js] ./src/foo.js 247 bytes {main} [built]
err yeah, thanks jQuery for killing my vendor chunk? What are you... WHAT?
Well turns out just as for the chunks the NamedModulesPlugin also only works for normal modules. That being said, the external module stole the id 0 from the multi preact module. Lets fix this once and for all.
Besides the normal modules there are a bunch of other modules webpack uses. Not all of them have are covered by the NormalModulesPlugin however. Therefore we need to roll our own (Maybe I should make this a package?):
For now lets add this to our webpack config:
// webpack.config.js
plugins: [
...
{
apply(compiler) {
compiler.plugin("compilation", (compilation) => {
compilation.plugin("before-module-ids", (modules) => {
modules.forEach((module) => {
if (module.id !== null) {
return;
}
module.id = module.identifier();
});
});
});
}
}
]
This works pretty much like the NormalModulesPlugin itself, except it uses the module#identifier method for all modules that do not have an id at this point.
Running our build once more we do get a new vendor cache but this once is persitant no matter how many external modules you may add!
async-bar.js.06896d922ee7bb8af159.js 324 bytes async-bar.js [emitted]
async-baz.js.fff77d118cec24487e5d.js 324 bytes async-baz.js [emitted]
main.8fad02c2ce285a92402a.js 1.52 kB main [emitted] main
runtime.a1ba8a5ef1cf9f245e87.js 6.01 kB runtime [emitted] runtime
vendor.12a52011916cc3cb208c.js 20.8 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/async-bar.js] ./src/async-bar.js 41 bytes {async-bar.js} [built]
[./src/async-baz.js] ./src/async-baz.js 41 bytes {async-baz.js} [built]
[./src/bar.js] ./src/bar.js 23 bytes {main} [built]
[./src/foo.js] ./src/foo.js 247 bytes {main} [built]
[multi preact] multi preact 28 bytes {vendor} [built]
As you can see the id of the multi preact switched from [0] to [multi preact].
So that's it? No. One last step!
The app grows further and we have a kind of second SPA. So we seperate this out into a new entry point and add it to our webpack config:
// webpack.config.js
...
entry: {
main: './src/foo',
other: './src/foo-two',
vendor: ['preact']
},
...
its content is something like this:
// foo-two.js
import bar from './bar';
import preact from 'preact';
import('./async-baz').then( a => console.log(a));
console.log(preact.toString() + "hello world again and again");
Running the build makes us (╯°□°）╯︵ ┻━┻.
The build now yields this:
Asset Size Chunks Chunk Names
async-bar.js.06896d922ee7bb8af159.js 324 bytes async-bar.js [emitted]
async-baz.js.fff77d118cec24487e5d.js 324 bytes async-baz.js [emitted]
main.8407140a96b58ecd9a36.js 1.32 kB main [emitted] main
other.e86bb03c185f1e6bfc33.js 863 bytes other [emitted] other
runtime.da520a6c3b0dab33c4dc.js 6.05 kB runtime [emitted] runtime
vendor.c0d706bb45788764e72e.js 21 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/async-bar.js] ./src/async-bar.js 41 bytes {async-bar.js} [built]
[./src/async-baz.js] ./src/async-baz.js 41 bytes {async-baz.js} [built]
[./src/bar.js] ./src/bar.js 23 bytes {vendor} [built]
[./src/foo-two.js] ./src/foo-two.js 168 bytes {other} [built]
[./src/foo.js] ./src/foo.js 247 bytes {main} [built]
[multi preact] multi preact 28 bytes {vendor} [built]
+ 1 hidden modules
Everything changed. The vendor, the main file, everything, but why?
Turns out that by default our vendor chunk does not only take what we specify but by default also everything we use in both of our entry-points. While this sometimes might be desired to share our common used own rolled modules, these should not end up in the vendor chunk. Maybe like now, we dont even want this at all. So lets do a final fix.
Adding minChunks: Infinity to our vendor commons chunk tells webpack we really only want what we specified in the entry:
new webpack.optimize.CommonsChunkPlugin({
name: 'vendor',
minChunks: Infinity
}),
And if we run our build again:
Asset Size Chunks Chunk Names
async-bar.js.06896d922ee7bb8af159.js 324 bytes async-bar.js [emitted]
async-baz.js.fff77d118cec24487e5d.js 324 bytes async-baz.js [emitted]
main.8fad02c2ce285a92402a.js 1.52 kB main [emitted] main
other.db28c745f0f45fa4d8ed.js 1.06 kB other [emitted] other
runtime.506eb2c73a3b28891b05.js 6.05 kB runtime [emitted] runtime
vendor.12a52011916cc3cb208c.js 20.8 kB vendor [emitted] vendor
[./node_modules/preact/dist/preact.js] ./~/preact/dist/preact.js 20.5 kB {vendor} [built]
[./src/async-bar.js] ./src/async-bar.js 41 bytes {async-bar.js} [built]
[./src/async-baz.js] ./src/async-baz.js 41 bytes {async-baz.js} [built]
[./src/bar.js] ./src/bar.js 23 bytes {main} {other} [built]
[./src/foo-two.js] ./src/foo-two.js 168 bytes {other} [built]
[./src/foo.js] ./src/foo.js 247 bytes {main} [built]
[multi preact] multi preact 28 bytes {vendor} [built]
We have exactly what we want. Our final webpack config looks like this:
const path = require('path');
const webpack = require('webpack');
module.exports = {
entry: {
main: './src/foo',
other: './src/foo-two',
vendor: ['preact']
},
output: {
path: path.join(__dirname, 'dist'),
filename: '[name].[chunkhash].js',
},
externals: {
jquery: 'jQuery'
},
plugins: [
new webpack.NamedModulesPlugin(),
new webpack.NamedChunksPlugin((chunk) => {
if (chunk.name) {
return chunk.name;
}
return chunk.modules.map(m => path.relative(m.context, m.request)).join("_");
}),
new webpack.optimize.CommonsChunkPlugin({
name: 'vendor',
minChunks: Infinity
}),
new webpack.optimize.CommonsChunkPlugin({
name: 'runtime'
}),
{
apply(compiler) {
compiler.plugin("compilation", (compilation) => {
compilation.plugin("before-module-ids", (modules) => {
modules.forEach((module) => {
if (module.id !== null) {
return;
}
module.id = module.identifier();
});
});
});
}
}
]
}
Thats it. At least as far as I was able to think of edge cases. I do hope this helps. If you still find querks, feel free to comment and I will try to find a solution for them.
Now go out into the world and enjoy. And stop using weird hashing plugins that do more harm than good. Please.
Also, if this read got weirder the further you got. Thats not you. Thats my alcohol level. Cheers mate!