Kolonial.no has benefited greatly for building its webshop using Django templates with great speed. However, when scaling an organization for growth and a fluent customer experience – the need for more javascript in the client rises. The biggest issue is that it is not straight-forward to just add more modern javascript to a template rendered app gradually, and a complete single-page application rebuild would be an immense effort.
While we recently also have started transitioning build of new features to use Next.js, we still needed to be able to provide modern stack for our existing site since it will live on for a long while. The approach and tooling we built is a good example of how one could do that – hence this blogpost.
Likewise a few Django JS loaders, we simply use a templatetag to inject a given file based on a key to the template.
{% rollup_bundle 'frontend/feed/feed.tsx' %}
The logic in pythonland is pretty simple.
- We keep a source of truth in using a manifest.json file.
- The manifest is built when bundling, meaning Django does not have to care about anything else.
The manifest looks like this:
https://gist.github.com/840d70859c032bcb1059bf7904baf016
Using that we can pretty much return <script src=f"{manifestValue['esm'}"> in the templatetag render method and have JS inserted. 🎉
The manifest is built when bundling, meaning Django does not have to care about anything else.
Okay, in our case a few more things:
You'll notice in the manifest that we build two different versions, one esm and one systemjs. It is basically the same unbundled code differentiated by modern and legacy support. We do not use the module/nomodule approach since we are using SystemJS – therefore we provide two different script loader functions esmPolyfills and systemJSPolyfills, that exposes injectESModule and injectSystemJSModule on window. Inspired by Philip Walton's blogpost, rollup-starter-code-splitting and systemjs docs
https://gist.github.com/365c2ba43cc570fe86249043e316aa67
In Django we use a simple html_tag helper and do:
https://gist.github.com/6d3f55a5f2fd3590d33228e2892f4078
rendering:
https://gist.github.com/615617773f9196143d8e96f2da0f4f52
We always try to import the es module first, and the logic falls sets a supportDynamicImport flag that is used to evaluate falling back to systemJS or not. The benefit is that we only load the es module bundle as long as it works, otherwise fetching and loading the systemJS bundle. Browser support for es modules and dynamic imports is pretty big with ~92% https://caniuse.com/es6-module-dynamic-import, so we feel we are better off helping e.g. mobile devices not fetch more than necessary rather than optimising for legacy that is often desktop computers.
https://gist.github.com/42f49f8b59c6410cbb5a29aaf34ff627
Again inspired by Philip Walton, this script allows us polyfilling basics and loading SystemJS bundles. It looks daunting, but what is simply does is:
- Check and add basic
Promise,fetchandSymbolpolyfills. - Check the
supportDynamicImportflag set in esmPolyfills – skip if true. - DOM-operation adding a systemJS loader script tag that loads our provided systemJS module on the
loadcallback.
Likewise esmPolyfills we do in Django:
https://gist.github.com/a6c8583375cdc0b91c3fb2fa62ad5b6b
rendering:
https://gist.github.com/9d5caca4c9324bf8cd04590085d77641
A benefit of having control over the injection logic is that we could really simple add a second parameter to the templatetag, giving us a type that can influence how we inject things.
By adding head_js we can inform the templatetag to add JS in a way that is supported in the HTML head, this means we are now capable of leveraging capabilities of the link tag's preload and modulepreload. Having splitted the imports from our bundler into imports and dynamicImports we can then decide if and how we want to load them. Exploiting this – the result is quicker rendering times!
{% rollup_bundle 'frontend/feed/feed.tsx' 'head_js' %}
⏬
https://gist.github.com/b9c61228408cb3c6410bcf9eb62324b2
⏬
https://gist.github.com/700875ee8a7a2f82bd864eaa3908cc33
Ability to inject any kind of javascript for any browser in Django, it only depends on the build-step ensuring that one supports the targeted browsers. We use it to build react-apps big and small with code-splitting across all input entrypoints.
ES modules gives us a lot of benefits, and it is actually the work of javascript standardization over 10 years.
- They are supported by all modern browsers
- Statically analyzable making them more viable for code-splitting
- Smaller footprint through great dead-code elimination
- Can load modules dynamically on-demand, only fetching pieces when you need them
- Enables easier browser caching
- Modern javascript features
and a lot more...
SystemJS is best described by their docs:
SystemJS is a hookable, standards-based module loader. It provides a workflow where code written for production workflows of native ES modules in browsers (like Rollup code-splitting builds), can be transpiled to the System.register module format to work in older browsers that don't support native modules, running almost-native module speeds while supporting top-level await, dynamic import, circular references and live bindings, import.meta.url, module types, import maps, integrity and Content Security Policy with compatibility in older browsers back to IE11.
So how can I start building ES module apps?
The sad thing is that the most popular bundler around, Webpack, does not support outputting ES modules – it only supports them as inputs. Meaning the JS web mostly today consists of commonjs bundles, however it has been requested since 2016 and is now intended that Webpack will start supporting ES module outputs from v5.1.
But there exist bundles that support ES modules today. The most noteworthy is rollup and has up generally only been used for library bundling, pretty much due to lacking code-splitting and developer tooling. Now, we do have code-splitting and decent developer tooling, making it a really viable choice for app development as well. The most recent and prominent alternative is snowpack which provides pretty much all you need (note that it uses rollup under the hood). However for our Django case we needed granular control over our manifest and stuff like that, so we opted for rollup, alongside nollup which supports rollup configs, provides hot-module-reloading and development performance optimizations.
Also, the author of nollup wrote a great blogpost on why he uses rollup instead of webpack, which is a great read: https://medium.com/@PepsRyuu/why-i-use-rollup-and-not-webpack-e3ab163f4fd3
I guess the most prominent and lacking features are:
- Server-side rendering
- No config bundler
However, given that we are incrementally adding modern features in a template rendered site – I feel that we strike an interesting balance between pragmatism and modern tech. Going from pure jQuery sprinkling to adding really modern js capabilities without a full SPA rewrite.
As mentioned we use rollup to bundle our JS, alongside nollup for dev tooling. However we have a few nifty solutions to automize the bundling we want to show you.
In order for us to automatically detect what code that is subject for bundling, we grep for the {% rollup_bundle ... %} tags in our .html templates. We use this output as input to rollup, which naturally will crash if the entrypoint file path does not exist.
https://gist.github.com/b051785967071b2545f9b817af9a9c28
This could probably be written better, but it does the job grepping and outputting an object split by the second argument, e.g. "js" as keys with arrays of entries.
Configurations for rollup are relatively simple, they require input, output and optional plugins, however we do specify a few extra options like sourcemaps and naming structure. Rollup itself supports outputs as a single object, or a list of objects, and in our case where we want to bundle two variants, esm and systemjs, we return a list with different formats.
rollup.config.js:
https://gist.github.com/2aad4c9440d91a7f525b2917baf32009
You might notice that we do provide plugins using a function to avoid duplication, as a lot of the plugins are pretty essential and it quickly becomes a few lines.
https://gist.github.com/cdf6b106cd86b6723b98d66f1270d1e5
I wont dive into details of all these plugins, but their purpose are written in comments inline and are pretty common in the rollup community, with the exception of bable and our manifestPlugin.
Babel is well-known in frontend development, and it is hard to get along without it. For modern things it should be pretty straight forward, target preset esmodules: true and you should be good. But if you care for older browsers like IE11, atleast one mine exists today. The finally prototype does not exist on corejs' es.promise, resulting in red text in the console and white pages in older browsers when using it 😅
Remember I spoke of the manifest Django accesses that is provided from build-step? The manifestPlugin is our custom plugin that serves one purpose. Take the output of rollup builds and place it in an object with the entrypath as key, structured by output-type. This manifest structure is a common pattern in other bundlers as well, but in our situation the flexibility is important as it allows us to do our Django templatetag tricks.
https://gist.github.com/c621dff619894a1250ac4f36da80d159
In contrast to how it looks on the surface, when breaking down this code it is quite simple. There are only a few rollup specifics one need to understand. Rollup exposes a few lifecycle methods for plugins, and we leverage the generateBundle step in that lifecycle. Our plugin receives all bundles that rollup are generating, and we use the different properties like name, facadeModuleId and whether it is an entry or .css file to determine whether to add it to our global manifest object. .mjs is esm, .js is system – naive and simple. Then we emit the file using rollup's API and it is available alongside the bundled js.
A small example that celebrates the release of our long living native apps' home feed on the web! Replacing our static template rendered frontpage with our feed API, written in modern JS using our new tooling.
PICTURE:
One of the main core concerns about adding non server-rendered content on our frontpage is our performance and potential for delayed rendering and page-flickering. Without SSR, we cannot work around using javascript to render our HTML with react, however we can optimize for really fast renders without making it noticable. The idea is basically to avoid the initial API call, perform a server-side render trick by preloading the API and add it as json in the template. Django enables us to do this with it |json and |json_script:"__NAME__" builtins, and then we either add it on the window, or do a simple DOM operation in our JS document.getElementById("__NAME__"). With libraries like react-query one can provide this json as initialData and still have rigged rest API's.
feed.gif
Our total bundle size: 55.6kb gzipped..!
This tooling was the basis for the shopping-assistant I demoed earlier. And it is becoming an internal success as we are already up to 9 apps built internally with this tooling, all of them code-splitted across each other!