Next.js static asset hoisting for Firebase Hosting CDN

next-aso-ssg-export.js

var shell = require("shelljs");
var nextjsConfig = require("../next.config");
var distDir = nextjsConfig.distDir || ".next";
var BUILD_ID = shell.cat(`${distDir}/BUILD_ID`);

function hoistPages(fileExt, outputPath) {
	console.log(
		`${distDir}/server/static/${BUILD_ID}/pages/**/*${fileExt} -> ${outputPath}/`
	);
	shell.mkdir("-p", outputPath);
	var match = new RegExp("\\" + `${fileExt}`);
	var filesToHoist = shell
		.find(`${distDir}/server/static/${BUILD_ID}/pages/`)
		.filter(function (file) {
			// ensure the file has the required extension and is not a dynamic route (/blog/[pid])
			return file.match(match) && file.match(/^((?!\[|\]).)*$/);
		});
	filesToHoist.forEach((filePath) => {
		var outPath = filePath.split("pages/")[1];
		if (outPath.includes("/")) {
			shell.mkdir(
				"-p",
				`${outputPath}/${outPath.substring(0, outPath.lastIndexOf("/"))}`
			);
		}
		shell.cp("-f", filePath, `${outputPath}/${outPath}`);
	});
}

console.log(
	"next export doesn't support getServerSideProps() so we perform our own copy of static assets to prepare our Firebase Hosting upload"
);
console.log(
	"Hoist public/ Next.js runtime and optimised chunks, computed .html and .json data\n"
);

console.log("public/ -> out/");
shell.mkdir("-p", "out/");
shell.cp("-Rf", "public/*", "out/");

console.log(`${distDir}/static/ -> out/_next/static/`);
shell.mkdir("-p", "out/_next/static/");
shell.cp("-Rf", `${distDir}/static/`, "out/_next/");

hoistPages(".html", "out");
hoistPages(".json", `out/_next/data/${BUILD_ID}`);

Goal

Hoist all Automatic Static Optimization pages and any SSG pages to the out/ dir copying across the public/ assets too. All of these assets can be cached in some way on our CDN.

History

Originally requested to have this enabled via ignoring getServerSideProps() pages on next export here vercel/next.js#12313

Details

Originally paired with my Firebase Next.js example.

Firebase

The hoisted pages would need to have their Cache-Controls set via the firebase.json headers configuration - https://firebase.google.com/docs/hosting/full-config#headers

I may end up making this an npm package that consumes the .next/build-manifest.json though the header config would be annoying to implement without editing peoples firebase.json 🤷‍♂️ Just thinking out loud.

old notes from my example's readme

• next export prepares a dir of static content to be uploaded to a CDN. Unfortunately, using getServerSideProps() forces this command to exit. Since we want to produce a CDN-friendly static content directory and have CDN misses rewritten to our Cloud Function, we want to skip GSSP pages and not error on them. To this end, the scripts/export.js script is used to prepare our static content into the out/ directory. This is just a monkey-patch, an official request for next export to ignore GSSP has been made in vercel/next.js#12313
  • NOTE: this is NOT required to use Next.js on Firebase. You can completely remove the node ./scripts/export.js part of the deploy script and the app will work. It just means the first request for each _next/* resource will come from the Cloud Function and then be cached by the CDN, instead of being cached right away.

Not sure if you've found a way around this? It'd be great to leverage Firebase Hosting to do what it's supposed to do 😃 (without having to call the cloud function the first time for static data).

I wish there was an easy way to combine the export with the build and then extract the truly static stuff (html + assets). Could there be a "pre-processing" task/plugin (added to webpack?) to remove GSSP functions (at build time?) and then only run the export?

I ended up deciding this wasn't worth pursuing further as only the first request for a static page hits the function. Next.js applies the correct cache headers so the CDN will cache the static page which all users (except the first) hit.

I'm not familiar with Webpack so can't help you there, sorry.

Sure thing, it makes sense (diminishing returns).

I've been trying to apply https://github.com/jthegedus/firebase-gcp-examples/tree/master/functions-nextjs but I only get the correct headers/CDN cache hit for pages with SSG (haven't tried SSR yet). Static pages however are the problem: they will always call the serverless function and sometimes the execution takes 60s... still trying to figure out why. Prefetch works fine, but hitting page reload (e.g. on the home page) can takes anywhere between 500ms and 60s (seems random). Have you experienced something like this?

I have not experienced behaviour you describe. Attached is a screenshot of my latencies, 99th percentile is 1.00 seconds. My demo is hosted here - https://nextjs-cloudfunctions.web.app/ - feel free to run Lighthouse against it. I see a perf result of 100 on most runs.

Regarding caching: I see pages with max-age or s-max-age coming from the CDN, not my Cloud Functions (more than once). If any asset has the cache control stale-while-revalidate set, then that will send a background request from the client which will hit the Cloud Function. These assets are served from the CDN initially and then the background request will update the cache and client if any changes are found. stale-while-revalidate is a user experience optimisation, not a cost optimisation.

I updated some aspects of my example, so check out the diff of the PR jthegedus/firebase-gcp-examples#141

See Firebase Hosting docs to see how it is supposed to handle max-age - https://firebase.google.com/docs/hosting/manage-cache

First of all, thanks for your reply!

The response time on that site (https://nextjs-cloudfunctions.web.app) is 350ms (exactly the same as I observed on my site). The problem I mentioned doesn't occur on the blog page as I had the correct behaviour: 1st run = function execution, later runs: Firebase hosting cache hits. Except that the blog pages of the demo site seem to still return in 350ms (cache miss...).

For: https://nextjs-cloudfunctions.web.app/ (root url, latest Chrome browser, no plugins) truncated headers:

cache-control: private
content-type: text/html; charset=utf-8
date: Mon, 05 Oct 2020 08:11:49 GMT
etag: "1dce-//rymyO24Ttd1PNVoF5gnsVteGQ"
function-execution-id: rf13n7pyy3hu
status: 304
vary: cookie,need-authorization, x-fh-requested-host, accept-encoding
x-cache: MISS
x-cache-hits: 0
x-country-code: GB
x-powered-by: Next.js
x-served-by: cache-lcy19227-LCY

This is after many refreshes (not navigation events within the site)... Aren't you having the same behaviour at all?

I identified the issue and "fixed" it on my example app, thanks for the push to investigate 🙏

The issue is that during development, we're just using next dev, but on deployment we're using the Next.js Custom Server feature. When we run the build it performs Automatic Static Optimizations and shows the output of:

Page                             Size     First Load JS
┌ ○ /                            1.7 kB         71.5 kB
├ ○ /404                         2.75 kB        61.8 kB
├ ● /about                       1.63 kB        67.3 kB
├ λ /blog                        1.75 kB        71.5 kB
├ ● /blog/[pid]                  2.77 kB        68.4 kB
└ ○ /blog/not-a-post             1.22 kB        66.9 kB
+ First Load JS shared by all    59.1 kB
  ├ chunks/commons.cb8287.js     10.5 kB
  ├ chunks/framework.9ec1f7.js   39.9 kB
  ├ chunks/main.695730.js        6.97 kB
  ├ chunks/pages/_app.4cb0d0.js  1.01 kB
  └ chunks/webpack.e06743.js     751 B

λ  (Server)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
○  (Static)  automatically rendered as static HTML (uses no initial props)
●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)
   (ISR)     incremental static regeneration (uses revalidate in getStaticProps)

With ○ / correctly showing it optimized the page for static serving. This would normally result in Cache-Control headers of public, max-age=31536000, immutable. However, the custom server doesn't respect this, as the docs state, and so it is returned with the Cache-Control set to private.

To get the server to send a page back with the correct Cache-Controls, we must mark it with getStaticProps which the custom server will understand and send the file with Cache-Controls of s-maxage=1, stale-while-revalidate like other GSP pages.

Page                             Size     First Load JS
┌ ● /                            1.72 kB        71.5 kB
├ ○ /404                         2.75 kB        61.8 kB
├ ● /about                       1.63 kB        67.3 kB
├ λ /blog                        1.75 kB        71.5 kB
├ ● /blog/[pid]                  2.77 kB        68.4 kB
└ ○ /blog/not-a-post             1.22 kB        66.9 kB
+ First Load JS shared by all    59.1 kB
  ├ chunks/commons.cb8287.js     10.5 kB
  ├ chunks/framework.9ec1f7.js   39.9 kB
  ├ chunks/main.695730.js        6.97 kB
  ├ chunks/pages/_app.4cb0d0.js  1.01 kB
  └ chunks/webpack.e06743.js     751 B

λ  (Server)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
○  (Static)  automatically rendered as static HTML (uses no initial props)
●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)
   (ISR)     incremental static regeneration (uses revalidate in getStaticProps)

see that the root route is marked as SSG with ● /. (I left /blog/not-a-post as an example to observe the Cache-Control: private issue).

This comes with two downsides however:

• we need to litter our app with GSP calls and manually opt-in to static pages
• our index pages is now .html with an additional .json file which holds empty an props object {"pageProps":{},"__N_SSG":true}

Given this outcome, I may renew the pursuit of this Gist to hoist the Automatic Static Optimization files to Firebase Hosting. However, an easier solution is to use Cloud Run instead of Cloud Functions as you can then just use next start instead of the custom server, skipping all of these issues (I've got a WIP example and blog post coming).

Again, thanks for opening the conversation and pushing me to resolve this properly.

Glad to see we're on the same page (pun intended)! I came to the same conclusion (forcing pages to SSG when they're actually static). The problem with that is that it's wrong at different levels and anti-pattern (a couple reasons you've mentioned, and I've observed on top that the 404 remains slow (can't take SSG...) and the header s-maxage=1, stale-while-revalidate isn't good enough as it will force a call to the Cloud Function for each request (stale while revalidate, which means revalidate (one function call per page load?) in the background...).

The only possibility is to self-host (in Cloud Run, for example) the Next server. This causes problems in terms of latency as well for the first render (although it's less of an issue) but the full Firebase cache gets cleared upon every deployment. If you have a site with many pages (which was my only reason to consider Next over Gatsby) then you'd lose the first page render cache every day if you deploy every day. On top, you're no longer leveraging Firebase as a solution (hosting + cloud function) but now have to deal with Cloud Run (some would be OK with that, it's not necessarily wrong but this pushes me too far!). Also, not sure if that Next server would set the correct cache headers? I don't see why it'd be different than the app instance invoked in the cloud function? I admit I haven't looked at your Cloud Run example however so tempting!

Since my first message, I gave up on Next and went back to Gatsby (1 click rollback & 1 commit revert) - not perfect but at least I get sub 10ms pages and prefetching etc. The site however has to be fully built at deploy time but I don't have "many pages" yet so I'll deal with that in the future (and maybe Gatsby would have solved this problem, their Cloud offering does solve this problem by refreshing only what's needed, probably similarly to what Vercel does on production when it comes to CDN optimisations).

Thanks again for the discussion and sharing your observations :)

Also, not sure if that Next server would set the correct cache headers? I don't see why it'd be different than the app instance invoked in the cloud function?

The difference here is that Cloud Functions requires a "custom server" (server.js) whereas Cloud Run can just run next start which is what Vercel do under the hood. This way you would get the automatic static optimizations.

then you'd lose the first page render cache every day if you deploy every day

Yes, Firebase Hosting performs a full cache purge on deployment to ensure it's serving the latest content.

Something I think you may be disregarding though is that it's only the first request for each page that isn't cached. So only the first user to each page has a sub-optimal experience, it's not the first request for all users.

I gave up on Next and went back to Gatsby

I'm not sure Gatsby can render new pages without a deployment. In my Next.js example, if you add blog content to Firebase you don't need to redeploy. You only deploy when your site changes, not when your content changes.

header s-maxage=1, stale-while-revalidate isn't good enough as it will force a call to the Cloud Function for each request (stale while revalidate, which means revalidate (one function call per page load?) in the background

Yes, stale-while-revalidate is not a cost optimization, but a user experience optimization. Users get the latest content, when content changes, without you needing to redeploy the site (avoiding a CDN cache flush).

---

Next.js will never be as cost efficient as a static site. It's useful when you have a higher frequency of content changes than code changes. This is achieved with stale-while-revalidate which doesn't reduce the backend function requests.

Thanks for raising this and the discussion. Good luck!

The difference here is that Cloud Functions requires a "custom server" (server.js) whereas Cloud Run can just run next start which is what Vercel do under the hood. This way you would get the automatic static optimizations.

Correct. But when they deploy on their platform, they add some extra sauce I believe that relate to their Edge cache that comes on top. Firebase's cache layer would be "standard cache".

Yes, Firebase Hosting performs a full cache purge on deployment to ensure it's serving the latest content.

Something I think you may be disregarding though is that it's only the first request for each page that isn't cached. So only the first user to each page has a sub-optimal experience, it's not the first request for all users.

Agreed in theory. In practice with cloud functions (entry level one in specs), I sometimes had to wait 60s to get the first page (the most I could tolerate for a first user load would be 200ms, no more, and even then it's 20x what I'd like to have). I believe this can be achieved with Cloud Run and get <100ms for the first request so that would work indeed!

I'm not sure Gatsby can render new pages without a deployment. In my Next.js example, if you add blog content to Firebase you don't need to redeploy. You only deploy when your site changes, not when your content changes.

Not sure you can selectively re-render unless using their cloud offering. Otherwise, you can use a webhook from your CMS and trigger a build - it'll eventually get there behind the scenes. If your content changes too often, this could become an issue. Not ideal either.

Yes, stale-while-revalidate is not a cost optimization, but a user experience optimization. Users get the latest content, when content changes, without you needing to redeploy the site (avoiding a CDN cache flush).

Correct, but this would come as a cost that makes Firebase cloud functions not suitable for a production site; you'd almost always would prefer Cloud Run with next start.

Next.js will never be as cost efficient as a static site. It's useful when you have a higher frequency of content changes than code changes. This is achieved with stale-while-revalidate which doesn't reduce the backend function requests.

Thanks for raising this and the discussion. Good luck!

Agreed! Thanks for the replies. I may revisit Next.js with Cloud Run next year (they seem to have some good work/updates in the pipe) if I start pushing past limits of a purely static site.

In practice with cloud functions (entry level one in specs), I sometimes had to wait 60s to get the first page (the most I could tolerate for a first user load would be 200ms

Cloud Run and Cloud Functions are almost the same product at this point and I wouldn't expect this discrepancy in times, perhaps there's an error in your initial route generation?

I'm not sure Gatsby can render new pages without a deployment. In my Next.js example, if you add blog content to Firebase you don't need to redeploy. You only deploy when your site changes, not when your content changes.

Not sure you can selectively re-render unless using their cloud offering. Otherwise, you can use a webhook from your CMS and trigger a build - it'll eventually get there behind the scenes. If your content changes too often, this could become an issue. Not ideal either.

It's not selective re-rendering. In my example, once it's deployed, you can add a new "blog post" to Firestore and the home page (with client-side request via SWR) and the blog/ page (being SSR) will render the links to all posts, even ones added AFTER deployment, this is the purpose of Incremental Static Regeneration.

When Next.js gets a request to blog/* (a ISR route), if it didn't build a static page for the specific slug at build time, it will run the blog component's GSP function (which in this example fetches data from Firestore) to get the content to pass as props to the page. So no redeployment on content change. It then uses stale-while-revalidate to regenerate the page props (by calling GSP again) once the revalidate time passes.

So if your content is the only thing changing, you don't need to redeploy your site to have it appear.

I think we're saying the same thing. The re-deployment was in reference to Gatsby as a fully static site. I just can redeploy the site whenever I change the CMS data, not a big deal with my use case.

What is not great with Next on Firebase - for my use case - is the latency of the first render of each route/page after each deployment with Cloud Run. A Vercel deployment wouldn't have this issue as they sort it out at the Edge cache (I assume they selectively update their Edge CDN from the metadata of the build vs previous build) when Firebase clears the cache when we don't want that to happen without control of what gets cleared or not.

Edit: unrelated to the Firebase/GCP approach, but https://www.gatsbyjs.com/cloud/ offers a way to also selectively update a deployment (which they do on your behalf), but this time only a static one where Vercel offers a mix of possibilities (static, SSG, ISR, and SSR).

Excellent thread, thank you for posting this!

@jthegedus what's your latest thinking? Did you end up:

1. resuming work on hosting Automatic Static Optimization pages?
2. relying on the SSG hack?
3. switching to Cloud Run?

Cloud Run looks great - I haven't used it yet.

@devth Glad someone else found this useful! 😅

I am in the process of updating my Next.js example yet again and actually publishing my new blog post.

I am intending on recommending Cloud Run, I would use this personally. It has far fewer concessions than Functions.

I think the SSG hack is sufficient for those who must use Cloud Functions, but the stale-while-revalidate on what should be static content is terribly annoying. I might call for others to help build out Automatic Static Optimization. There may already be a solution contained within https://github.com/serverless-nextjs/serverless-next.js we could extract.

TBH the effort involved circumventing the SSG hack doesn't seem worth it when Cloud Run is so easily usable and available by Firebase users. Just look at the Issues and effort to maintain the aws-serverless plugin linked above.

Makes sense! I'm going to look more into Cloud Run when I get a chance. Looking forward to your blog post.

The SSG hack isn't really viable after testing it. Non-cached requests to a cold start app are still insanely slow, like 18 seconds:

Cold

± curl -w "@curl-format.txt" -I "https://dev.converge.is/"
HTTP/2 200
cache-control: s-maxage=31536000, stale-while-revalidate
content-type: text/html; charset=utf-8
etag: "c29f-ueawi3pUlnGm86tZic53Ydjm4r4"
function-execution-id: lw8l0koc8idb
server: Google Frontend
x-cloud-trace-context: e5840d6192499cffb974a6c1f5aa8505;o=1
x-country-code: US
x-powered-by: Next.js
accept-ranges: bytes
date: Mon, 10 May 2021 14:30:17 GMT
x-served-by: cache-sea4446-SEA
x-cache: MISS
x-cache-hits: 0
x-timer: S1620657000.692516,VS0,VE17493
vary: Accept-Encoding,cookie,need-authorization, x-fh-requested-host, accept-encodin
g
content-length: 49823

    time_namelookup:  0.277409s
        time_connect:  0.336877s
     time_appconnect:  0.520884s
    time_pretransfer:  0.521728s
       time_redirect:  0.000000s
  time_starttransfer:  18.091856s
                     ----------
          time_total:  18.092007s

Warm

± curl -w "@curl-format.txt" -I "https://dev.converge.is/"
HTTP/2 200
cache-control: s-maxage=31536000, stale-while-revalidate
content-type: text/html; charset=utf-8
etag: "c29f-ueawi3pUlnGm86tZic53Ydjm4r4"
function-execution-id: lw8l0koc8idb
server: Google Frontend
x-cloud-trace-context: e5840d6192499cffb974a6c1f5aa8505;o=1
x-country-code: US
x-powered-by: Next.js
accept-ranges: bytes
date: Mon, 10 May 2021 14:40:52 GMT
x-served-by: cache-sea4437-SEA
x-cache: HIT
x-cache-hits: 1
x-timer: S1620657653.725299,VS0,VE1
vary: Accept-Encoding,cookie,need-authorization, x-fh-requested-host, accept-encodin
g
content-length: 49823

    time_namelookup:  0.002825s
        time_connect:  0.077373s
     time_appconnect:  0.237598s
    time_pretransfer:  0.237705s
       time_redirect:  0.000000s
  time_starttransfer:  0.321439s
                     ----------
          time_total:  0.321567s

18s!!! :O I haven't seen a cold start over 2s!

Yeah, doesn't seem right. Guessing it's either because I'm not paying enough money yet or I have too many deps. (I'm in the process of removing redux everywhere but not there yet).

At this point I'm dropping SSR almost everywhere, running a next export and trying to cover as many getStaticPaths as I can, while relying on /api routes continuing to be served by Cloud Functions.

Maybe some day I'll migrate to Cloud Run, just don't have bandwidth atm.

package.json deps

  "@date-io/date-fns": "1.x",
  "@google-cloud/bigquery": "^5.5.0",
  "@google/maps": "^1.0.1",
  "@material-ui/core": "^4.11.0",
  "@material-ui/icons": "^4.9.1",
  "@material-ui/lab": "^4.0.0-alpha.56",
  "@material-ui/pickers": "^3.2.10",
  "@mui-treasury/styles": "^0.5.0",
  "@react-hook/window-size": "^3.0.6",
  "autosuggest-highlight": "^3.1.1",
  "axios": "^0.19.2",
  "browser-or-node": "^1.2.1",
  "clsx": "^1.0.4",
  "common-tags": "^1.8.0",
  "core-js": "^3.2.1",
  "date-fns": "^2.4.1",
  "date-fns-tz": "^1.0.12",
  "email-addresses": "^3.1.0",
  "firebase": "^8.2.7",
  "firebase-admin": "^9.5.0",
  "firebase-functions": "^3.13.1",
  "ical-generator": "^1.15.1",
  "integrify": "^3.0.1",
  "isomorphic-unfetch": "^3.0.0",
  "js-cookie": "^2.2.1",
  "lodash": "^4.17.15",
  "next": "^10.0.6",
  "next-cookies": "^2.0.3",
  "next-images": "^1.1.2",
  "next-redux-wrapper": "^6.0.2",
  "node": "^14.2.0",
  "nodemailer": "^6.2.1",
  "nookies": "^2.5.2",
  "papaparse": "^5.2.0",
  "pluralize": "^8.0.0",
  "prop-types": "^15.7.2",
  "react": "^17.0.1",
  "react-dom": "^17.0.1",
  "react-dropzone": "^10.2.2",
  "react-flip-move": "^3.0.3",
  "react-ga": "^2.5.7",
  "react-redux": "^7.2.1",
  "react-redux-firebase": "^3.10.0",
  "react-responsive-modal": "^4.0.1",
  "react-scroll-parallax": "^2.4.0",
  "reactfire": "^3.0.0-rc.0",
  "recompose": "^0.30.0",
  "redux": "^4.0.5",
  "redux-devtools-extension": "^2.13.8",
  "redux-firestore": "^0.13.0",
  "redux-thunk": "^2.3.0",
  "rrule": "^2.6.6",
  "uuid": "^8.1.0"

This post on trimming deps seems relevant.