Skip to content

Instantly share code, notes, and snippets.

@jthegedus
Last active April 20, 2022 18:57
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jthegedus/8e820d37e1f3768f991886fb65de154f to your computer and use it in GitHub Desktop.
Save jthegedus/8e820d37e1f3768f991886fb65de154f to your computer and use it in GitHub Desktop.
Next.js static asset hoisting for Firebase Hosting CDN
var shell = require("shelljs");
var nextjsConfig = require("../next.config");
var distDir = nextjsConfig.distDir || ".next";
var BUILD_ID = shell.cat(`${distDir}/BUILD_ID`);
function hoistPages(fileExt, outputPath) {
console.log(
`${distDir}/server/static/${BUILD_ID}/pages/**/*${fileExt} -> ${outputPath}/`
);
shell.mkdir("-p", outputPath);
var match = new RegExp("\\" + `${fileExt}`);
var filesToHoist = shell
.find(`${distDir}/server/static/${BUILD_ID}/pages/`)
.filter(function (file) {
// ensure the file has the required extension and is not a dynamic route (/blog/[pid])
return file.match(match) && file.match(/^((?!\[|\]).)*$/);
});
filesToHoist.forEach((filePath) => {
var outPath = filePath.split("pages/")[1];
if (outPath.includes("/")) {
shell.mkdir(
"-p",
`${outputPath}/${outPath.substring(0, outPath.lastIndexOf("/"))}`
);
}
shell.cp("-f", filePath, `${outputPath}/${outPath}`);
});
}
console.log(
"next export doesn't support getServerSideProps() so we perform our own copy of static assets to prepare our Firebase Hosting upload"
);
console.log(
"Hoist public/ Next.js runtime and optimised chunks, computed .html and .json data\n"
);
console.log("public/ -> out/");
shell.mkdir("-p", "out/");
shell.cp("-Rf", "public/*", "out/");
console.log(`${distDir}/static/ -> out/_next/static/`);
shell.mkdir("-p", "out/_next/static/");
shell.cp("-Rf", `${distDir}/static/`, "out/_next/");
hoistPages(".html", "out");
hoistPages(".json", `out/_next/data/${BUILD_ID}`);
@jthegedus
Copy link
Author

jthegedus commented Oct 6, 2020

In practice with cloud functions (entry level one in specs), I sometimes had to wait 60s to get the first page (the most I could tolerate for a first user load would be 200ms

Cloud Run and Cloud Functions are almost the same product at this point and I wouldn't expect this discrepancy in times, perhaps there's an error in your initial route generation?

I'm not sure Gatsby can render new pages without a deployment. In my Next.js example, if you add blog content to Firebase you don't need to redeploy. You only deploy when your site changes, not when your content changes.

Not sure you can selectively re-render unless using their cloud offering. Otherwise, you can use a webhook from your CMS and trigger a build - it'll eventually get there behind the scenes. If your content changes too often, this could become an issue. Not ideal either.

It's not selective re-rendering. In my example, once it's deployed, you can add a new "blog post" to Firestore and the home page (with client-side request via SWR) and the blog/ page (being SSR) will render the links to all posts, even ones added AFTER deployment, this is the purpose of Incremental Static Regeneration.

When Next.js gets a request to blog/* (a ISR route), if it didn't build a static page for the specific slug at build time, it will run the blog component's GSP function (which in this example fetches data from Firestore) to get the content to pass as props to the page. So no redeployment on content change. It then uses stale-while-revalidate to regenerate the page props (by calling GSP again) once the revalidate time passes.

So if your content is the only thing changing, you don't need to redeploy your site to have it appear.

@akmoulai
Copy link

akmoulai commented Oct 7, 2020

I think we're saying the same thing. The re-deployment was in reference to Gatsby as a fully static site. I just can redeploy the site whenever I change the CMS data, not a big deal with my use case.

What is not great with Next on Firebase - for my use case - is the latency of the first render of each route/page after each deployment with Cloud Run. A Vercel deployment wouldn't have this issue as they sort it out at the Edge cache (I assume they selectively update their Edge CDN from the metadata of the build vs previous build) when Firebase clears the cache when we don't want that to happen without control of what gets cleared or not.

Edit: unrelated to the Firebase/GCP approach, but https://www.gatsbyjs.com/cloud/ offers a way to also selectively update a deployment (which they do on your behalf), but this time only a static one where Vercel offers a mix of possibilities (static, SSG, ISR, and SSR).

@devth
Copy link

devth commented Feb 10, 2021

Excellent thread, thank you for posting this!

@jthegedus what's your latest thinking? Did you end up:

  1. resuming work on hosting Automatic Static Optimization pages?
  2. relying on the SSG hack?
  3. switching to Cloud Run?

Cloud Run looks great - I haven't used it yet.

@jthegedus
Copy link
Author

@devth Glad someone else found this useful! 😅

I am in the process of updating my Next.js example yet again and actually publishing my new blog post.

I am intending on recommending Cloud Run, I would use this personally. It has far fewer concessions than Functions.

I think the SSG hack is sufficient for those who must use Cloud Functions, but the stale-while-revalidate on what should be static content is terribly annoying. I might call for others to help build out Automatic Static Optimization. There may already be a solution contained within https://github.com/serverless-nextjs/serverless-next.js we could extract.

TBH the effort involved circumventing the SSG hack doesn't seem worth it when Cloud Run is so easily usable and available by Firebase users. Just look at the Issues and effort to maintain the aws-serverless plugin linked above.

@devth
Copy link

devth commented Feb 11, 2021

Makes sense! I'm going to look more into Cloud Run when I get a chance. Looking forward to your blog post.

@devth
Copy link

devth commented May 10, 2021

The SSG hack isn't really viable after testing it. Non-cached requests to a cold start app are still insanely slow, like 18 seconds:

Cold

± curl -w "@curl-format.txt" -I "https://dev.converge.is/"
HTTP/2 200
cache-control: s-maxage=31536000, stale-while-revalidate
content-type: text/html; charset=utf-8
etag: "c29f-ueawi3pUlnGm86tZic53Ydjm4r4"
function-execution-id: lw8l0koc8idb
server: Google Frontend
x-cloud-trace-context: e5840d6192499cffb974a6c1f5aa8505;o=1
x-country-code: US
x-powered-by: Next.js
accept-ranges: bytes
date: Mon, 10 May 2021 14:30:17 GMT
x-served-by: cache-sea4446-SEA
x-cache: MISS
x-cache-hits: 0
x-timer: S1620657000.692516,VS0,VE17493
vary: Accept-Encoding,cookie,need-authorization, x-fh-requested-host, accept-encodin
g
content-length: 49823

    time_namelookup:  0.277409s
        time_connect:  0.336877s
     time_appconnect:  0.520884s
    time_pretransfer:  0.521728s
       time_redirect:  0.000000s
  time_starttransfer:  18.091856s
                     ----------
          time_total:  18.092007s

Warm

± curl -w "@curl-format.txt" -I "https://dev.converge.is/"
HTTP/2 200
cache-control: s-maxage=31536000, stale-while-revalidate
content-type: text/html; charset=utf-8
etag: "c29f-ueawi3pUlnGm86tZic53Ydjm4r4"
function-execution-id: lw8l0koc8idb
server: Google Frontend
x-cloud-trace-context: e5840d6192499cffb974a6c1f5aa8505;o=1
x-country-code: US
x-powered-by: Next.js
accept-ranges: bytes
date: Mon, 10 May 2021 14:40:52 GMT
x-served-by: cache-sea4437-SEA
x-cache: HIT
x-cache-hits: 1
x-timer: S1620657653.725299,VS0,VE1
vary: Accept-Encoding,cookie,need-authorization, x-fh-requested-host, accept-encodin
g
content-length: 49823

    time_namelookup:  0.002825s
        time_connect:  0.077373s
     time_appconnect:  0.237598s
    time_pretransfer:  0.237705s
       time_redirect:  0.000000s
  time_starttransfer:  0.321439s
                     ----------
          time_total:  0.321567s

@jthegedus
Copy link
Author

18s!!! :O I haven't seen a cold start over 2s!

@devth
Copy link

devth commented May 11, 2021

Yeah, doesn't seem right. Guessing it's either because I'm not paying enough money yet or I have too many deps. (I'm in the process of removing redux everywhere but not there yet).

At this point I'm dropping SSR almost everywhere, running a next export and trying to cover as many getStaticPaths as I can, while relying on /api routes continuing to be served by Cloud Functions.

Maybe some day I'll migrate to Cloud Run, just don't have bandwidth atm.

package.json deps
  "@date-io/date-fns": "1.x",
  "@google-cloud/bigquery": "^5.5.0",
  "@google/maps": "^1.0.1",
  "@material-ui/core": "^4.11.0",
  "@material-ui/icons": "^4.9.1",
  "@material-ui/lab": "^4.0.0-alpha.56",
  "@material-ui/pickers": "^3.2.10",
  "@mui-treasury/styles": "^0.5.0",
  "@react-hook/window-size": "^3.0.6",
  "autosuggest-highlight": "^3.1.1",
  "axios": "^0.19.2",
  "browser-or-node": "^1.2.1",
  "clsx": "^1.0.4",
  "common-tags": "^1.8.0",
  "core-js": "^3.2.1",
  "date-fns": "^2.4.1",
  "date-fns-tz": "^1.0.12",
  "email-addresses": "^3.1.0",
  "firebase": "^8.2.7",
  "firebase-admin": "^9.5.0",
  "firebase-functions": "^3.13.1",
  "ical-generator": "^1.15.1",
  "integrify": "^3.0.1",
  "isomorphic-unfetch": "^3.0.0",
  "js-cookie": "^2.2.1",
  "lodash": "^4.17.15",
  "next": "^10.0.6",
  "next-cookies": "^2.0.3",
  "next-images": "^1.1.2",
  "next-redux-wrapper": "^6.0.2",
  "node": "^14.2.0",
  "nodemailer": "^6.2.1",
  "nookies": "^2.5.2",
  "papaparse": "^5.2.0",
  "pluralize": "^8.0.0",
  "prop-types": "^15.7.2",
  "react": "^17.0.1",
  "react-dom": "^17.0.1",
  "react-dropzone": "^10.2.2",
  "react-flip-move": "^3.0.3",
  "react-ga": "^2.5.7",
  "react-redux": "^7.2.1",
  "react-redux-firebase": "^3.10.0",
  "react-responsive-modal": "^4.0.1",
  "react-scroll-parallax": "^2.4.0",
  "reactfire": "^3.0.0-rc.0",
  "recompose": "^0.30.0",
  "redux": "^4.0.5",
  "redux-devtools-extension": "^2.13.8",
  "redux-firestore": "^0.13.0",
  "redux-thunk": "^2.3.0",
  "rrule": "^2.6.6",
  "uuid": "^8.1.0"

This post on trimming deps seems relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment