Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Firebase Hosting Fetch All Files

Fetch All Files from Firebase Hosting

This script fetches all of the files from the currently deployed version of a Firebase Hosting site. You must be signed in via the Firebase CLI and have "Site Viewer" permission on the site in question to be able to properly run the script.

Running via NPX

npx https://gist.github.com/mbleigh/9c8680cf319ace2f506f57380da66e7d <site_name>

Where <site_name> is your Firebase Hosting site (e.g. my-site from my-site.firebaseapp.com). The files will be downloaded into a folder <site_name>_<version_id> where the version ID is the current version of your site.

#!/usr/bin/env node
const requireAuth = require('firebase-tools/lib/requireAuth');
const api = require('firebase-tools/lib/api');
const fs = require('fs-extra');
const request = require('request');
if (!process.argv[2]) {
console.error(`
ERROR: Must supply a site name. Usage:
node fetchFiles.js <site_name>`);
process.exit(1);
}
const site = process.argv[2];
async function getLatestVersionName() {
const result = await api.request('GET', `/v1beta1/sites/${site}/releases?pageSize=1`, {
auth: true,
origin: api.hostingApiOrigin,
});
const release = (result.body.releases || [])[0];
if (release) {
return release.version.name;
}
return null;
}
const LIST_PAGE_SIZE = 1000;
async function listFiles(versionName, existing = [], pageToken = null) {
const result = await api.request('GET', `/v1beta1/${versionName}/files?pageSize=${LIST_PAGE_SIZE}${pageToken ? `&pageToken=${pageToken}` : ''}`, {auth: true, origin: api.hostingApiOrigin});
result.body.files.forEach(file => existing.push(file.path));
if (result.body.nextPageToken) {
return await listFiles(versionName, existing, result.body.nextPageToken);
}
return existing;
}
const MAX_FETCHES = 100;
(async function() {
try {
await requireAuth({}, ['https://www.googleapis.com/auth/cloud-platform']);
const v = await getLatestVersionName();
const vid = v.split('/')[v.split('/').length - 1];
const toFetch = await listFiles(v);
const dirName = `${site}_${vid}`;
let fetchesOutstanding = 0;
let fetchCount = 0;
function fetch() {
if (fetchesOutstanding >= MAX_FETCHES) {
return;
} else if (toFetch.length === 0) {
console.log();
console.log("Complete. Fetched", fetchCount, "files.");
return;
}
const f = toFetch.shift();
console.log('Fetching', f);
fetchesOutstanding++;
fetchCount++;
fs.ensureFileSync(dirName + f);
const q = request(`https://${site}.firebaseapp.com${f}`)
const ws = fs.createWriteStream(dirName + f);
q.pipe(ws);
ws.on('finish', () => {
console.log('Fetched ', f);
fetchesOutstanding--;
fetch();
});
}
fetch();
} catch (e) {
console.error("ERROR:", e.stack);
process.exit(1);
}
})();
{
"name": "firebase-hosting-fetch-version",
"version": "0.1.0",
"bin": "./fetchFiles.js",
"dependencies": {
"firebase-tools": "^7.4.0",
"fs-extra": "^8.1.0",
"request": "^2.88.0"
}
}
@ozzyv42

This comment has been minimized.

Copy link

@ozzyv42 ozzyv42 commented Nov 6, 2019

I'm getting ERROR: HTTP Error: 404, Requested entity was not found. Any idea why this could be happening?

@mbleigh

This comment has been minimized.

Copy link
Owner Author

@mbleigh mbleigh commented Nov 6, 2019

One of the API calls is failing. I just updated e.message to e.stack which should print a trace. The most likely would be a) site typo or b) site doesn't have any deployed versions.

@ozzyv42

This comment has been minimized.

Copy link

@ozzyv42 ozzyv42 commented Nov 6, 2019

One of the API calls is failing. I just updated e.message to e.stack which should print a trace. The most likely would be a) site typo or b) site doesn't have any deployed versions.

Got it, thank you!

@ozzyv42

This comment has been minimized.

Copy link

@ozzyv42 ozzyv42 commented Nov 6, 2019

@douwevdijk

This comment has been minimized.

Copy link

@douwevdijk douwevdijk commented Aug 30, 2020

Create thank you. Is there a possibility to exclude folders ( e.g. node_modules )

@GuilhermeCouto

This comment has been minimized.

Copy link

@GuilhermeCouto GuilhermeCouto commented Sep 23, 2020

Why am I getting this error?

npm ERR! code ENOENT
npm ERR! syscall spawn git
npm ERR! path git
npm ERR! errno -4058
npm ERR! enoent Error while executing:
npm ERR! enoent undefined ls-remote -h -t https://gist.github.com/9c8680cf319ace2f506f57380da66e7d.git
npm ERR! enoent
npm ERR! enoent
npm ERR! enoent spawn git ENOENT
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\COUTO-PC\AppData\Roaming\npm-cache_logs\2020-09-23T21_49_34_369Z-debug.log
Install for [ 'git+https://gist.github.com/9c8680cf319ace2f506f57380da66e7d.git' ] failed with code 4294963238

@axelmierczuk

This comment has been minimized.

Copy link

@axelmierczuk axelmierczuk commented Sep 25, 2020

Same as @GuilhermeCouto

@axelmierczuk

This comment has been minimized.

Copy link

@axelmierczuk axelmierczuk commented Sep 25, 2020

@GuilhermeCouto You need to have git installed.

@GuilhermeCouto

This comment has been minimized.

Copy link

@GuilhermeCouto GuilhermeCouto commented Sep 25, 2020

I got it by downloading the files and running:
npm install
node fetchFiles.js <site_name>

@saran141998

This comment has been minimized.

Copy link

@saran141998 saran141998 commented Sep 26, 2020

Why I am getting this?

FirebaseError: HTTP Error: 404, Requested entity was not found.
at module.exports (C:\Projects\React\NodeFetch\node_modules\firebase-tools\lib\responseToError.js:38:12)
at Request._callback (C:\Projects\React\NodeFetch\node_modules\firebase-tools\lib\api.js:40:35)
at Request.self.callback (C:\Projects\React\NodeFetch\node_modules\request\request.js:185:22)
at Request.emit (events.js:315:20)
at Request. (C:\Projects\React\NodeFetch\node_modules\request\request.js:1154:10)
at Request.emit (events.js:315:20)
at IncomingMessage. (C:\Projects\React\NodeFetch\node_modules\request\request.js:1076:12)
at Object.onceWrapper (events.js:421:28)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1220:12)

@anubhavbagri

This comment has been minimized.

Copy link

@anubhavbagri anubhavbagri commented Sep 29, 2020

Why I am getting this?

FirebaseError: HTTP Error: 404, Requested entity was not found.
at module.exports (C:\Projects\React\NodeFetch\node_modules\firebase-tools\lib\responseToError.js:38:12)
at Request._callback (C:\Projects\React\NodeFetch\node_modules\firebase-tools\lib\api.js:40:35)
at Request.self.callback (C:\Projects\React\NodeFetch\node_modules\request\request.js:185:22)
at Request.emit (events.js:315:20)
at Request. (C:\Projects\React\NodeFetch\node_modules\request\request.js:1154:10)
at Request.emit (events.js:315:20)
at IncomingMessage. (C:\Projects\React\NodeFetch\node_modules\request\request.js:1076:12)
at Object.onceWrapper (events.js:421:28)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1220:12)

I'm getting the same error. Could someone help?

@dradeng

This comment has been minimized.

Copy link

@dradeng dradeng commented Oct 3, 2020

Same as the last two guys above

@DangerDay

This comment has been minimized.

Copy link

@DangerDay DangerDay commented Oct 7, 2020

Getting the same error: FirebaseError: HTTP Error: 404, Requested entity was not found.

@antiochdev

This comment has been minimized.

Copy link

@antiochdev antiochdev commented Oct 26, 2020

@GuilhermeCouto followed his lead on this... 👆🏼

@dagalti

This comment has been minimized.

Copy link

@dagalti dagalti commented Nov 26, 2020

@dradgeng, @DangerDay npx https://gist.github.com/mbleigh/9c8680cf319ace2f506f57380da66e7d <site_name>

check <site_name> . thats not with firebaseapp.com .

@sudheerb9

This comment has been minimized.

Copy link

@sudheerb9 sudheerb9 commented Dec 23, 2020

Hi, I tried using this with one of my firebase apps, it resulted in the following error.
(node:7084) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency (Use node --trace-warnings ... to show where the warning was created)
ERROR: FirebaseError: HTTP Error: 404, Requested entity was not found.
at module.exports (C:\Users\sreed\AppData\Roaming\npm-cache_npx\7084\node_modules\firebase-hosting-fetch-version\node_modules\firebase-tools\lib\responseToError.js:38:12)
at Request._callback (C:\Users\sreed\AppData\Roaming\npm-cache_npx\7084\node_modules\firebase-hosting-fetch-version\node_modules\firebase-tools\lib\api.js:40:35)
at Request.self.callback (C:\Users\sreed\AppData\Roaming\npm-cache_npx\7084\node_modules\firebase-hosting-fetch-version\node_modules\request\request.js:185:22)
at Request.emit (events.js:315:20)
at Request. (C:\Users\sreed\AppData\Roaming\npm-cache_npx\7084\node_modules\firebase-hosting-fetch-version\node_modules\request\request.js:1154:10)
at Request.emit (events.js:315:20)
at IncomingMessage. (C:\Users\sreed\AppData\Roaming\npm-cache_npx\7084\node_modules\firebase-hosting-fetch-version\node_modules\request\request.js:1076:12)
at Object.onceWrapper (events.js:421:28)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)

Please help me resolving this.
Thanks in advance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment