Skip to content

Instantly share code, notes, and snippets.

@BojoDimov
Created May 25, 2021 21:21
Show Gist options
  • Save BojoDimov/506ddef41b0e65923ae1660b8fe5c393 to your computer and use it in GitHub Desktop.
Save BojoDimov/506ddef41b0e65923ae1660b8fe5c393 to your computer and use it in GitHub Desktop.
Extracts XHR from HAR files
// How will this work
// 1. Stream read the json
// https://www.npmjs.com/package/stream-json
// 2. Extract valuable information
// 3. Rewrite/Rewire request urls
// 4. Put Request/Response pairs in the cache
// https://developer.mozilla.org/en-US/docs/Web/API/Cache/put
// 5. Visualize some of the extracted data
const fs = require('fs');
const path = require('path');
const { chain } = require('stream-chain');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');
const { pick } = require('stream-json/filters/Pick');
const HAR_FILE = path.join(__dirname, '../', 'resources', 'www.facebook.com.har');
const xhrFilter = (data) => data.value._resourceType === 'xhr' ? data.value : undefined
const pipeline = chain([
fs.createReadStream(HAR_FILE),
parser(),
pick({ filter: 'log.entries' }),
streamArray(),
xhrFilter,
data => {
console.log(data.request.url);
return data;
}
]);
let counter = 0;
pipeline.on('data', () => counter++);
pipeline.on('end', () =>
console.log(`Found ${counter} XHR requests.`));
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment