Skip to content

Instantly share code, notes, and snippets.

@dougg0k
Created September 24, 2021 16:27
Show Gist options
  • Save dougg0k/072b1ab4f9d89a3b78217dba6a12ea55 to your computer and use it in GitHub Desktop.
Save dougg0k/072b1ab4f9d89a3b78217dba6a12ea55 to your computer and use it in GitHub Desktop.
Get data from Cloudflare WAF protected url. In this case, was for GraphQL.
const puppeteer = require("puppeteer");
const Xvfb = require("xvfb");
async function getDataFromProtectedUrl({ url, operationName, query, variables }) {
const xvfb = new Xvfb({
silent: true,
xvfb_args: ["-screen", "0", "1280x720x24", "-ac"],
});
xvfb.start((err) => {
if (err) console.error(err);
});
const browser = await puppeteer.launch({
headless: false,
args: [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-gpu",
"--disable-dev-shm-usage",
"--enable-features=NetworkService",
"--display=" + xvfb._display,
],
ignoreHTTPSErrors: true,
dumpio: false,
defaultViewport: null,
});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on("request", (interceptedRequest) => {
interceptedRequest.continue({
headers: {
"Content-Type": "application/json",
Accept: "application/json",
},
method: "GET",
postData: JSON.stringify({
operationName,
query,
variables,
}),
});
});
const response = await page.goto(url, {
waitUntil: "networkidle0",
});
await page.content();
const data = await response.json();
await browser.close();
xvfb.stop();
return data;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment