Skip to content

Instantly share code, notes, and snippets.

@rastandy
Forked from cvan/HOWTO.md
Created March 17, 2021 13:50
Show Gist options
  • Save rastandy/6bfc7c5eb185cedce287ce3997f02615 to your computer and use it in GitHub Desktop.
Save rastandy/6bfc7c5eb185cedce287ce3997f02615 to your computer and use it in GitHub Desktop.
get all URLs of all network requests made on a web page from the Chrome Dev Tools (from an exported HAR)

Setup

Using Homebrew on Mac OS X:

brew install jq

Add these aliases to your profile (e.g., ~/.zshrc, ~/.bashrc, ~/.profile, etc.):

alias hurlp='pbpaste | jq ".log.entries" | tee >(jq --raw-output "[.[] | .request.url] | sort | unique | .[]")'
alias hurld='pbpaste | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | harurls | tee >(xargs -n 1 curl -O $1)'

Usage

  1. Open the Chrome Developer Tools (Command+Option+i)
  2. Click the Network tab
  3. Reload the page (Command+r)
  4. In the Network tab, hover over any area in the table that is not a link (e.g., the Status column)
  5. Right click (Control+click) to open the context menu and click Copy All as HAR
  6. Open your favourite command-line interface (e.g., Terminal)
  7. Enter hurlp to print the URLs or hurld to download the files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment