Skip to content

Instantly share code, notes, and snippets.

@Fusion
Created August 31, 2018 07:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Fusion/8e07337c24ef02f7508a3115088c0298 to your computer and use it in GitHub Desktop.
Save Fusion/8e07337c24ef02f7508a3115088c0298 to your computer and use it in GitHub Desktop.
A curl command that relies on chromium's full page rendering
#!/bin/bash
pos=()
while [[ $# -gt 0 ]]; do
case "$1" in
-h)
echo "Syntax: $0 [-o <file name>][-O] URL"
exit 0
;;
-o)
outname="$2"
shift && shift
;;
-O)
shift
;;
*)
pos+=("$1")
shift
;;
esac
done
set -- "${pos[@]}"
if [ "$outname" == "" ]; then
outname="${1##*/}"
fi
if [ "$1" == "" ]; then
echo 'Error: you need to provide a URL (wrong parameters use?)' >&2
exit 1
fi
url="$1"
tout=""
which timeout &>/dev/null
if [ $? -eq 0 ]; then
tout='timeout 30'
else
which gtimeout &>/dev/null
if [ $? -eq 0 ]; then
tout='gtimeout 30'
fi
fi
which chromium-browser &>/dev/null
if [ $? -eq 0 ]; then
browser='chromium-browser'
else
if [ -f /Applications/Chromium.app/Contents/MacOS/Chromium ]; then
browser='/Applications/Chromium.app/Contents/MacOS/Chromium'
fi
fi
if [ "$browser" == "" ]; then
echo 'Please install chromium browser first.'
exit 1
fi
if [ "$outname" == "" ]; then
$tout $browser --headless --no-sandbox --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537. 36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' --dump-dom $url 2>/dev/null
else
$tout $browser --headless --no-sandbox --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537. 36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' --dump-dom $url > $outname 2>/dev/null
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment