Skip to content

Instantly share code, notes, and snippets.

@kylebarron

kylebarron/scrape-sitemap.sh

Forked from pix0r/scrape-sitemap.sh
Created April 1, 2019 23:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kylebarron/be38d3612113f462850af55953430251 to your computer and use it in GitHub Desktop.
Save kylebarron/be38d3612113f462850af55953430251 to your computer and use it in GitHub Desktop.
Download ZIP
Use wget to scrape all URLs from a sitemap.xml Usage: scrape-sitemap.sh http://domain.com/sitemap.xml
Raw
scrape-sitemap.sh
#!/bin/sh
SITEMAP=$1
if [ "$SITEMAP" = "" ]; then
echo "Usage: $0 http://domain.com/sitemap.xml"
exit 1
fi
XML=`wget -O - --quiet $SITEMAP`
URLS=`echo $XML | egrep -o "<loc>[^<>]*</loc>" | sed -e 's:</*loc>::g'`
echo $URLS | tr ' ' '\n' | wget -O /dev/null -i - --wait=1 --random-wait -nv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment