Skip to content

Instantly share code, notes, and snippets.

@pix0r

pix0r/scrape-sitemap.sh

Last active Sep 15, 2020
Star
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Use wget to scrape all URLs from a sitemap.xml Usage: scrape-sitemap.sh http://domain.com/sitemap.xml
Raw
scrape-sitemap.sh
#!/bin/sh
SITEMAP=$1
if [ "$SITEMAP" = "" ]; then
echo "Usage: $0 http://domain.com/sitemap.xml"
exit 1
fi
XML=`wget -O - --quiet $SITEMAP`
URLS=`echo $XML | egrep -o "<loc>[^<>]*</loc>" | sed -e 's:</*loc>::g'`
echo $URLS | tr ' ' '\n' | wget -O /dev/null -i - --wait=1 --random-wait -nv
@hanchiang

This comment has been minimized.

Sign in to view
Copy link

@hanchiang hanchiang commented Dec 1, 2018

Thanks a lot for this bash script! Saved me a lot of head banging 💯
@FlusherDock1

This comment has been minimized.

Sign in to view
Copy link

@FlusherDock1 FlusherDock1 commented Jan 17, 2020

Thanks for the script!
@plittlefield

This comment has been minimized.

Sign in to view
Copy link

@plittlefield plittlefield commented Apr 19, 2020

Wow, just what I was looking for, thanks!
@rabihmb

This comment has been minimized.

Sign in to view
Copy link

@rabihmb rabihmb commented Sep 15, 2020

Is there a vise versa? Can I build a sitemap of a website?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment