zkkmin/bash

Created March 28, 2019 06:28

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/zkkmin/e81eb7ca39146f726b842a7c6c0643be.js"></script>
Save zkkmin/e81eb7ca39146f726b842a7c6c0643be to your computer and use it in GitHub Desktop.

Download ZIP

Raw

bash

curl -s https://www.docdoc.com.sg/medicaltourism_sitemap_profile_1.xml.gz | zcat | xq -r '.urlset.url | map(.loc) | .[]' | sed -e 's/\.com/\.com\.sg/' | xargs curl --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -s -I -L --write-out "============== END ==============\n" | tee medicaltourism_sitemap_profile_1_crawl.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment