molotovbliss/gist:6c22ff02b50ccce3a643

Created February 17, 2015 07:43

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/molotovbliss/6c22ff02b50ccce3a643.js"></script>
Save molotovbliss/6c22ff02b50ccce3a643 to your computer and use it in GitHub Desktop.

Download ZIP

sitemap.xml to .txt list of urls one liner, used with siege

Raw

gistfile1.php

php -r '$x=new SimpleXMLElement(file_get_contents("sitemap.xml"));foreach($x->url as $n) echo $n->loc.PHP_EOL;' > urls.txt

Author

molotovbliss commented Aug 19, 2019

With just wget
php -r '$x=new SimpleXMLElement(file_get_contents("sitemap.xml"));foreach($x->url as $n) echo $n->loc.PHP_EOL;' | xargs -n 1 -P 8 wget -q

https://stackoverflow.com/questions/7577615/parallel-wget-in-bash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment