Skip to content

Instantly share code, notes, and snippets.

@CajuCLC

CajuCLC/magento_crawler.sh

Created June 5, 2018 17:42
Show Gist options
  • Save CajuCLC/b7941bcf0b4f33b975d5b3adcd3822d1 to your computer and use it in GitHub Desktop.
Save CajuCLC/b7941bcf0b4f33b975d5b3adcd3822d1 to your computer and use it in GitHub Desktop.
Download ZIP
Magento Crawler
Raw
magento_crawler.sh
#!/bin/bash
URL='domain.com'
wget --quiet http://$URL/sitemap.xml --no-cache --output-document - | egrep -o "http://$URL[^<]+" | while read line; do
time curl -A 'Cache Warmer' -s -L $line > /dev/null 2>&1
echo $line
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment