Skip to content

Instantly share code, notes, and snippets.

@Porrapat
Created January 9, 2020 11:19
Show Gist options
  • Save Porrapat/5411cbded6e7e65e9f9eb28397b06b9b to your computer and use it in GitHub Desktop.
Save Porrapat/5411cbded6e7e65e9f9eb28397b06b9b to your computer and use it in GitHub Desktop.
# Login To Laravel Web App And Scrape Some Data : Customize for Laravel Web App
# requirement : you must install HTML-XML-utils first by
# sudo apt-get install html-xml-utils
# provide your information here
url=https://xxx.xxx.com
username=YOUR_USERNAME_HERE
password=YOUR_PASSWORD_HERE
result=$(curl -c form-cookie.txt -sL "$url/login")
result=$(echo $result | hxnormalize -x)
result=$(echo $result | hxselect 'input[name=_token]')
# echo $result
token=$(echo $result | grep -oP '(?<=value=")[^"]*(?=")')
echo token is $token > /dev/null
# Login and follow redirect to first protected page.
curl -s -L -b form-cookie.txt -c login-cookie.txt -F "username=$username" -F "password=$password" -F "_token=$token" $url/login > /dev/null
# Scrape Some Protected Page
curl -s -b login-cookie.txt "$url/dashboard" | hxnormalize -x 2>/dev/null | cat > html/dashboard.html
# Scrape Some Protected Page
curl -s -b login-cookie.txt "$url/customer/list" | hxnormalize -x 2>/dev/null | hxselect '.page-content' > html/customer.html
# Scrape Some Protected Page
curl -s -b login-cookie.txt "$url/inventory/product" | hxnormalize -x 2>/dev/null | hxselect '.page-content' > html/product.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment