Skip to content

Instantly share code, notes, and snippets.

@fortran01

fortran01/scraper.sh

Last active March 31, 2023 18:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fortran01/fb205390f42f364baa5b03ae83454dff to your computer and use it in GitHub Desktop.
Save fortran01/fb205390f42f364baa5b03ae83454dff to your computer and use it in GitHub Desktop.
Download ZIP
Scrape SickKids research site and email
Raw
scraper.sh
(curl 'https://research4kids.research.sickkids.ca/findaclinicalresearchstudy' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Origin: https://research4kids.research.sickkids.ca' \
-H 'Referer: https://research4kids.research.sickkids.ca/findaclinicalresearchstudy' \
--data-raw 'selectedResearchTopics=&researcherId=&departmentId=&searchKeyWord=Allergy&search=Search' \
--compressed > output.html \
&& perl -i -pe 's|(?<=href=")(/[^"]+)|https://research4kids.research.sickkids.ca$1|g' output.html \
&& echo -e "To: your@email.com\nFrom: your@email.com\nSubject: Clinical Research Study Results\nMIME-Version: 1.0\nContent-Type: text/html\n\n$(cat output.html)") \
| /opt/homebrew/bin/msmtp -a gmail your@email.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment