Skip to content

Instantly share code, notes, and snippets.

Website Scrapping Automation

I have recently moved to the Netherlands and due to the social distancing measures, I have been having some issues with the immigration department, the IND, which is closed. Basically you cannot schedule an appointment with them before the 28 of April, and even after that they cannot promise that the system will be back and running, as no one can predict how the policy regarding social distancing will be then.

Their orientation is to check the website every now and then to see anything has changed, and obviously since this is a tedious (however important) task, I decided to automate it. This is a somewhat common problem that also provides many different learning opportunities, so I decided to share my solution with you. The solution uses Python, bs4, libnotify and systemd timers and the code can be found here.

Summary

The idea is very simple, create a scrapper to fetch data from the page and see if the message has changed.