Skip to content

Instantly share code, notes, and snippets.

Website Scrapping Automation

I have recently moved to the Netherlands and due to the social distancing measures, I have been having some issues with the immigration department, the IND, which is closed. Basically you cannot schedule an appointment with them before the 28 of April, and even after that they cannot promise that the system will be back and running, as no one can predict how the policy regarding social distancing will be then.

Their orientation is to check the website every now and then to see anything has changed, and obviously since this is a tedious (however important) task, I decided to automate it. This is a somewhat common problem that also provides many different learning opportunities, so I decided to share my solution with you. The solution uses Python, bs4, libnotify and systemd timers and the code can be found here.

Summary

The idea is very simple, create a scrapper to fetch data from the page and see if the message has changed.

@RXWatcher
RXWatcher / .env
Created March 26, 2021 08:34 — forked from robertbaker/.env
Traefik 2.0 example config with google forward-auth
UID=1000
TLD=mydomain.com
# ACME
ACME_EMAIL=admin@mydomain.com
CLOUDFLARE_EMAIL=admin@mydomain.com
CLOUDFLARE_API_KEY=xxxx
# OAUTH
OAUTH_CLIENT_ID=xxxx

Keybase proof

I hereby claim:

  • I am rxwatcher on github.
  • I am rxwatcher (https://keybase.io/rxwatcher) on keybase.
  • I have a public key ASDPzGTYftbGe4YIzn1bSF-09QTkEFX45j4bT0TDtb8bCAo

To claim this, I am signing this object:

@RXWatcher
RXWatcher / restic-daily-maintenance.service
Created May 19, 2019 04:46 — forked from MatthewVance/restic-daily-maintenance.service
Restic backup maintenance script, scheduled with a systemd timer rather than cron job.
[Unit]
Description=Runs daily restic maintenance script
[Service]
Type=oneshot
ExecStart=/etc/restic/restic-maintenance-daily.sh
User=restic
@RXWatcher
RXWatcher / restic-exclude-file-linux
Created May 19, 2019 04:46 — forked from MatthewVance/restic-exclude-file-linux
Restic backup exclude file example for Linux.
/dev/*
/home/*/.bash_history
/home/*/.cache/chromium
/home/*/.local/share/Trash
/home/*/.mozilla/firefox/*/Cache
/home/*/Documents/Exclude/*
/lib/modules/*/volatile/.mounted
/media/*
/mnt/*
/proc/*
@RXWatcher
RXWatcher / restic-aws-daily.service
Created May 19, 2019 04:45 — forked from MatthewVance/restic-aws-daily.service
Example script and systemd scheduled service to sync local repo to AWS S3. This is if you prefer to backup locally on your network and sync a copy to S3 rather than use Restic to automatically interface with a S3 based repo.
[Unit]
Description=Runs daily aws sync script
[Service]
Type=oneshot
ExecStart=/etc/restic/aws.sh
@RXWatcher
RXWatcher / restic-snapshots.sh
Created May 19, 2019 04:45 — forked from MatthewVance/restic-snapshots.sh
Example bash shell script to return a list of Restic snapshots.
#!/bin/bash
#: Title : restic
#: Date : July 22 2018
#: Author : Matt Vance
#: Version : 1.0
#: Description : Script to fully sytem backlup
#: License : MIT License (MIT)
# Copyright (C) 2018 Matthew Vance
@RXWatcher
RXWatcher / restic-init.sh
Created May 19, 2019 04:44 — forked from MatthewVance/restic-init.sh
Example bash shell script to initialize a new Restic repo.
#!/bin/bash
#: Title : restic-init-local
#: Date : July 16 2018
#: Author : Matt Vance
#: Version : 1.0
#: Description : Script to create local restic repo
#: License : MIT License (MIT)
# Copyright (C) 2018 Matthew Vance
@RXWatcher
RXWatcher / restic-backup-daily.sh
Created May 19, 2019 04:44 — forked from MatthewVance/restic-backup-daily.sh
Restic daily and hourly backup scripts with systemd timers.
#!/bin/bash
#: Title : restic
#: Date : July 15 2018
#: Author : Matt Vance
#: Version : 1.1
#: Description : Script to fully sytem backlup
#: License : MIT License (MIT)
# Copyright (C) 2019 Matthew Vance
@RXWatcher
RXWatcher / restic-backup-weekly.sh
Created May 19, 2019 04:44 — forked from MatthewVance/restic-backup-weekly.sh
Script to backup a Raspberry Pi weekly to a Restic over the network via REST.
#!/bin/bash
#: Title : restic
#: Date : October 13, 2018
#: Author : Matt Vance
#: Version : 1.0
#: Description : Script to run regular restic backups
#: License : MIT License (MIT)
# Copyright (C) 2018 Matthew Vance