UPDATED 3.12.2020
The old version of youtube with "disable_polymer" is not working anymore, so the script also stopped working.
Thanks to JanTheDeveloper we have a new working script. Big up!
I just changed the '//span[contains(text(),"Watch later")]',
to '//span[contains(text(),"Remove from")]',
and it should work for any playlist, not just the watch later one. (thanks to hudsonite for the tip)
setInterval(function () {
document.querySelector('#primary button[aria-label="Action menu"]').click();