Pipelines done right
- Diego Lemos @dlresende, Derik Evangelista @kirederik
- this talk is about pipelines and how you can use best practices to write better pipelines
- who here works with concourse? [lots of hands]
I wrote a thing to keep the prison index service prototype up-to-date. It works by polling the RSF for updates, and only downloads new entries.
However, the prison register is in discovery and I’m told it will be deleted and reloaded (possibly a number of times). This means that if I only download entries from the last entry number I saw, I’ll miss the entries which have been completely rewritten.