Skip to content

Instantly share code, notes, and snippets.

@matagus
Last active March 9, 2016 16:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save matagus/45ce438090162d2475e2 to your computer and use it in GitHub Desktop.
Save matagus/45ce438090162d2475e2 to your computer and use it in GitHub Desktop.
Task base class for Spotify's Luigi framework to force a task to clean-up its outputs, ie reset and re-run a task. Credits to https://github.com/spotify/luigi/issues/595#issuecomment-194323344
import luigi
class ForceableTask(luigi.Task):
force = luigi.BoolParameter(significant=False, default=False)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# To force execution, we just remove all outputs before `complete()` is called
if self.force is True:
outputs = luigi.task.flatten(self.output())
for out in outputs:
if out.exists():
os.remove(self.output().path)
@matagus
Copy link
Author

matagus commented Mar 9, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment