Skip to content

Instantly share code, notes, and snippets.

@osule
Last active April 28, 2020 15:55
Show Gist options
  • Save osule/2c12fce91d6a4d83aeba6e28f72246db to your computer and use it in GitHub Desktop.
Save osule/2c12fce91d6a4d83aeba6e28f72246db to your computer and use it in GitHub Desktop.
What Airflow v1.0.3 Scheduler Job is doing

It sets taskinstance state in the following transition

State.SCHEDULED -> State.QUEUED

Task dependencies are not checked until task is running :(

There is no direct way to change delay scheduling of a TaskInstance using a custom dependency that subclasses BaseTiDep unless dep_context.QUEUED_DEPS is patched during run time.

One could attempt throwing a AirflowRescheduleException in TaskInstance.pre_execute but that introduces an unnecessary delay to the task execution that should be avoided.

There is also no surety that invariants of the custom dependency are satisfied in the next run of the TaskInstance.

TaskInstance can only be resheduled once without resorting to Sensor type tasks because of special handling in SchedulerJob for rescheduled cases.

There is the rare possibility of still looking into patching dep_context.QUEUED for adding custom dependency rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment