dlstadther

from jenkins/jenkins:2.143
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

dlstadther

Initialize

gitflow	git
git flow init	git init
	git commit --allow-empty -m "Initial commit"
	git checkout -b develop master

Connect to the remote repository

dlstadther

<style>
figure {
  float: center;
  width: 100%;
  text-align: center;
  font-style: italic;
  font-size: smaller;
  text-indent: 0;
  border: thin silver solid;
  margin: 2em;

dlstadther

"""
Test pipeline to test out complex dependency building
http://bionics.it/posts/luigi-tutorial

Concept breaks down when dynamic dependencies are introduced - ListParameters of Tasks cannot be serialized currently

# Note to self:
execute via `~/venv/luigi-3/bin/python3 test_dep.py --local-scheduler TaskA
"""

dlstadther

Keybase proof

I hereby claim:

• I am dlstadther on github.
• I am dlstadther (https://keybase.io/dlstadther) on keybase.
• I have a public key ASCcMLcPfjZ-tVrhnXXdE8z-5dRLknpOg9Y1xscJ-P555Ao

To claim this, I am signing this object:

dlstadther

# import python core modules
import datetime
import logging

# import external modules
import pandas as pd
import requests

# import luigi modules
import luigi

dlstadther

# import luigi modules
import luigi
from luigi.contrib import redshift
from luigi import configuration, s3

# import python core modules
import os
import shutil
import datetime