Skip to content

Instantly share code, notes, and snippets.

@cereniyim
Created June 6, 2020 10:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save cereniyim/16e15ad3b70fa21944cfe0ae369ad0fb to your computer and use it in GitHub Desktop.
Save cereniyim/16e15ad3b70fa21944cfe0ae369ad0fb to your computer and use it in GitHub Desktop.
make dataset task
class MakeDatasets(DockerTask):
"""Task to separate training and test sets"""
in_csv = luigi.Parameter(default='/usr/share/data/raw/wine_dataset.csv')
out_dir = luigi.Parameter(default='/usr/share/data/interim/')
flag = luigi.Parameter('.SUCCESS_MakeDatasets')
@property
def image(self):
return f'code-challenge/make-dataset:{VERSION}'
def requires(self):
return DownloadData()
@property
def command(self):
return [
'python', 'make_dataset.py',
'--in-csv', self.in_csv,
'--out-dir', self.out_dir,
'--flag', self.flag
]
def output(self):
return luigi.LocalTarget(
path=str(Path(self.out_dir) / self.flag)
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment