Skip to content

Instantly share code, notes, and snippets.

@sdaros
Last active January 21, 2021 15:15
Show Gist options
  • Save sdaros/5bd2a84ee9c9fd4471d38171ba09e257 to your computer and use it in GitHub Desktop.
Save sdaros/5bd2a84ee9c9fd4471d38171ba09e257 to your computer and use it in GitHub Desktop.
Troubles with propagated build inputs
# Contents of `default.nix`
{ pkgs ?
import (fetchTarball "http://nixos.org/channels/nixos-20.09/nixexprs.tar.xz")
{ } }:
pkgs.python3Packages.buildPythonApplication {
pname = "load";
src = ./.;
version = "0.3.0";
propagatedBuildInputs = with pkgs.python38Packages; [
python-dotenv
joblib
multiprocess
];
}
# Contents of `load.py`
#!/usr/bin/env python
import multiprocessing
from optparse import OptionParser
from pathlib import Path
from dotenv import dotenv_values
from joblib import Parallel
num_cores = multiprocessing.cpu_count()
parser = OptionParser(
usage="%prog [options] file...\n\nLoad CSV files into a sql database",
version="%prog v0.3.0"
)
parser.add_option("-d", "--drop-schema", dest="drop_schema", action="store_true",
help="Drop existing database schema before create")
parser.add_option("-f", "--environment-file", dest="environment_file",
help="Location of environment file containing credentials",
default=Path('.') / '.env')
parser.add_option("-s", "--db-schema", dest="db_schema", type="string",
help="The database schema to load .csv files into")
parser.add_option("-t", "--type", dest="db_type", type="string", default="postgresql",
help="Type of sql database (postgresql or athena)")
(options, args) = parser.parse_args()
def print_arg(arg):
print(f"arg: {arg}")
def run():
Parallel(n_jobs=num_cores)(delayed(print_arg)(i) for i in args)
if __name__ == "__main__":
run()
# Contents of `setup.py`
from setuptools import setup
setup(
name='load',
version='0.3.0',
py_modules=['load'],
entry_points={
'console_scripts': ['load = load:run']
},
)
@sdaros
Copy link
Author

sdaros commented Jan 21, 2021

The resulting ./result/bin/load keeps returning the following error when I try to run it on the command line:

> ./result/bin/load
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'joblib'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment