Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save GregHilston/cf7f58d65c474d13ed478d81887c383c to your computer and use it in GitHub Desktop.
Save GregHilston/cf7f58d65c474d13ed478d81887c383c to your computer and use it in GitHub Desktop.
The following is my Metaflow conda environment error when running locally, on OSX, using a local datastore. I do not get any environment issues when running remotely on AWS Batch and get a different set of Conda issues when running locally with a S3 datastore.
CONDA_CHANNELS=anaconda,conda-forge,maciejkula python3 flows/my_flow.py --environment=conda --no-pylint run 130 ?
Metaflow 2.2.5 executing MyFlow for user:ghilston
Validating your flow...
The graph looks good!
cli.py about to create a MetaflowPackage
MetaflowPackage about to init_environment for flow {flow} in environment {environment}
2020-12-11 09:47:30.165 Bootstrapping conda environment...(this could take a few minutes)!
step <bound method MyFlow.start of <__main__.MyFlow object at 0x7fbfb89cff98>> of type <class 'method'>
deco conda<decorated libraries={} python=None disabled=True> of type <class 'metaflow.plugins.conda.conda_step_decorator.CondaStepDecorator'>
step <bound method MyFlow.get_data of <__main__.MyFlow object at 0x7fbfb89cff98>> of type <class 'method'>
deco resources<decorated cpu=4 gpu=0 memory=32768> of type <class 'metaflow.plugins.aws.batch.batch_decorator.ResourcesDecorator'>
deco conda<cli libraries={} python=None disabled=None> of type <class 'metaflow.plugins.conda.conda_step_decorator.CondaStepDecorator'>
Conda ran into an error while setting up environment.:
Step: get_data, Error: UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package python-dateutil conflicts for:
pandas==1.0.0 -> python-dateutil[version='>=2.6.1']
boto3==1.14.47 -> botocore[version='>=1.17.47,<1.18.0'] -> python-dateutil[version='>=2.1,<3.0.0']
Package bzip2 conflicts for:
requests==2.24.0 -> python -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
scikit-learn==0.21.2 -> python[version='>=3.7,<3.8.0a0'] -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
pandas==1.0.0 -> python[version='>=3.7,<3.8.0a0'] -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
click==7.1.2 -> python -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
spotlight==0.1.5 -> python[version='>=3.6,<3.7.0a0'] -> bzip2[version='>=1.0.6,<2.0a0']
coverage==4.5.4 -> python[version='>=3.6,<3.7.0a0'] -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
boto3==1.14.47 -> python -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
Package ca-certificates conflicts for:
spotlight==0.1.5 -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates
requests==2.24.0 -> python -> ca-certificates
python==3.6.3 -> openssl[version='>=1.0.2m,<1.0.3a'] -> ca-certificates
coverage==4.5.4 -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates
click==7.1.2 -> python -> ca-certificates
boto3==1.14.47 -> python -> ca-certificates
Package libcxxabi conflicts for:
python==3.6.3 -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|8.0.1|8.0.0|8.0.0|8.0.0|8.0.0',build='1|2|3|4|0|hebd6815_0|hcfea43d_1']
pandas==1.0.0 -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|8.0.1|8.0.0|8.0.0|8.0.0|8.0.0',build='1|2|3|4|0|hebd6815_0|hcfea43d_1']
scikit-learn==0.21.2 -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|8.0.1|8.0.0|8.0.0|8.0.0|8.0.0',build='1|2|3|4|0|hebd6815_0|hcfea43d_1']
Package urllib3 conflicts for:
spotlight==0.1.5 -> requests -> urllib3[version='>=1.21.1,<1.22|>=1.21.1,<1.23|>=1.21.1,<1.24|>=1.21.1,<1.25|>=1.21.1,<1.26,!=1.25.0,!=1.25.1']
boto3==1.14.47 -> botocore[version='>=1.17.47,<1.18.0'] -> urllib3[version='>=1.20,<1.26']
requests==2.24.0 -> urllib3[version='>=1.21.1,<1.26,!=1.25.0,!=1.25.1']
Package tzdata conflicts for:
click==7.1.2 -> python -> tzdata
boto3==1.14.47 -> python -> tzdata
requests==2.24.0 -> python -> tzdata
Package requests conflicts for:
requests==2.24.0
spotlight==0.1.5 -> requests
Package python conflicts for:
spotlight==0.1.5 -> h5py -> python[version='2.7.|3.5.|3.6.|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|3.4.']
requests==2.24.0 -> python
scikit-learn==0.21.2 -> joblib[version='>=0.11'] -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=3.6|>=3.5,<3.6.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|3.4.']
scikit-learn==0.21.2 -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
spotlight==0.1.5 -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0']
boto3==1.14.47 -> python
pandas==1.0.0 -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0']
click==7.1.2 -> python
boto3==1.14.47 -> jmespath[version='>=0.7.1,<1.0.0'] -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0|3.4.|>=3.8,<3.9.0a0|>=3|>=3.9,<3.10.0a0']
pandas==1.0.0 -> numpy[version='>=1.14.6,<2.0a0'] -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.9,<3.10.0a0|>=3']
requests==2.24.0 -> certifi[version='>=2017.4.17'] -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|>=3.9,<3.10.0a0|<4.0']
coverage==4.5.4 -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0']
python==3.6.3
Package setuptools conflicts for:
scikit-learn==0.21.2 -> joblib[version='>=0.11'] -> setuptools
python==3.6.3 -> pip -> setuptools
Package libgfortran conflicts for:
pandas==1.0.0 -> numpy[version='>=1.14.6,<2.0a0'] -> libgfortran[version='>=3.0,<4.0.0.a0|>=3.0.1|>=3.0.1,<4.0.0.a0']
scikit-learn==0.21.2 -> numpy[version='>=1.11.3,<2.0a0'] -> libgfortran[version='4.|5.|>=3.0,<4.0.0.a0|>=3.0.1|>=3.0.1,<4.0.0.a0|>=4.0.0,<5.0.0.a0']
spotlight==0.1.5 -> numpy[version='>=1.9'] -> libgfortran[version='4.|5.|>=3.0,<4.0.0.a0|>=3.0.1|>=3.0.1,<4.0.0.a0|>=4.0.0,<5.0.0.a0']
Package scikit-learn conflicts for:
scikit-learn==0.21.2
spotlight==0.1.5 -> scikit-learn
Package numpy conflicts for:
spotlight==0.1.5 -> numpy[version='>=1.9']
spotlight==0.1.5 -> h5py -> numpy[version='1.10.|1.11.|1.12.|1.13.|>=1.11.3,<2.0a0|>=1.14.6,<2.0a0|>=1.16.5,<2.0a0|>=1.19.4,<2.0a0|>=1.19.2,<2.0a0|>=1.9.3,<2.0a0|>=1.8|>=1.8,<1.14|>=1.18.5,<2.0a0|>=1.16.6,<2.0a0|>=1.15.1,<2.0a0|>=1.18.1,<2.0a0|>=1.11']
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment