Created
March 4, 2016 14:45
-
-
Save ericdill/4415a840ed6b191f3ac7 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Here's a fun one! | |
``` | |
$ conda-build-all recipes --upload-channels lightsource2 --matrix-conditions "numpy >=1.10" "python >=3.4" --inspect-channels lightsource2 | |
Using Anaconda Cloud api site https://api.anaconda.org | |
Fetching package metadata: .............. | |
Resolving distributions from 2 recipes... | |
Computed that there are 4 distributions from the 2 recipes: | |
Fetching package metadata: .. | |
Resolved dependencies, will be built in the following order: | |
pcaspy-0.5.1.post17-1_g05ec8a9_py34 (will be built: False) | |
pcaspy-0.5.1.post17-1_g05ec8a9_py35 (will be built: False) | |
pcaspy-0.5.1-py34_2 (will be built: False) | |
pcaspy-0.5.1-py35_2 (will be built: True) | |
``` | |
So it looks like `pcaspy-0.5.1-py34_2` will not be built. When | |
`conda-build-all` gets to this recipe in the build process, I get the | |
following stack trace: | |
``` | |
Using Anaconda Cloud api site https://api.anaconda.org | |
Nothing to be done for pcaspy - it is already on lightsource2/main. | |
Nothing to be done for pcaspy - it is already on lightsource2/main. | |
Traceback (most recent call last): | |
File "/home/edill/mc/bin/conda-build-all", line 9, in <module> | |
load_entry_point('conda-build-all', 'console_scripts', 'conda-build-all')() | |
File "/home/edill/dev/conda/conda-build-all/conda_build_all/cli.py", line 85, in main | |
b.main() | |
File "/home/edill/dev/conda/conda-build-all/conda_build_all/builder.py", line 213, in main | |
self.post_build(meta, built_dist_location, was_built) | |
File "/home/edill/dev/conda/conda-build-all/conda_build_all/builder.py", line 230, in post_build | |
artefact_destination.make_available(meta, built_dist_location, was_built) | |
File "/home/edill/dev/conda/conda-build-all/conda_build_all/artefact_destination.py", line 119, in make_available | |
raise NotImplementedError('cross owner copying not yet implemented.') | |
NotImplementedError: cross owner copying not yet implemented. | |
``` | |
Running this again with a pdb.set_trace() at line 119 shows me that this is | |
indeed the python 3.4 build for tag 0.5.1: | |
``` | |
(pdb) l | |
116 elif not just_built: | |
117 # The distribution already existed, but not under the target owner. | |
118 if 'http://' in built_dist_path or 'https://' in built_dist_path: | |
119 pdb.set_trace() | |
120 -> raise NotImplementedError('cross owner copying not yet implemented.') | |
(pdb) meta | |
BakedDistribution({'meta': {'test': {'imports': ['pcaspy'], 'commands': [], 'requires': [], 'files': []}, 'source': {'git_rev': '0.5.1', 'md5': '', 'git_url': 'https://github.com/paulscherrerinstitute/pcaspy', 'svn_rev': '', 'path': '', 'patches': []}, 'package': {'version': '0.5.1', 'name': 'pcaspy'}, 'build': {'features': [], 'string': '', 'entry_points': [], 'pin_depends': '', 'script_env': [], 'number': '2', 'track_features': []}, 'requirements': {'build': ['python', 'setuptools', 'epics-base', 'swig'], 'conflicts': [], 'run': ['python', 'epics-base']}, 'about': {'license': 'BSD', 'home': 'http://code.google.com/p/pcaspy/'}, 'app': {}}, 'path': '/tmp/recipes/pcaspy-0.5.1', 'requirements_path': '/tmp/recipes/pcaspy-0.5.1/requirements.txt', 'meta_path': '/tmp/recipes/pcaspy-0.5.1/meta.yaml'}, (('python', '3.4'),)) | |
``` | |
So, what the heck is going on here? Does this file even exist? | |
``` | |
$ anaconda show lightsource2/pcaspy/0.5.1 | |
Using Anaconda Cloud api site https://api.anaconda.org | |
version 0.5.1 | |
+ linux-64/linux-64-pcaspy-0.5.1-py27_0.tar.bz2 | |
+ linux-64/linux-64-pcaspy-0.5.1-py34_2.tar.bz2 | |
None | |
``` | |
Ok, definitely seems like we have the version that I'm looking for on my | |
anaconda channel. | |
``` | |
(pdb) already_on_channel | |
False | |
``` | |
Why is `already_on_channel` reporting False? Let's see what file | |
`inspect_binstar.distribution_exists_on_channel` is looking for... | |
14: fname = '{}/{}.tar.bz2'.format(conda.config.subdir, metadata.dist()) | |
``` | |
(Pdb) fname = '{}/{}.tar.bz2'.format(conda.config.subdir, meta.dist()) | |
(Pdb) fname | |
'linux-64/pcaspy-0.5.1-py34_2.tar.bz2' | |
``` | |
Interesting... looks like the file on anaconda.org/lightsource2 has an extra | |
"linux-64-" preceding the build name of "pcaspy-0.5.1-py34_2.tar.bz2". It is | |
worth noting that this file was uploaded ~7 months ago. Maybe conda-build | |
changed how they make the file name? But conda-build-all correctly reported | |
that it didn't need to build this file earlier in the build process. Let's go | |
investigate that one. | |
Lines 200-202 in `builder.py` seemed to report that I did not need to build | |
this file. | |
``` | |
recipes_and_dist_locn = self.find_existing_built_dists(all_distros) | |
print('Resolved dependencies, will be built in the following order: \n\t{}'.format( | |
'\n\t'.join(['{} (will be built: {})'.format(meta.dist(), dist_locn is None) | |
for meta, dist_locn in recipes_and_dist_locn]))) | |
``` | |
Ok, so it looks like the function conda.api.get_index() is returning a reporting that | |
the file `linux-64-pcaspy-0.5.1-py34_2.tar.bz2` is indeed on the channel `anaconda.org/lightsource2` | |
``` | |
(Pdb) meta, dist_location = recipes[2] | |
(Pdb) meta.pkg_fn() | |
'pcaspy-0.5.1-py34_2.tar.bz2' | |
(Pdb) pprint(index[meta.pkg_fn()]) | |
{'arch': 'x86_64', | |
'binstar': {'channel': 'main', | |
'owner_id': '55cb5cf17d5d4d57053e3ca8', | |
'package_id': '55cb6f487d5d4d56ff3e3d3b'}, | |
'build': 'py34_2', | |
'build_number': 2, | |
'channel': 'https://conda.anaconda.org/t/er-e0184389-c356-4a84-9216-17878c0703d0/lightsource2/linux-64/', | |
'depends': ['epics-base', 'python 3.4*'], | |
'license': 'BSD', | |
'machine': 'x86_64', | |
'md5': '0cc4a3d5942a2fa31319a4936e05d9a7', | |
'name': 'pcaspy', | |
'operatingsystem': 'linux', | |
'platform': 'linux', | |
'requires': [], | |
'size': 430581, | |
'subdir': 'linux-64', | |
'target-triplet': 'x86_64-any-linux', | |
'version': '0.5.1'} | |
``` | |
I cannot figure out why these files have an extra "linux-64-" prepended. That | |
being said, this problem has shown me that there is an asymmetry in how | |
conda-build-all determines if it should build the package by using `conda.api.get_index()` | |
and then `binstar_cli.distribution()` to check if the specific file exists. Would | |
you be opposed to a PR that uses `conda.api.get_index()` for both of these checks | |
instead of using two different mechanisms to check what is (in theory) the | |
same thing? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment