Skip to content

Instantly share code, notes, and snippets.

@ericdill
Created March 4, 2016 14:45
Show Gist options
  • Save ericdill/4415a840ed6b191f3ac7 to your computer and use it in GitHub Desktop.
Save ericdill/4415a840ed6b191f3ac7 to your computer and use it in GitHub Desktop.
Here's a fun one!
```
$ conda-build-all recipes --upload-channels lightsource2 --matrix-conditions "numpy >=1.10" "python >=3.4" --inspect-channels lightsource2
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ..............
Resolving distributions from 2 recipes...
Computed that there are 4 distributions from the 2 recipes:
Fetching package metadata: ..
Resolved dependencies, will be built in the following order:
pcaspy-0.5.1.post17-1_g05ec8a9_py34 (will be built: False)
pcaspy-0.5.1.post17-1_g05ec8a9_py35 (will be built: False)
pcaspy-0.5.1-py34_2 (will be built: False)
pcaspy-0.5.1-py35_2 (will be built: True)
```
So it looks like `pcaspy-0.5.1-py34_2` will not be built. When
`conda-build-all` gets to this recipe in the build process, I get the
following stack trace:
```
Using Anaconda Cloud api site https://api.anaconda.org
Nothing to be done for pcaspy - it is already on lightsource2/main.
Nothing to be done for pcaspy - it is already on lightsource2/main.
Traceback (most recent call last):
File "/home/edill/mc/bin/conda-build-all", line 9, in <module>
load_entry_point('conda-build-all', 'console_scripts', 'conda-build-all')()
File "/home/edill/dev/conda/conda-build-all/conda_build_all/cli.py", line 85, in main
b.main()
File "/home/edill/dev/conda/conda-build-all/conda_build_all/builder.py", line 213, in main
self.post_build(meta, built_dist_location, was_built)
File "/home/edill/dev/conda/conda-build-all/conda_build_all/builder.py", line 230, in post_build
artefact_destination.make_available(meta, built_dist_location, was_built)
File "/home/edill/dev/conda/conda-build-all/conda_build_all/artefact_destination.py", line 119, in make_available
raise NotImplementedError('cross owner copying not yet implemented.')
NotImplementedError: cross owner copying not yet implemented.
```
Running this again with a pdb.set_trace() at line 119 shows me that this is
indeed the python 3.4 build for tag 0.5.1:
```
(pdb) l
116 elif not just_built:
117 # The distribution already existed, but not under the target owner.
118 if 'http://' in built_dist_path or 'https://' in built_dist_path:
119 pdb.set_trace()
120 -> raise NotImplementedError('cross owner copying not yet implemented.')
(pdb) meta
BakedDistribution({'meta': {'test': {'imports': ['pcaspy'], 'commands': [], 'requires': [], 'files': []}, 'source': {'git_rev': '0.5.1', 'md5': '', 'git_url': 'https://github.com/paulscherrerinstitute/pcaspy', 'svn_rev': '', 'path': '', 'patches': []}, 'package': {'version': '0.5.1', 'name': 'pcaspy'}, 'build': {'features': [], 'string': '', 'entry_points': [], 'pin_depends': '', 'script_env': [], 'number': '2', 'track_features': []}, 'requirements': {'build': ['python', 'setuptools', 'epics-base', 'swig'], 'conflicts': [], 'run': ['python', 'epics-base']}, 'about': {'license': 'BSD', 'home': 'http://code.google.com/p/pcaspy/'}, 'app': {}}, 'path': '/tmp/recipes/pcaspy-0.5.1', 'requirements_path': '/tmp/recipes/pcaspy-0.5.1/requirements.txt', 'meta_path': '/tmp/recipes/pcaspy-0.5.1/meta.yaml'}, (('python', '3.4'),))
```
So, what the heck is going on here? Does this file even exist?
```
$ anaconda show lightsource2/pcaspy/0.5.1
Using Anaconda Cloud api site https://api.anaconda.org
version 0.5.1
+ linux-64/linux-64-pcaspy-0.5.1-py27_0.tar.bz2
+ linux-64/linux-64-pcaspy-0.5.1-py34_2.tar.bz2
None
```
Ok, definitely seems like we have the version that I'm looking for on my
anaconda channel.
```
(pdb) already_on_channel
False
```
Why is `already_on_channel` reporting False? Let's see what file
`inspect_binstar.distribution_exists_on_channel` is looking for...
14: fname = '{}/{}.tar.bz2'.format(conda.config.subdir, metadata.dist())
```
(Pdb) fname = '{}/{}.tar.bz2'.format(conda.config.subdir, meta.dist())
(Pdb) fname
'linux-64/pcaspy-0.5.1-py34_2.tar.bz2'
```
Interesting... looks like the file on anaconda.org/lightsource2 has an extra
"linux-64-" preceding the build name of "pcaspy-0.5.1-py34_2.tar.bz2". It is
worth noting that this file was uploaded ~7 months ago. Maybe conda-build
changed how they make the file name? But conda-build-all correctly reported
that it didn't need to build this file earlier in the build process. Let's go
investigate that one.
Lines 200-202 in `builder.py` seemed to report that I did not need to build
this file.
```
recipes_and_dist_locn = self.find_existing_built_dists(all_distros)
print('Resolved dependencies, will be built in the following order: \n\t{}'.format(
'\n\t'.join(['{} (will be built: {})'.format(meta.dist(), dist_locn is None)
for meta, dist_locn in recipes_and_dist_locn])))
```
Ok, so it looks like the function conda.api.get_index() is returning a reporting that
the file `linux-64-pcaspy-0.5.1-py34_2.tar.bz2` is indeed on the channel `anaconda.org/lightsource2`
```
(Pdb) meta, dist_location = recipes[2]
(Pdb) meta.pkg_fn()
'pcaspy-0.5.1-py34_2.tar.bz2'
(Pdb) pprint(index[meta.pkg_fn()])
{'arch': 'x86_64',
'binstar': {'channel': 'main',
'owner_id': '55cb5cf17d5d4d57053e3ca8',
'package_id': '55cb6f487d5d4d56ff3e3d3b'},
'build': 'py34_2',
'build_number': 2,
'channel': 'https://conda.anaconda.org/t/er-e0184389-c356-4a84-9216-17878c0703d0/lightsource2/linux-64/',
'depends': ['epics-base', 'python 3.4*'],
'license': 'BSD',
'machine': 'x86_64',
'md5': '0cc4a3d5942a2fa31319a4936e05d9a7',
'name': 'pcaspy',
'operatingsystem': 'linux',
'platform': 'linux',
'requires': [],
'size': 430581,
'subdir': 'linux-64',
'target-triplet': 'x86_64-any-linux',
'version': '0.5.1'}
```
I cannot figure out why these files have an extra "linux-64-" prepended. That
being said, this problem has shown me that there is an asymmetry in how
conda-build-all determines if it should build the package by using `conda.api.get_index()`
and then `binstar_cli.distribution()` to check if the specific file exists. Would
you be opposed to a PR that uses `conda.api.get_index()` for both of these checks
instead of using two different mechanisms to check what is (in theory) the
same thing?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment