Skip to content

Instantly share code, notes, and snippets.

@msarahan
Last active August 22, 2018 19:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save msarahan/f1ca05bcfb91098eb9eacb5dce559893 to your computer and use it in GitHub Desktop.
Save msarahan/f1ca05bcfb91098eb9eacb5dce559893 to your computer and use it in GitHub Desktop.
Metadata mistakes & how to address them

pin_compatible on numpy's numpy-base dependency

Comes from:

# metapackage for things that don't use numpy's C interface, or things
- name: numpy
  requirements:
    build:
      # for runtime alignment
      - {{ compiler('c') }}
      - {{ compiler('fortran') }}
    host:
      - python
    run:
      - python
      - {{ pin_subpackage('numpy-base') }}
      # openblas or mkl runtime included with run_exports
      - mkl_fft # [blas_impl == 'mkl']
      - mkl_random # [blas_impl == 'mkl' and (not win or vc>=14)]

Results in:

The following NEW packages will be INSTALLED:

numpy:           1.11.3-py37h648b28d_9   defaults
numpy-base:      1.15.0-py37h8a80b8c_9   defaults

Fixed by:

# metapackage for things that don't use numpy's C interface, or things
- name: numpy
  requirements:
    build:
      # for runtime alignment
      - {{ compiler('c') }}
      - {{ compiler('fortran') }}
    host:
      - python
    run:
      - python
      - {{ pin_subpackage('numpy-base', exact=True) }}
      # openblas or mkl runtime included with run_exports
      - mkl_fft # [blas_impl == 'mkl']
      - mkl_random # [blas_impl == 'mkl' and (not win or vc>=14)]

lack of blas metapackage dependency

"mkl-devel-2018.0.3-1.tar.bz2": {
  "build": "1",
  "build_number": 1,
  "depends": [
    "mkl 2018.0.3 1",
    "mkl-include 2018.0.3 1",
  ],
  "md5": "86f9ce6e1eb4ec793215939bf460cffc",
  "name": "mkl-devel",
  "sha256": "e49fdacdbf31d37101b447c16b79bc457c8e3c08216930f3d7baea6a168ac7c8",
  "size": 6190,
  "subdir": "linux-64",
  "timestamp": 1528122418745,
  "version": "2018.0.3"
},

leads to things like:

The following NEW packages will be INSTALLED:

blas:            1.0-mkl                 defaults
ca-certificates: 2018.03.07-0            defaults
certifi:         2018.8.13-py37_0        defaults
cython:          0.28.5-py37h0a44026_0   defaults
intel-openmp:    2018.0.3-0              defaults
libcxx:          4.0.1-h579ed51_0        defaults
libcxxabi:       4.0.1-hebd6815_0        defaults
libedit:         3.1.20170329-hb402a30_2 defaults
libffi:          3.2.1-h475c297_4        defaults
libgfortran:     3.0.1-h93005f0_2        defaults
libopenblas:     0.3.2-hdc02c5d_1        local
mkl:             2018.0.3-1              defaults
ncurses:         6.1-h0a44026_0          defaults
nomkl:           2.0-0                   defaults
numpy:           1.11.3-py37h648b28d_9   defaults
numpy-base:      1.11.3-py37h8a80b8c_9   defaults
openblas-devel:  0.3.2-0                 local
openssl:         1.0.2p-h1de35cc_0       defaults
python:          3.7.0-hc167b69_0        defaults
readline:        7.0-hc1231fa_4          defaults
setuptools:      40.0.0-py37_0           defaults
sqlite:          3.24.0-ha441bb4_0       defaults
tk:              8.6.7-h35a86e2_3        defaults
xz:              5.2.4-h1de35cc_4        defaults
zlib:            1.2.11-hf3cbc9b_2       defaults

presence of legacy packages (free channel)

"numpy-1.11.3-py36_nomkl_0.tar.bz2": {
  "build": "py36_nomkl_0",
  "build_number": 0,
  "date": "2017-01-10",
  "depends": [
    "libgfortran 3.0.0",
    "openblas 0.2.19",
    "python 3.6*"
  ],
  "features": "nomkl",
  "license": "BSD",
  "md5": "16df6508002e78e1639ce0416ca76165",
  "name": "numpy",
  "size": 6996427,
  "version": "1.11.3"
},

Conda will find old packages that lack constraints, and sub these in. It is probably best to omit the "free" channel right now:

conda config --add default_channels https://repo.anaconda.com/pkgs/main conda config --add default_channels https://repo.anaconda.com/pkgs/r

On windows, also add:

conda config --add default_channels https://repo.anaconda.com/pkgs/msys2

Processes to fix

rebuilding packages

  • Does not usually fix the problem. Conda can/will still find old packages and use them when it's not obvious
  • hard to troubleshoot when old packages are still available

hotfixing metadata

  • Requires central administration of index (works for defaults, not conda-forge). Work underway to host static index that can be hotfixed for conda-forge via cloudflare.

      if record['name'] == 'mkl-devel':
          if not any(d.startswith('blas') for d in record['depends']):
              record['depends'].append("blas * mkl")
    

revoking packages

  • when new builds are available, but are not getting picked up due to a past metadata mistake
  • Add an unsatisfiable constraint to effectively remove confusing packages from being picked up
  • Can add a special channel to make that constraint satisfiable, if you really need the old package
  • Implementation: for package something-1.0-1.tar.bz2, place a something-1.0-1.tar.bz2.REVOKED file alongside it, and re-run the index command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment