Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@jperkin

jperkin/pkgin-refresh.md

Last active April 9, 2018 01:09
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jperkin/5584bef9bebc4c3917fdfc30d9db3bd8 to your computer and use it in GitHub Desktop.
Save jperkin/5584bef9bebc4c3917fdfc30d9db3bd8 to your computer and use it in GitHub Desktop.
Download ZIP
pkgin refresh changes for testing
Raw
pkgin-refresh.md

Introduction

pkgsrc does things a bit different to most other package managers. When a package is rebuilt using our bulk build mechanism, every other package that depends on it is also rebuilt. So for example, if we upgrade OpenSSL from 1.0.2m to 1.0.2n then every package that uses OpenSSL (and every package that depends on those packages, and so on..) is rebuilt.

We do this to ensure consistency and avoid any nasty surprises. Just because an upstream thinks the ABI is identical it doesn't mean that is true, and you cannot rely on this behaviour. By rebuilding everything that could be affected we have confidence that at least the changes have gone through a build test, and we're not just hoping for the best.

This has served us well in the past and shown up issues that other package managers have been unaware of.

Issues

This approach does mean a few things have to be done correctly for it to all work in harmony.

If we do detect an ABI change, then we mark it and do a recursive revision bump of all packages (the "nb" at the end of a package version increases). This ensures that end users doing an upgrade receive new packages for everything affected.

The problem is if we don't detect an ABI change, or if it's marked incorrectly (for example if we backport an OpenSSL update, but forget to backport all of the "nb" PKGREVISION bumps too).

This results in issues like TritonDataCenter/pkgsrc#80. libevent was bumped, every package that depends on it (including tmux) was rebuilt, but the tmux version number didn't change. The result is that users performing a full-upgrade received a new libevent but not a rebuilt tmux, and the existing tmux can no longer find the previous libevent library version.

Fixes

Whilst we endeavour to ensure PKGREVISION bumps are handled correctly, we're not perfect, and occasionally they will be forgotten or performed incorrectly. Sometimes this can be very difficult to detect, for example when a Unix .so version remains the same, but building the same package on macOS results in a different .dylib library name.

We will continue to add checks to ensure these are handled correctly from the pkgsrc side, but we are also proposing a change to pkgin to always pull in a refreshed package, i.e. one that has the same version number but has been rebuilt.

The way to do this is to compare the BUILD_DATE metadata that we have on the local machine with the one available in the remote repository. If they are different, we will fetch and refresh the package. This ensures consistency with the remote repository and hopefully avoids all of the aforementioned issues.

Changes

While implementing the refresh behaviour I also took the opportunity to update the pkgin output to make it more consistent and clearer about what it is planning to do.

Example

Performing a simple install using the current pkgin on my macOS desktop:

$ pkgin in lynx
calculating dependencies... done.

nothing to upgrade.
2 packages to be installed (5943K to download, 16M to install):

gettext-tools-0.19.8.1nb1 lynx-2.8.8.2nb7

proceed ? [Y/n] n

A simple install using the patched pkgin results in much cleaner output, and also shows a number of packages that need to be refreshed (those that are dependencies of the package we are installing):

$ pkgin in lynx
calculating dependencies... done.

5 packages to refresh:
  ncursesw-6.0nb3 ncurses-6.0nb4 zlib-1.2.11 xz-5.2.3nb1 gettext-lib-0.19.8.1

2 packages to install:
  gettext-tools-0.19.8.1nb1 lynx-2.8.8.2nb7

5 to refresh, 0 to upgrade, 0 to remove, 2 to install
8816K to download, 16M to install

proceed ? [Y/n] n

Perform a full upgrade using the existing pkgin:

$ pkgin full-upgrade
calculating dependencies... done.

3 packages to be upgraded:

nodejs-9.4.0 ncursesw-6.0nb3 ncurses-6.0nb4

3 packages to be installed (13M to download, -18K to install):

ncurses-6.0nb5 nodejs-9.5.0 ncursesw-6.0nb4

proceed ? [Y/n] n

Are they being upgraded or installed? It's not clear. Why duplicate them? Let's try with the new client:

$ pkgin full-upgrade
calculating dependencies... done.

155 packages to refresh:
  yarn-1.3.2 w3m-0.5.3.0.20170102 ruby23-redcarpet-3.4.0
  ruby23-pygments.rb-1.1.2 ruby23-jekyll-3.5.2nb1 pkgsrc-gnupg-keys-20170418
  pkgdiff-1.8nb1 pkg_alternatives-1.6 p5-Perl4-CoreLibs-0.004 nbsed-20120308
  nawk-20121220nb1 mutt-1.9.3 cy2-plain-2.1.26nb1 bsdinstall-20160108
  bmake-20150505 abcde-2.8.1nb1 ruby23-multi_json-1.12.2 ruby23-safe_yaml-1.0.4
  ruby23-rouge-1.11.1 ruby23-pathutil-0.14.0 ruby23-mercenary-0.3.6
  ruby23-liquid-4.0.0 ruby23-kramdown-1.15.0 ruby23-jekyll-watch-1.5.0nb1
  ruby23-jekyll-sass-converter-1.5.0 ruby23-jekyll-minima-2.1.1
  ruby23-jekyll-feed-0.9.2 ruby23-colorator-1.1.0 ruby23-bundler-1.15.4
  ruby23-addressable-2.5.2 pkg_install-20171030 libass-0.14.0 libarchive-3.3.2
  gpgme-1.10.0 gnupg2-2.2.4 bootstrap-mk-files-20170802 autoconf-2.69nb7
  p5-WebService-MusicBrainz0-0.94 p5-MusicBrainz-DiscID-0.04 eyeD3-0.7.11
  ruby23-listen-3.1.5nb1 ruby23-sass-3.4.24 ruby23-public_suffix-3.0.0
  ruby23-forwardable-extended-2.6.0 python27-2.7.14 p5-XML-LibXML-2.0132
  p5-Test-Pod-1.51nb2 p5-Class-Accessor-0.51 openldap-client-2.4.45 zlib-1.2.11
  ruby23-rb-inotify-0.9.10 ruby23-rb-fsevent-0.10.2 ruby23-dep-1.5.0
  p5-XML-SAX-0.99nb6 p5-XML-NamespaceSupport-1.12nb1 p5-WWW-RobotRules-6.02nb6
  p5-Net-HTTP-6.17 p5-LWP-Protocol-https-6.07nb1 p5-IO-Socket-SSL-2.052
  p5-LWP-MediaTypes-6.02nb6 p5-HTTP-Negotiate-6.01nb6 p5-HTTP-Daemon-6.01nb6
  p5-HTTP-Cookies-6.04 p5-HTML-Tagset-3.20nb9 p5-HTML-Parser-3.72nb2
  p5-HTTP-Date-6.02nb6 p5-File-Listing-6.04nb6 p5-Encode-Locale-1.05nb3
  p5-Crypt-SSLeay-0.72nb4 mozilla-rootcerts-1.0.20170121nb6 libxml2-2.9.7
  py27-expat-2.7.14 editline-3.1.20150325 bzip2-1.0.6nb1 xz-5.2.3nb1
  ruby23-ffi-1.9.18 p5-XML-SAX-Base-1.09nb1 p5-Path-Class-0.37nb1
  p5-Net-LibIDN-0.12nb9 p5-Mozilla-CA-20160104nb2 p5-IO-Socket-INET6-2.72nb3
  p5-IO-HTML-1.001nb3 gettext-lib-0.19.8.1 p5-Socket6-0.28nb1

3 packages to upgrade:
  nodejs-9.4.0 ncursesw-6.0nb3 ncurses-6.0nb4

155 to refresh, 3 to upgrade, 0 to remove, 0 to install
104M to download, -299K to install

proceed ? [Y/n] n

pkgin now tries to detect the window size and will reformat the list of packages to fit, mirroring the output of apt-get which pkgin is modelled on. We also no longer duplicate the list of packages, and have a nice summary at the end.

Testing

New pkgin binaries are available for testing, as this is a reasonably large change to behaviour I'd like folks to try it out on any unimportant hosts and let me know if there are any issues.

You can download them and run them from where you fetched them to, they do not have to be installed over the top of your existing pkgin binary. This makes it easy to compare output.

EXPERIMENTAL!

If you're brave, I also have versions which include the above patches, as well as changing the upgrade behaviour to use 'pkg_add -U' (i.e. an inplace upgrade) rather than remove+install. This should be a lot faster and less error prone, but is experimental so may well lose data!

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment