The EUPS distrib facility (eups distrib ...) has support for packaging and
publishing binary builds in a package format called "tarballs". Unlike the
"eupspkg" (source) package format that was the sole DM published product
format, "tarballs" are inherently tied to the microarchitecture and ABI of a
platform. As there are currently multiple supported "stack" build platforms, a
means of coordinating multiple independent binary builds from identical sources
is needed both to ensure consistency of product versions across platforms and
to enable the ability to reproduce a past binary release.
The present scheme for orchestrating multiple platforms is to first publish/tag
a set of "eupspkg"s from a canonical lsstsw based build env. This product/tag
tuple is then used to initiate per platform builds via EUPS distrib. As the
EUPS scm version+build version strings are preserved, product versions will
be consistent across multiple build envs regardless of any pre-exiting EUPS
product state. All platforms publish their build products to a local
"tarballs" repository, which is then merged into a remote directory structure
hosted in an S3 bucket. The S3 bucket serves dual purposes as both a workspace
in which to merge multiple platform builds and as a highly reliable archiving
mechanism. Products are made available by periodically syncing the S3 bucket to
a production webserver which supports the "Apache" HTML directory indexing
format expected by EUPS distrib.
These are the initial build platforms for binary EUPS products.
- el6 / devtoolset-3
- el7 / system gcc
- osx 10.9 / xcode clang-800.0.42.1
Note that the OSX build is being made on 10.11 (el capitan) with
MACOSX_DEPLOYMENT_TARGET=10.9.
Due the unique behavior of /usr/bin/env under osx when "SIP" is enabled, the
shebangs of any installed python scripts need to be updated to the fully
qualified path of the desired python interpreter. The present scheme for
handling this is for the end user to manually invoke a post "tarballs" install
script to mangle the shebang paths.
A new, opinionated, directory structure heuristic is being developed with the
intent of allowing tarball and source (eupspkg) products to be configured by
only specifying a "base url" to the newinstall.sh script.
The current directory is:
- binary products --
<base url>/<os family>/<platform>/<compiler>/<miniconda py[23]>-<minconda version>-<lsstsw abbrev-ref> - source products --
<base url>/src
Eg.:
<base url>/osx/<MACOSX_DEPLOYMENT_TARGET>/<compiler>/<miniconda py[23]>-<minconda version>-<lsstsw abbrev-ref>
<base url>/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67
<base url>/redhat/el6/devtoolset-3/miniconda3-4.2.12-7c8e67
Such that if the base URL given is https://eups.lsst.codes/stack on a EL7
Linux system configured for python 3, the EUPS_PKGROOT would be automatically
constructed as:
This would result in source products being used if a binary package is unavailable and published EUPS tags being merged from both binary and source repos.
-
EUPS re-downloads all distrib tags on each invocation which adds considerably to the wall-clock time when installing a small number of products.
-
Bit-for-bit binary packages are not currently produced by EUPS distrib when creating a tarball of an already built/installed product due to timestamps in both the tar and gzip formats. The
-nflag togzipwill disable timestamps. GNU tar supports a--mtimeflag that allows the mtime value to be forced to a value but there does not appear to be an equivalent with the BSD tar variant shipped with OSX. -
The tarball repo directory structure is different from eupspkg and most files are dropped under the root of the repo rather that being organized into files.
-
eups tags/eups distrib listdoes not appear to have a means of listing tags available only for binary products. Eg.
$ eups distrib list -f `eups flavor` lsst_distrib
Available products from primary server: https://eups.lsst.codes/stack/osx/10.11
lsst_distrib DarwinX86 13.0
lsst_distrib DarwinX86 13.0-2-g14ce290
lsst_distrib DarwinX86 13.0-2-g14ce290+1-
The current mechanism for building/publishing eupspkgs only requires that the build passes on el7 (the de facto reference platform). It would be more desirable for build issues to be caught before getting to the binary tarball construction phase.
-
There is presently no exposed mechanism to disable unit tests -- this would great accelerate binary product build times (which has been a pain-point during testing).
-
Binary products are essentially being published independently on a per platform basis. Is is likely desirable to do some sort of staging so that publishing is all or nothing across all platforms.
-
The S3 -> www server sync is incremental (per file) which results in their being a time window in which not all files for a published platform are available. It also means there is a lag between when a binary publish finishes and when those products may be used by a downstream process. Eg., constructing docker containers from the newly published "tarballs".
-
Relieving an osx 11.12 end-user from having to manually run a script to fix shebang paths. Perhaps an EUPS product post-install script hook?
Go to the EUPS tarball path, Eg:
https://jhoblitt-curly-eups.lsst.codes/stack/redhat/el7/gcc-system/miniconda2-4.2.12-7c8e67/
and look for files that end with ".list".
The present published tags are:
lsst_distrib/w_2017_15
export EUPS_PKGROOT_BASE_URL=https://eups.lsst.codes/stack # default
export LSST_PYTHON_VERSION=3 # unset for py2
curl -sSL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh | bash -s -- -cbt
. ./loadLSST.bash
eups distrib install lsst_distrib -t w_2017_15 -vvv
# OSX with SIP enabled needs shebangs updated to point to the desired python
# interpreter. This prototype script requires that EUPS be setup in the
# environment
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
# run the demo
curl -sSL https://raw.githubusercontent.com/lsst-sqre/buildbot-scripts/master/runManifestDemo.sh | bash -s -- --small --tag w_2017_15