Skip to content

Instantly share code, notes, and snippets.

@timfoster

timfoster/flag-day-mail.md Secret

Last active February 6, 2019 13:27
Show Gist options
  • Save timfoster/ebd30a91087ca2c09b2d68a03ecbaba4 to your computer and use it in GitHub Desktop.
Save timfoster/ebd30a91087ca2c09b2d68a03ecbaba4 to your computer and use it in GitHub Desktop.
Download ZIP
Flag day: Lullaby 3 - Improving the Manta/Triton build
Raw
flag-day-mail.md

Flag day: Lullaby 3 - Improving the Manta/Triton build

Hi everyone,

We've just pushed the first phase of the TOOLS-2043 changes, which brings in the build framework changes described as part of RFD 145.

You should update any local copies of repositories mentioned in this flag day message.

This work allows:

  • components to use eng.git as a submodule to access shared build tooling
  • build common agents directly on the build system rather than downloading tarballs from Manta
  • assemble Triton/Manta SmartOS images directly on the build system
  • upload build artifacts to manta and updates.joyent.com

all without using Mountain Gorilla. The changes are backwards compatible with mountain-gorilla.git but only if you pull the changes also pushed to MG. (see later in this mail for details)

We're also making changes to how Jenkins builds are run for converted Manta/Triton components, in particular, removing the current use of 'TRY_BRANCH' in favor of simply setting 'BRANCH'. We will discuss those later in this mail.

The first set of components to use the updated build framework are:

  • sdc-papi.git
  • sdc-imgapi.git
  • sdc-booter.git

along with the commonly used agents that are bundled in Manta/Triton images:

  • sdc-amon.git
  • registrar.git
  • sdc-config-agent.git
  • manta-mackerel.git
  • manta-minnow.git
  • waferlock.git

Over time, we'll be rolling out the changes to the rest of the Manta/Triton git repositories. Hopefully that work will complete during the first quarter of this year, but it depends on how smoothly this initial phase goes.

We are not yet targeting major changes to the platform build (illumos-joyent or smartos-live) other than replacing platform-build functionality that's currently part of Mountain Gorilla, namely image manifest generation and Manta/updates.joyent.com uploading.

While this only immediately affects you if you're building the components above, now is a good time to talk about the impact to you as a developer on Manta or Triton as the changes will eventually roll out to all components.

You'll probably need to update or reprovision your dev zones. Here's why:

  • We're now more strict as to which pkgsrc versions your dev zones need to be running when building different components. This is so that we're always building and testing bits the same way as Jenkins will build them after your changes integrate.

  • Your dev zones need delegated datasets enabled in order to use the new image assembly tooling.

  • Your build environment (PATH, specifically) needs to be set so that the correct build tools are picked up.

  • The build will now run as a non-root user, so long as the 'Primary Administrator' profile has been assigned to the current user.

All of the above are tested for during the build, and solution-oriented error messages are emitted in case of problems. If you want to test your build environment before attempting a build, use the 'validate-buildenv' make target.

If you're provisioning a new dev zone, we strongly recommend the use of the official "jenkins agent" images: they contain the correct software out of the box, and are the easiest/fastest way to get a supported build system.

There are currently 4 different pkgsrc versions used across Manta/Triton, so you may need that many dev zones depending on which components you are working on.

When using a jenkins-agent image, you should probably modify your vm.conf user-script to enable the manifest-import service, which will enable ssh in your dev zone:

"user-script": "/usr/sbin/mdata-get root_authorized_keys > ~root/.ssh/authorized_keys ; /usr/sbin/mdata-get root_authorized_keys > ~ad
min/.ssh/authorized_keys; svcadm enable manifest-import; "

To show validate-buildenv in action, here's us attempting to build on an unsupported system:

$ cd sdc-papi.git
$ make validate-buildenv
/mnt/tank/projects/submodule/sdc-papi.git/deps/eng/tools/validate-buildenv.sh
This build machine should not be used to build this component.

expected pkgsrc version 2015Q4
 running pkgsrc version 2011Q4

This component should build on an image based on triton-origin-multiarch-15.4.1@1.0.1
The following jenkins-agent image will work: 1356e735-456e-4886-aebd-d6677921694c

To retrieve this image on Triton, use:
# sdc-imgadm import -S 'https://updates.joyent.com?channel=experimental' 1356e735-456e-4886-aebd-d6677921694c

on SmartOS, use:
# imgadm import -S 'https://updates.joyent.com?channel=experimental' 1356e735-456e-4886-aebd-d6677921694c

or import by hand, with:
# curl -k -o img.manifest 'https://updates.joyent.com/images/1356e735-456e-4886-aebd-d6677921694c?channel=experimental'
# curl -k -o img.gz 'https://updates.joyent.com/images/1356e735-456e-4886-aebd-d6677921694c/file?channel=experimental'
and then
# imgadm install -m img.manifest -f img.gz

The current user should have the 'Primary Administrator' profile
which is needed to perform some parts of the build, e.g.
'buildimage'.
To configure this, as root, run:
# usermod -P 'Primary Administrator' timf

The current devzone does not have a delegated zfs dataset,
which is required for 'buildimage' to function.
Please recreate this devzone, ensuring it has a delegated ds.

To do this, when using vmadm in SmartOS or sdc-vmapi in
Triton, add:
'delegate_dataset': true,
to the json configuration. If using the Triton admin interface,
select 'Delegate Dataset' when provisioning the instance.
For more information, see:
https://docs.joyent.com/private-cloud/instances//delegated-data-sets

Error: unexpected $PATH

The $PATH in this shell environment has /bin, /usr/bin, /sbin
/usr/sbin or /opt/tools/bin appearing before /opt/local/bin.

Several of the Manta/Triton component Makefiles contain
assumptions on the GNU implementations of UNIX utilities.

/opt/tools/bin contains compilers that are not currently
supported by the build.
The $PATH should be changed so that /opt/local/bin appears
before /usr/bin, /usr/sbin, /opt/tools/bin and /bin.

The current path is:
/usr/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin:/home/timf/bin:/root/opt/node/bin:/home/timf/projects/node_modules/.bin:/usr/ccs/bin:/usr/bin:/bin:/usr/sbin:/sbin:/root/bin
The current platform image, 20181206T012147Z, is not valid.
This component should instead be built on 20151126T062538Z

To disable this check, set
$ENGBLD_SKIP_VALIDATE_BUILD_PLATFORM in the environment.


make: *** [validate-buildenv] Error 1
$

and here's us using the correct build machine, and showing the 'show-buildenv' target, which prints a summary of the expected build environment:

$ cd sdc-papi.git
$ make validate-buildenv
/mnt/tank/projects/submodule/sdc-papi.git/deps/eng/tools/validate-buildenv.sh
$ make show-buildenv
2015Q4 triton-origin-multiarch-15.4.1@1.0.1
$

There is an escape hatch to bypass all of these checks, but if you use it, please can you get in touch to explain why. We'd really like the opportunity to improve our development process so that everyone is using the same environment, giving us fewer chances for "It Works on My [Build] Machine" syndrome to occur.

One exception is the platform check, which may be individually overridden by setting $ENGBLD_SKIP_VALIDATE_BUILD_PLATFORM in the environment, though note that not all components currently build on newer platforms. We hope to someday get time to work on building with a 'sysroot' so that building on modern platforms is fully supported.

If you wish to build on exactly the build environment used by Jenkins today, the 'joyent-retro-20151126T062538Z' ( bd83a9b3-65cd-4160-be2e-f7c4c56e0606 ) can be used to host jenkins-agent images on the minimum platform image we support, using bhyve or kvm on Triton/SmartOS. Be sure to add a disk to store user data when provisioning it.

[Edit: 31st Jan 2019] The following are the current jenkins agent image names and uuids which will work as dev zones:

Image Name Image UUID
jenkins-agent-i386-1.6.3-2.1.0 956f365d-2444-4163-ad48-af2f377726e0
jenkins-agent-i386-14.2.0-2.1.0 83708aad-20a8-45ff-bfc0-e53de610e418
jenkins-agent-multiarch-15.4.1-2.1.0 1356e735-456e-4886-aebd-d6677921694c
jenkins-agent-multiarch-18.1.0-2.1.0 8b297456-1619-4583-8a5a-727082323f77

[Edit: 6th Feb 2019] At the time of writing, the list of agents per image is at

https://gist.github.com/timfoster/6e2d76584aee72293d7592cf37cd16ae

though note that not all of these repositories have been converted to use this build framework.

Mountain Gorilla users

Make sure your mountain-gorilla clone has the latest changes, pushed as part of TOOLS-2133 in order for it to retrieve the correct agent packages.

While the build changes introduce new features (see below), repositories can still be built with Mountain Gorilla, will use the same Manta directory hierarchy for build artifacts, and can still use the "old" mechanism of assembling zfs images using MG's 'prep_dataset_in_jpc.sh' script, though that should not be necessary given 'buildimage'.

Jenkins jobs for converted components will not use Mountain Gorilla.

Using the new features

We have a few new Makefile targets that are likely to be of interest:

  • buildimage

    This allows you to build a zfs image of your component directly on your build zone. The resulting images are available in the 'bits' directory of your repository.

  • bits-upload / bits-upload-latest

    This uploads your build artifacts to Manta by default to the $MANTA_USER account, at the $ENGBLD_DEST_OUT_PATH location (which defaults to '/public/builds', with subdirectories created according to the component you're building)

    You may optionally choose to publish bits to a local (or NFS) directory by setting $ENGBLD_BITS_UPLOAD_LOCAL in your environment, bypassing a Manta upload.

    You may also optionally publish the image to the updates.joyent.com imgapi instance by setting $ENGBLD_BITS_UPLOAD_IMGAPI in your environment. This uses the 'updates-imgadm' command, and you may set:

    • $UPDATES_IMGADM_CHANNEL (default value computed based on $TRY_BRANCH or $BRANCH)
    • $UPDATES_IMGADM_USER (defaults to 'mg')
    • $UPDATES_IMGADM_IDENTITY (defaults to '~/.ssh/automation.id_rsa')

    if needed.

    The 'bits-upload-latest' target will attempt to upload the latest bits again, which may be useful in case of network outages.

  • print-<ANYTHING>

    Useful when debugging Makefiles, you can now have the build emit the value of any Makefile macro by building the name of that macro, prefixed with 'print-'. For example:

    $ pwd
/home/timf/projects/submodule/putback/sdc-booter.git
$ make print-BUILDIMAGE_NAME
BUILDIMAGE_NAME=dhcpd
$ make print-STAMP
STAMP=grr-TRITON-1079-20190111T150626Z-gbe5ed09
$

  • building everything

    Typically doing:

    $ make all release publish buildimage bits-upload

    in any of the converted component repositories will produce the build artifacts that mountain-gorilla previously produced.

Jenkins changes, and TRY_BRANCH

Mountain Gorilla supported the concept of a TRY_BRANCH build - where developers could push their changes to a development branch on github, then start a Jenkins job, supplying the name of that branch with the 'TRY_BRANCH' build parameter. This would cause the Jenkins job to pass parameters to Mountain Gorilla which would then checkout the requested branch.

That was not ideal, because the Jenkins build metadata would not reflect the git change that was actually built and users would have to delve into logs to determine what was built.

With the rollout of the build changes, we're removing TRY_BRANCH parameter for the component build in favour of simply specifying the BRANCH build parameter to build any given development or release branch.

We're also adding Gerrit repositories to Jenkins jobs for each component, allowing users to start Jenkins jobs that build specific Gerrit patch sets.

To do this, simply pass Jenkins a BRANCH parameter of the form:

    [last 2 digits of gerrit id]/[gerrit id]/[patch set number]

For example, to build https://cr.joyent.us/#/c/5349/2 use a BRANCH parameter with the value:

    49/5349/2

This would allow future work to have Jenkins automatically build and 'vote' for a patch set prior to its integration based on whether the build succeeded, similar to the Gerrit 'check' job today.

Building images with specific agent changes

As mentioned earlier, any agents needed for a given image are now built as part of the image assembly process, caching the agent builds on the build machine.

In order to provide more control over which branches of each agent repository are built, we use the following:

  • First try to checkout the branch indicated by $AGENT_BRANCH
  • Then try to checkout the branch indicated by $BRANCH
  • Finally, fall back to master

By introducing the AGENT_BRANCH parameter, we provide a way for you to use agents built from development branches of an agent source repository in a given Manta/Triton component.

deps/eng is now a submodule

Finally, a reminder that eng.git is now a submodule of component repositories, and may be found under deps/eng. The mechanism to update content in deps/eng is described in boilerplate comments in *./deps/eng/tools/mk/Makefile.**.

You don't have to do anything to use this submodule, simply running 'make' will cause the submodule to be checked out.

There's a lot of moving parts here, so for reference, a complete set of tickets that cover this first phase of the integration is listed below.

If you've any questions about this work, please give us a shout on triton-dev and we'll do our best to assist! Many thanks for all your patience in waiting for these changes, and special thanks to everyone who helped review this work, provide feedback along the way and cheerlead its progress.

 tim

Build framework:

  • TOOLS-2043 Lullaby 3: Improving the Manta/Triton build
  • TOOLS-2062 eng.git should be a git submodule of Triton/Manta repositories
  • TOOLS-2063 shared tooling needed to build Manta/Triton agents
  • TOOLS-2064 the build should do more pre-build validation
  • TOOLS-2065 Makefile.targ should include a print-% target
  • TOOLS-2066 Triton/Manta components should be able to build local zfs images
  • TOOLS-2067 Triton/Manta components need a way to upload build artifacts

For the agents:

  • TOOLS-2043 Lullaby 3: Improving the Manta/Triton build
  • TOOLS-2062 eng.git should be a git submodule of Triton/Manta repositories
  • TOOLS-2063 shared tooling needed to build Manta/Triton agents

For MG:

  • TOOLS-2133 MG needs to look for .tar.gz agent tarballs after TOOLS-2063
  • TRITON-1077 convert imgapi to engbld framework
  • TRITON-1078 convert papi to engbld framework
  • TRITON-1079 convert booter to engbld framework

For the components fixes:

  • TRITON-1077 convert imgapi to engbld framework
  • TRITON-1078 convert papi to engbld framework
  • TRITON-1079 convert booter to engbld framework

For Jenkins

  • TOOLS-2151 deprecate TRY_BRANCH functionality
  • TOOLS-2149 job configuration should support buildng gerrit "branches"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment