Skip to content

Instantly share code, notes, and snippets.

@danrue

danrue/tuxmake_tuxbuild_demo.md

Last active November 18, 2020 19:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save danrue/44803f346b2a45218a5a4bb808ce33f5 to your computer and use it in GitHub Desktop.
Save danrue/44803f346b2a45218a5a4bb808ce33f5 to your computer and use it in GitHub Desktop.
Download ZIP
TuxMake and TuxBuild Demo for ClangBuiltLinux
Raw
tuxmake_tuxbuild_demo.md

Demo for Clang Built Linux

Prepared by dan.rue@linaro.org on 2020-11-16.

https://clangbuiltlinux.github.io/

Agenda

  • Intro to TuxMake, especially related to Clang
  • Intro to TuxBuild, especially related to Clang
  • How to integrate TuxBuild with GitHub

TuxMake

TuxMake provides portable and repeatable Linux kernel builds across a variety of architectures, toolchains, kernel configurations, and make targets.

Links!

License: MIT

Strong Defaults

tuxmake ## does _something_

Correctness First

Read-only source tree; all artifacts go to single-use directory by default.

Portable Build Environments

tuxmake --runtime docker
tuxmake --runtime podman

Kconfig

tuxmake -r podman --kconfig tinyconfig --kconfig-add "CONFIG_KASAN=y" --kconfig-add "https://raw.githubusercontent.com/Linaro/meta-lkft/sumo/recipes-kernel/linux/files/lkft.config"

Targets

Build only config and kernel modules.

tuxmake -r podman config modules

Targets supported: config kernel modules dtbs debugkernel. By default all targets are built.

Architectures

tuxmake -r podman --target-arch arm64

Supported architectures: hppa, powerpc, s390, sh, arc, mips, sparc, arm64, arm, riscv, i386, x86_64

Toolchains

tuxmake -r podman --toolchain llvm-11

Supported toolchains: gcc (8,9,10), clang (10,11), llvm (10,11)

llvm = clang + LLVM tools (LVM=1)

Artifacts

build.log  bzImage  config  metadata.json  modules.tar.xz  System.map  vmlinux.xz

metadata.json contains tons of structured info about the build and build environment.

llvm example at https://gist.github.com/danrue/c27addbe35dedc6200684878b50eeacd

Support Matric

tuxmake -r podman -p
         arc     arm     arm64   hppa    i386    mips    powerpc riscv   s390    sh      sparc   x86_64
clang    no      yes     yes     no      yes     yes     no      yes     no      no      no      yes
clang-10 no      yes     yes     no      yes     yes     no      yes     no      no      no      yes
clang-11 no      yes     yes     no      yes     yes     no      yes     no      no      no      yes
gcc      yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes
gcc-10   no      yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes
gcc-8    yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes
gcc-9    yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes     yes
llvm     no      yes     yes     no      yes     yes     no      yes     no      no      no      yes
llvm-10  no      yes     yes     no      yes     yes     no      yes     no      no      no      yes
llvm-11  no      yes     yes     no      yes     yes     no      yes     no      no      no      yes

Getting Started

pip3 install --user -U tuxmake

TuxBuild

Cloud native tuxmake-as-a-service.

CLI project and documentation @ https://gitlab.com/Linaro/tuxbuild

Service orchestration and implemntation is private/commercial.

Single Build

Similar semantics to tuxmake. Builds in real-time.

tuxbuild build --git-repo 'https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git' --git-ref v5.10-rc4 --target-arch arm64 --kconfig defconfig --toolchain clang-11

Output:

Building Linux Kernel https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git at v5.10-rc4
⏳ Queued:  arm64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQn48kQ2ZnIVE3FVtcoBWBPeDr/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") arm64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQn48kQ2ZnIVE3FVtcoBWBPeDr/
👾 Pass (9 warnings): 09162bc32c88 ("Linux 5.10-rc4") arm64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQn48kQ2ZnIVE3FVtcoBWBPeDr/

Artifacts

TuxMake artifacts + additional metadata files, hosted at highly available public URL. Example @ https://builds.tuxbuild.com/1kQn48kQ2ZnIVE3FVtcoBWBPeDr/

Build Set

cat <<EOF > basic.yaml
sets:
  - name: basic
    builds:
      - {target_arch: arm,     toolchain: clang-11, kconfig: multi_v5_defconfig}
      - {target_arch: arm,     toolchain: clang-11, kconfig: aspeed_g5_defconfig}
      - {target_arch: arm,     toolchain: clang-11, kconfig: multi_v7_defconfig}
      - {target_arch: arm64,   toolchain: clang-11, kconfig: defconfig}
      - {target_arch: i386,    toolchain: clang-11, kconfig: defconfig}
      #- {target_arch: i386,    toolchain: llvm-11,  kconfig: defconfig}
      - {target_arch: mips,    toolchain: clang-11, kconfig: malta_kvm_guest_defconfig}
      #- {target_arch: powerpc, toolchain: clang-11, kconfig: ppc44x_defconfig}
      #- {target_arch: powerpc, toolchain: clang-11, kconfig: pseries_defconfig}
      - {target_arch: riscv,   toolchain: clang-11, kconfig: defconfig}
      #- {target_arch: s390,    toolchain: clang-11, kconfig: defconfig}
      - {target_arch: x86_64,  toolchain: clang-11, kconfig: defconfig}
      #- {target_arch: x86_64,  toolchain: llvm-11,  kconfig: defconfig}
EOF
# Build the build set defined in the config file named 'basic.yaml'
tuxbuild build-set --git-repo 'https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git' --git-ref v5.10-rc4 --tux-config basic.yaml --set-name basic

Example output:

Building Linux Kernel build set basic
⏳ Queued:  arm (multi_v5_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmNjzagJdIpqVt16HqYz7wYD/
⏳ Queued:  arm (aspeed_g5_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmT8irISJv0qZsUJgVKiUc5s/
⏳ Queued:  arm (multi_v7_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmRtqAdE9W3qwQ6C0q8PT2Ex/
⏳ Queued:  arm64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmNi2w5SlrBIGpd3ZxZEGgeA/
⏳ Queued:  i386 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmPIDCOt3OLBtvO6yEMEPIIH/
⏳ Queued:  mips (malta_kvm_guest_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmSol4sgpZVd96OY3qGnKlJB/
⏳ Queued:  riscv (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmSX98U5fGEDMXkE96s5q5Xh/
⏳ Queued:  x86_64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmTA3w2rBpS5b3DGlj0lpcZI/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") i386 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmPIDCOt3OLBtvO6yEMEPIIH/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") arm (multi_v5_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmNjzagJdIpqVt16HqYz7wYD/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") arm (multi_v7_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmRtqAdE9W3qwQ6C0q8PT2Ex/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") arm64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmNi2w5SlrBIGpd3ZxZEGgeA/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") mips (malta_kvm_guest_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmSol4sgpZVd96OY3qGnKlJB/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") riscv (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmSX98U5fGEDMXkE96s5q5Xh/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") arm (aspeed_g5_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmT8irISJv0qZsUJgVKiUc5s/
⚗️  Building: 09162bc32c88 ("Linux 5.10-rc4") x86_64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmTA3w2rBpS5b3DGlj0lpcZI/
👾 Pass (1 warning): 09162bc32c88 ("Linux 5.10-rc4") riscv (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmSX98U5fGEDMXkE96s5q5Xh/
🎉 Pass: 09162bc32c88 ("Linux 5.10-rc4") arm (aspeed_g5_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmT8irISJv0qZsUJgVKiUc5s/
👹 Fail (1 error) with status message 'build failed due to compilation error(s)': 09162bc32c88 ("Linux 5.10-rc4") mips (malta_kvm_guest_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmSol4sgpZVd96OY3qGnKlJB/
👾 Pass (32 warnings): 09162bc32c88 ("Linux 5.10-rc4") i386 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmPIDCOt3OLBtvO6yEMEPIIH/
🎉 Pass: 09162bc32c88 ("Linux 5.10-rc4") arm (multi_v5_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmNjzagJdIpqVt16HqYz7wYD/
👾 Pass (2 warnings): 09162bc32c88 ("Linux 5.10-rc4") x86_64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmTA3w2rBpS5b3DGlj0lpcZI/
🎉 Pass: 09162bc32c88 ("Linux 5.10-rc4") arm (multi_v7_defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmRtqAdE9W3qwQ6C0q8PT2Ex/
👾 Pass (9 warnings): 09162bc32c88 ("Linux 5.10-rc4") arm64 (defconfig) with clang-11 @ https://builds.tuxbuild.com/1kQmmNi2w5SlrBIGpd3ZxZEGgeA/

Kconfig

Works the same as tuxmake, except it requires one or more --kconfig (rather than using --kconfig-add), and doesn't support local fragment files (urls and individual frags work fine though).

tuxbuild build --git-repo 'https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git' --git-ref v5.10-rc4 --target-arch arm64 --kconfig tinyconfig --kconfig "CONFIG_KASAN=y" --kconfig "https://raw.githubusercontent.com/Linaro/meta-lkft/sumo/recipes-kernel/linux/files/lkft.config"

CI Features

Use --json-out <filename> to write out a json file with build status.

Use --show-logs to print stderr of build to console.

Use --quiet to supress all informational output.

Use --download to download all artifacts.

Getting Started

pip3 install --user -U tuxbuild

Request credentials at tuxbuild@linaro.org.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment