Istio FIPS Build

istio-fips-build.sh

#!/bin/bash -ex

# yum install -y docker git patch jq
# systemctl start docker
# docker info

ISTIO_VERSION=${ISTIO_VERSION:-1.19.3}
MAJOR_ISTIO_VERSION=$(cut -f1-2 -d. <<< ${ISTIO_VERSION})

# Need a custom build-tools-proxy image for 1.18.3+
# https://istio.slack.com/archives/C6FCV6WN4/p1696463622534729
# https://github.com/istio/tools/pull/2566

git clone https://github.com/istio/tools.git --depth 1
cd tools
git fetch --tags
git checkout "${ISTIO_VERSION}"

# Patch tools
sed -i'' \
    -e 's/FROM ubuntu:xenial AS clang_context_amd64/FROM ubuntu:jammy AS clang_context_amd64/' \
    -e 's/FROM ubuntu:xenial AS build_env_proxy_amd64/FROM ubuntu:jammy AS build_env_proxy_amd64/' \
    -e 's/ENV UBUNTU_RELEASE_CODE_NAME=xenial/ENV UBUNTU_RELEASE_CODE_NAME=jammy/' \
    -e 's/ENV DOCKER_VERSION=5:20.10.7~3-0~ubuntu/ENV DOCKER_VERSION=5:20.10.14~3-0~ubuntu/' \
    -e 's/ENV CONTAINERD_VERSION=1.4.6-1/ENV CONTAINERD_VERSION=1.6.12-1/' \
    -e 's/python \\/#python \\/' \
    docker/build-tools/Dockerfile

# Build tools
cd docker/build-tools/
DRY_RUN=true ./build-and-push.sh
cd ../../..

git clone https://github.com/istio/proxy.git --depth 1
pushd proxy
git fetch --tags
git checkout "${ISTIO_VERSION}"
export GOOS=linux

# Patch Makefile for BAZEL_BIN_PATH in 1.19.3
# https://github.com/istio/proxy/pull/5087
sed -i '/exportcache:/i \
exportcache: BAZEL_BIN_PATH ?= $(shell bazel info $(BAZEL_BUILD_ARGS) $(BAZEL_CONFIG_CURRENT) bazel-bin)' \
Makefile.core.mk

# Compile envoy with FIPS: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl#fips-140-2
echo "build --define boringssl=fips" >> .bazelrc
IMG=gcr.io/istio-testing/build-tools-proxy:release-${MAJOR_ISTIO_VERSION}-latest-amd64 BUILD_WITH_CONTAINER=1 BAZEL_BUILD_ARGS=--config=release TARGET_OS=linux make build_wasm build build_envoy exportcache

popd

git clone https://github.com/istio/istio.git --depth 1
pushd istio
git fetch --tags
git checkout "${ISTIO_VERSION}"

# Pre-built binaries need to copied with SHA in name, otherwise build process will download it from gc bucket
# https://github.com/istio/istio/blob/1.18.1/bin/init.sh#L106

# Populate the git version for istio/proxy (i.e. Envoy)
# PROXY_REPO_SHA="${PROXY_REPO_SHA:-$(grep PROXY_REPO_SHA istio.deps  -A 4 | grep lastStableSHA | cut -f 4 -d '"')}"
PROXY_REPO_SHA=$(jq -r '.[] | select(.name == "PROXY_REPO_SHA").lastStableSHA' istio.deps)

# Copy locally built binaries
mkdir -p out/linux_amd64/release
cp -f ../proxy/out/linux_amd64/envoy out/linux_amd64/release/envoy-${PROXY_REPO_SHA}
cp -f out/linux_amd64/release/envoy-${PROXY_REPO_SHA} out/linux_amd64/release/envoy

# Patch Makefile to use BoringCrypto: https://github.com/tetratelabs/istio/blob/tetrate-workflow/tetrateci/docs/fips.md
sed -i'' -e 's%GOOS=linux%CGO_ENABLED=1 GOEXPERIMENT=boringcrypto GOOS=linux%' Makefile.core.mk

# Envoy built with BoringSSL requires libc++ installed in the docker image
# Patch pilot/docker/Dockerfile.proxyv2 to install libc++
cat > Dockerfile.proxyv2.patch << EOF
RUN apt-get update \
  && apt-get upgrade -y \
  && apt-get install -y libc++1 \
  && apt-get autoremove -y \
  && apt-get clean \
  && rm -rf /tmp/* /var/tmp/* \
  && rm -rf /var/lib/apt/lists/*
EOF
sed -i'' '/FROM ${BASE_DISTRIBUTION/r Dockerfile.proxyv2.patch' pilot/docker/Dockerfile.proxyv2
rm Dockerfile.proxyv2.patch

# Build pilot and proxy
TARGET_OS=linux make docker.pilot docker.proxyv2

# Confirm version
docker run --rm --entrypoint="" localhost:5000/proxyv2 envoy --version
docker run --rm --entrypoint="" localhost:5000/proxyv2 pilot-agent version
docker run --rm --entrypoint="" localhost:5000/pilot pilot-discovery version

# docker tag localhost:5000/proxyv2 quay.io/jeesmon/proxyv2:${ISTIO_VERSION}
# docker tag localhost:5000/pilot:latest quay.io/jeesmon/pilot:${ISTIO_VERSION}
# docker login quay.io
# docker push quay.io/jeesmon/proxyv2:${ISTIO_VERSION}
# docker push quay.io/jeesmon/pilot:${ISTIO_VERSION}

@jacob-kuder Updated gist to use sed instead of patch

Thanks so much! I'll give the new script a whirl on my environment and I'll let you know how it goes.

@jacob-kuder Which linux distro do you use for your host? I tried only on Amazon Linux 2. I think someone else successfully built on Ubuntu. Can we please collaborate on this thread in istio slack? https://istio.slack.com/archives/C6FCV6WN4/p1696463622534729

Updated for 1.19.3

Experimental distroless build script: https://gist.github.com/jeesmon/e5672fdc6362156ad756295e1b5b1521

@jacob-kuder Which linux distro do you use for your host? I tried only on Amazon Linux 2. I think someone else successfully built on Ubuntu. Can we please collaborate on this thread in istio slack? https://istio.slack.com/archives/C6FCV6WN4/p1696463622534729

Can confirm have successfully built on Ubuntu 20.

We can remove the BAZEL_BIN_PATH fix for exportcache now since they've merged the PR to master and cherry picked the changes into release-1.18 as well as release-1.19. 🎉

Also the version variable at the top no longer allows for override. Not sure if that was intentional. We had been setting ISTIO_VERSION when calling the script like:

ISTIO_VERSION=1.19.3 ./istio-fips-build.sh

which worked when the line:

ISTIO_VERSION=${ISTIO_VERSION:-1.18.5}

was at the top but not anymore.

@sspaeth-r7 ISTIO_VERSION variable is fixed. As we are checking out proxy version by tag, exportcache fix is still not there yet: https://github.com/istio/proxy/blob/1.19.3/Makefile.core.mk#L177. May be wait until 1.19.4 is released?

Ah good point. I mistakenly thought we were using the release branches for a minute. Yeah I guess we have to wait since they wouldn't change their existing tags.

I am trying to build this for istio 1.20.0 version on amazon linux2 box. I am getting the following errors:

• export GOOS=linux
• GOOS=linux
• echo 'build --define boringssl=fips'
• IMG=gcr.io/istio-testing/build-tools-proxy:release-1.20-latest-amd64
• BUILD_WITH_CONTAINER=1
• BAZEL_BUILD_ARGS=--config=release
• TARGET_OS=linux
• make build_wasm build build_envoy exportcache
  make[1]: *** No rule to make target 'build_wasm'. Stop.
  make: *** [build_wasm] Error 2

I removed build_wasm target and reran from that step

ERROR: /home/.cache/bazel/_bazel_user/1e0bb3bee2d09d2e4ad3523530d3b40c/external/boringssl_fips/BUILD.bazel:25:8: Executing genrule @boringssl_fips//:build failed: (Exit 1): bash failed: error executing command (from target @boringssl_fips//:build) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
clang: error while loading shared libraries: libtinfo.so.6: cannot open shared object file: No such file or directory
/home/.cache/bazel/_bazel_user/1e0bb3bee2d09d2e4ad3523530d3b40c/sandbox/processwrapper-sandbox/2/execroot/io_istio_proxy/external/boringssl_fips /home/.cache/bazel/_bazel_user/1e0bb3bee2d09d2e4ad3523530d3b40c/sandbox/processwrapper-sandbox/2/execroot/io_istio_proxy
ERROR: Clang version doesn't match.
ERROR: /work/extensions/common/BUILD:100:24 Linking extensions/common/proto_util_test failed: (Exit 1): bash failed: error executing command (from target @boringssl_fips//:build) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 95.662s, Critical Path: 73.61s
INFO: 512 processes: 66 internal, 446 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
Makefile.core.mk:56: recipe for target 'build' failed
make[1]: *** [build] Error 1
make: *** [build] Error 2

@vinodshoreline I did not have any issues building 1.20.x after successfully removing build_wasm. Please make sure the build tools version is matching for 1.20.x as well.

@stanpalatnik are you able to build 1.20.3? We're getting a checksum mismatch error on that version using the same build script that works fine for 1.20.2.

@sspaeth-r7 & whoever is facing the checksum issue : istio/istio#49561
Edit: Looks like @sspaeth-r7 was the one to report the issue in the istio slack :)

@vinodshoreline not sure if you managed to figure out the issue, but I found for the 1.20.2 tag the tools dockerfile had changed and the sed lines in the script were doing nothing so it was still using bionic which does not have libncurses6 available and it, at some point, became required by the envoy fips build.

Make the changes from this PR and it should build.