ci/latest-failures

_summary.md

the bellow bash script (k8s-release-dev-debug.sh) was executed for about 1 hour until it started catching problems in the cross-build.

it fetches versions using:

• gsutil cat gs://kubernetes-release-dev/ci/latest.txt
• gsutil cat gs://kubernetes-release-dev/ci/latest-bazel.txt

and based on the versions tries to gsutil ls the contents of:

gsutil ls gs://kubernetes-release-dev/ci/$(gsutil cat gs://kubernetes-release-dev/ci/latest.txt)/bin/linux/amd64
gsutil ls gs://kubernetes-release-dev/ci/$(gsutil cat gs://kubernetes-release-dev/ci/latest-bazel.txt)/bin/linux/amd64

then tries to find the following files in those lists: kubelet kubeadm kubectl kube-proxy.tar kube-apiserver.tar kube-scheduler.tar kube-controller-manager.tar

if a file is missing it prints an error for it.

once latest.txt started pointing to v1.17.0-alpha.1.365+a8478746554769, the script started catching the following error in the non-bazel build:

Fri Oct 11 21:21:11 UTC 2019 * gsutil ls gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64
CommandException: One or more URLs matched no objects.

this indicates that the folder was missing.

after a while the ls command started succeeding, but some files were still missing:

Fri Oct 11 21:21:17 UTC 2019 * gsutil ls gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64
Fri Oct 11 21:21:19 UTC 2019 * ERROR: cannot find artifact gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64/kubelet
Fri Oct 11 21:21:19 UTC 2019 * ERROR: cannot find artifact gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64/kube-apiserver.tar
Fri Oct 11 21:21:19 UTC 2019 * ERROR: cannot find artifact gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64/kube-controller-manager.tar

shortly after all files were there:

Fri Oct 11 21:21:21 UTC 2019 * v1.17.0-alpha.1.365+a8478746554769-bazel
Fri Oct 11 21:21:21 UTC 2019 * gsutil ls gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769-bazel/bin/linux/amd64
Fri Oct 11 21:21:24 UTC 2019 * v1.17.0-alpha.1.365+a8478746554769
Fri Oct 11 21:21:24 UTC 2019 * gsutil ls gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64
Fri Oct 11 21:21:27 UTC 2019 * v1.17.0-alpha.1.365+a8478746554769-bazel
Fri Oct 11 21:21:27 UTC 2019 * gsutil ls gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769-bazel/bin/linux/amd64
Fri Oct 11 21:21:31 UTC 2019 * v1.17.0-alpha.1.365+a8478746554769
Fri Oct 11 21:21:31 UTC 2019 * gsutil ls gs://kubernetes-release-dev/ci/v1.17.0-alpha.1.365+a8478746554769/bin/linux/amd64
...

the latest-bazel.txt never had the artifacts missing.

k8s-release-dev-debug.sh

#!/bin/bash
LANG=en_US
BUCKET=gs://kubernetes-release-dev/ci
FILES=( kubelet kubeadm kubectl kube-proxy.tar kube-apiserver.tar kube-scheduler.tar kube-controller-manager.tar )
while true
do
    VER=$(gsutil cat "${BUCKET}/latest-bazel.txt")
    echo "$(date -u) * ${VER}"
    URL="${BUCKET}/${VER}/bin/linux/amd64"
    echo "$(date -u) * gsutil ls" "${URL}"
    LIST_STR="$(gsutil ls "${URL}")"
    mapfile -t LIST <<< "$LIST_STR"
    for i in "${FILES[@]}"
    do
        MATCH=false
        for j in "${LIST[@]}"
        do
            if [[ "${URL}/$i" = "$j" ]]; then
                MATCH=true
                break
            fi
        done
        if [[ "${MATCH}" = false ]] ; then
            echo "$(date -u) * ERROR: cannot find artifact ${URL}/$i"
        fi
    done
    VER=$(gsutil cat "${BUCKET}/latest.txt")
    echo "$(date -u) * ${VER}"
    URL="${BUCKET}/${VER}/bin/linux/amd64"
    echo "$(date -u) * gsutil ls" "${URL}"
    LIST_STR="$(gsutil ls "${URL}")"
    mapfile -t LIST <<< "$LIST_STR"
    for i in "${FILES[@]}"
    do
        MATCH=false
        for j in "${LIST[@]}"
        do
            if [[ "${URL}/$i" = "$j" ]]; then
                MATCH=true
                break
            fi
        done
        if [[ "${MATCH}" = false ]] ; then
            echo "$(date -u) * ERROR: cannot find artifact ${URL}/$i"
        fi
    done
done