compare single flake attempt to multiple retries - top 20 flakes over past 8 weeks

00_results.md

ci-kubernetes-e2e-gci-gce-single-flake-attempt	count	ci-kubernetes-e2e-gci-gce	count
[k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate]	93	[k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate]	91
Up	38	Timeout	40
[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when CSIDriver does not exist	24	[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance]	34
[sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs [Conformance]	23	Up	31
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance]	21	[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=false	21
[sig-apps] CronJob should delete successful/failed finished jobs with limit of one job	20	[sig-apps] CronJob should delete successful/failed finished jobs with limit of one job	21
[sig-storage] Mounted volume expand Should verify mounted devices can be resized	19	[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly]	20
[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=false	19	[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=nil	19
[sig-cli] Kubectl client kubectl get output should contain custom columns for each resource	16	[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when CSIDriver does not exist	19
[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=nil	16	[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly]	18
[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly]	16	[sig-cli] Kubectl client kubectl get output should contain custom columns for each resource	16
[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works	14	[sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs [Conformance]	15
[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly]	14	TearDown	14
[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory	13	[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path	14
[sig-network] Network should set TCP CLOSE_WAIT timeout	13	[sig-storage] Mounted volume expand Should verify mounted devices can be resized	14
[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource	13	[k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Conformance]	14
[k8s.io] Security Context when creating a pod in the host PID namespace should show its pid in the host PID namespace [LinuxOnly] [NodeFeature:HostAccess]	12	[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it	13
[sig-scheduling] PreemptionExecutionPath runs ReplicaSets to verify preemption running path	11	[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource	13
[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod	11	[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory	12
[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)] volumes should store data	10	[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: inline ephemeral CSI volume] ephemeral should support two pods which share the same volume	12

10_flakes.sql

select
    job,
    build_consistency,
    commit_consistency,
    flakes,
    runs,
    commits,
    array(
      select as struct
        i.n name,
        count(i.failures) flakes
      from tt.tests i
      group by name
      having name not in ('Test', 'DiffResources', 'DumpClusterLogs', 'DumpFederationLogs')  /* uninteresting tests */
      order by flakes desc
      limit 20 /* top three flakiest tests in this job */
    ) flakiest
  from (
    select
      job, /* name of job */
      round(sum(if(flaked=1,passed,runs))/sum(runs),3) build_consistency, /* percentage of runs that did not flake */
      round(1-sum(flaked)/count(distinct commit),3) commit_consistency, /* percentage of commits that did not flake */
      sum(flaked) flakes, /* number of times it flaked */
      sum(runs) runs, /* number of times the job ran */
      count(distinct commit) commits, /* number of commits tested */
      array_concat_agg(tests) tests /* array of flaking tests in this job */
    from (
      select
        job,
        commit,
        if(passed = runs or passed = 0, 0, 1) flaked, /* consistent: always pass or always fail */
        passed,
        safe_cast(runs as int64) runs,
        array(
          select as struct
            i.name n, /* test name */
            countif(i.failed) failures /* number of times it flaked */
          from tt.tests i
          group by n
          having failures > 0 and failures < tt.runs /* same consistency metric */
          order by failures desc
        ) tests
      from (
        select
          job,
          commit,
          sum(if(result='SUCCESS',1,0)) passed,
          count(result) runs,  /* count the number of times we ran a job on this commit for this PR */
          array_concat_agg(test) tests /* create an array of tests structs */
        from (
          SELECT
            job,
            if(substr(job, 0, 3) = 'pr:', 'pull', 'ci') kind,  /* pull or ci */
            version, /* bootstrap git version, empty for ci  */
            if(substr(job, 0, 3) = 'pr:',
              regexp_extract(
                (
                  select i.value
                  from t.metadata i
                  where i.key = 'repos'
                ),
                r'[^,]+,\d+:([a-f0-9]+)"'
              ),
              version
            ) commit,  /* repo commit for PR or version for CI */
            result,  /* SUCCESS if the build passed */
            test  /* repeated tuple of tests */
          FROM `k8s-gubernator.build.all` as t
          where
            datetime(started) > datetime_sub(current_datetime(), interval 56 DAY)
            and job in (
              'ci-kubernetes-e2e-gci-gce',
              'ci-kubernetes-e2e-gci-gce-single-flake-attempt'
            )
            and version != 'unknown'
            and (
              (substr(job, 0, 3) = 'ci-' and version != 'unknown') or
              exists(
                select as struct
                  i
                from t.metadata i
                where i.key = 'repos' and
                array_length(split(replace(i.value,', ', ''), ',')) = 2 /*serial pr jobs only (# of PR refs +1 == 2)*/
              )
            )
        )
        group by job, commit
      ) as tt
    ) as tt
    group by job /* summarize info for this job across all commits/builds */
  ) as tt
  order by flakes desc, commit_consistency, build_consistency, job /* flakiest jobs first */

20_results.json

[
  {
    "job": "ci-kubernetes-e2e-gci-gce-single-flake-attempt",
    "build_consistency": "0.687",
    "commit_consistency": "0.38",
    "flakes": "282",
    "runs": "2111",
    "commits": "455",
    "flakiest": [
      {
        "name": "[k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate]",
        "flakes": "93"
      },
      {
        "name": "Up",
        "flakes": "38"
      },
      {
        "name": "[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when CSIDriver does not exist",
        "flakes": "24"
      },
      {
        "name": "[sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs  [Conformance]",
        "flakes": "23"
      },
      {
        "name": "[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance]",
        "flakes": "21"
      },
      {
        "name": "[sig-apps] CronJob should delete successful/failed finished jobs with limit of one job",
        "flakes": "20"
      },
      {
        "name": "[sig-storage] Mounted volume expand Should verify mounted devices can be resized",
        "flakes": "19"
      },
      {
        "name": "[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=false",
        "flakes": "19"
      },
      {
        "name": "[sig-cli] Kubectl client kubectl get output should contain custom columns for each resource",
        "flakes": "16"
      },
      {
        "name": "[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=nil",
        "flakes": "16"
      },
      {
        "name": "[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly]",
        "flakes": "16"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works",
        "flakes": "14"
      },
      {
        "name": "[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly]",
        "flakes": "14"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory",
        "flakes": "13"
      },
      {
        "name": "[sig-network] Network should set TCP CLOSE_WAIT timeout",
        "flakes": "13"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource",
        "flakes": "13"
      },
      {
        "name": "[k8s.io] Security Context when creating a pod in the host PID namespace should show its pid in the host PID namespace [LinuxOnly] [NodeFeature:HostAccess]",
        "flakes": "12"
      },
      {
        "name": "[sig-scheduling] PreemptionExecutionPath runs ReplicaSets to verify preemption running path",
        "flakes": "11"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod",
        "flakes": "11"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)] volumes should store data",
        "flakes": "10"
      }
    ]
  },
  {
    "job": "ci-kubernetes-e2e-gci-gce",
    "build_consistency": "0.693",
    "commit_consistency": "0.4",
    "flakes": "274",
    "runs": "2096",
    "commits": "457",
    "flakiest": [
      {
        "name": "[k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate]",
        "flakes": "91"
      },
      {
        "name": "Timeout",
        "flakes": "40"
      },
      {
        "name": "[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance]",
        "flakes": "34"
      },
      {
        "name": "Up",
        "flakes": "31"
      },
      {
        "name": "[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=false",
        "flakes": "21"
      },
      {
        "name": "[sig-apps] CronJob should delete successful/failed finished jobs with limit of one job",
        "flakes": "21"
      },
      {
        "name": "[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly]",
        "flakes": "20"
      },
      {
        "name": "[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when podInfoOnMount=nil",
        "flakes": "19"
      },
      {
        "name": "[sig-storage] CSI mock volume CSI workload information using mock driver should not be passed when CSIDriver does not exist",
        "flakes": "19"
      },
      {
        "name": "[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly]",
        "flakes": "18"
      },
      {
        "name": "[sig-cli] Kubectl client kubectl get output should contain custom columns for each resource",
        "flakes": "16"
      },
      {
        "name": "[sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs  [Conformance]",
        "flakes": "15"
      },
      {
        "name": "TearDown",
        "flakes": "14"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path",
        "flakes": "14"
      },
      {
        "name": "[sig-storage] Mounted volume expand Should verify mounted devices can be resized",
        "flakes": "14"
      },
      {
        "name": "[k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Conformance]",
        "flakes": "14"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it",
        "flakes": "13"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource",
        "flakes": "13"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory",
        "flakes": "12"
      },
      {
        "name": "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: inline ephemeral CSI volume] ephemeral should support two pods which share the same volume",
        "flakes": "12"
      }
    ]
  }
]