maelvls/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Testing ingress controllers

We want to use Kubernetes 1.20 or lower since we need both the v1 and v1beta1 of
the Ingress resource.

ingress-gce

v1beta1 Ingress with just annotation
v1 Ingress with spec.IngressClassName but no IngressClass object
v1 Ingress with spec.ingressClassName and IngressClass object
v1 Ingress with just annotation and no IngressClass object
v1 Ingress with just annotation and IngressClass object
v1 Ingress with both annotation and spec.ingressClassName (not compliant with spec, but possible to create by applying over the top of an old ingress)


skipper

v1beta1 Ingress with just annotation
v1 Ingress with spec.IngressClassName but no IngressClass object
v1 Ingress with spec.ingressClassName and IngressClass object
v1 Ingress with just annotation and no IngressClass object
v1 Ingress with just annotation and IngressClass object
v1 Ingress with both annotation and spec.ingressClassName (not compliant with spec, but possible to create by applying over the top of an old ingress)


ingress-gce

gcloud container clusters get-credentials smoke-test --zone europe-west2-b --project jetstack-mael-valais
gcloud container clusters create smoke-test --zone=europe-west2-b --cluster-version 1.19
kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  labels:
    app: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: k8s.gcr.io/echoserver:1.4
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  type: NodePort # ClusterIP can't be used with ingress-gce and annotation "gce".
  ports:
  - name: web
    port: 80
    targetPort: 8080
  selector:
    app: echoserver
EOF

yel="\033[33m"
gray="\033[90m"
end='\033[0m'
color() {
  while read -r line; do
    printf "${1}%s${end}\n" "$line"
  done
}
uncolor() {
  sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g"
}
# https://superuser.com/questions/184307/bash-create-anonymous-fifo
PIPE=$(mktemp -u); mkfifo "$PIPE"; exec 3<>"$PIPE"; rm "$PIPE"; exec 3>/dev/stderr
trace() { # Usage: trace ls /usr/local
  printf "${yel}%s${end} " "$1"
  LANG=C perl -e 'print join(" ", map { $_ =~ / / ? "\"".$_."\"" : $_} @ARGV)' -- "${@:2}" $'\n'

  # (1) First, if stdin is attached, display stdin.
  # (2) Then, run the command and print stdout/stderr.
  if ! [ -t 0 ]; then
    tee >(cat >&3) | command "$@" 2> >(uncolor | color "$gray" >&3) > >(tee >(uncolor | color "$gray" >&3))
    # <-------------(1)-------> <------------------------------------------(2)-------------------------------------------------------->
  else
    command "$@" 2> >(uncolor | color "$gray" >&3) > >(tee >(uncolor | color "$gray" >&3))
    # <--------------------(2)------------------------------------------------------------->
  fi
}
v1beta1 Ingress with just annotation

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
meta
  annotations:
    kubernetes.io/ingress.class: gce
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          serviceName: echoserver
          servicePort: 80
EOF
(i=0 && while [[ $i -lt 5 ]] && ! test -n "$(trace kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')"; do (( i++ )); sleep 5; done \
  && i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "$(kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')/.well-known/test"; do (( i++ )); sleep 5; done) || echo "FAIL"
trace kubectl delete ingress example
-> 200
v1 Ingress with spec.IngressClassName but no IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: gce
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          serviceName: echoserver
          servicePort: 80
EOF
(i=0 && while [[ $i -lt 5 ]] && ! test -n "$(trace kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')"; do (( i++ )); sleep 5; done \
  && i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "$(kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')/.well-known/test"; do (( i++ )); sleep 5; done) || echo "FAIL"
trace kubectl delete ingress example
-> status.loadBalancer never comes up.
v1 Ingress with spec.ingressClassName and IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: gce
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: gce
spec:
  controller: k8s.io/gce # Arbitrary since GCE does not support IngressClass.
EOF
(i=0 && while [[ $i -lt 5 ]] && ! test -n "$(trace kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')"; do (( i++ )); sleep 5; done \
  && i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "$(kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')/.well-known/test"; do (( i++ )); sleep 5; done) || echo "FAIL"
kubectl delete ingress example
kubectl delete ingressclass gce
v1 Ingress with just annotation and no IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: gce
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
EOF
(i=0 && while [[ $i -lt 5 ]] && ! test -n "$(trace kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')"; do (( i++ )); sleep 5; done \
  && i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "$(kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')/.well-known/test"; do (( i++ )); sleep 5; done) || echo "FAIL"
kubectl delete ingress example
v1 Ingress with just annotation and IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: gce
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: gce
spec:
  controller: k8s.io/gce # Arbitrary since GCE does not support IngressClass.
EOF
(i=0 && while [[ $i -lt 5 ]] && ! test -n "$(trace kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')"; do (( i++ )); sleep 5; done \
  && i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "$(kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')/.well-known/test"; do (( i++ )); sleep 5; done) || echo "FAIL"
kubectl delete ingress example
kubectl delete ingressclass gce
v1 Ingress with both annotation and spec.ingressClassName (not compliant with spec, but possible to create by applying over the top of an old ingress)

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: gce
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: gce
spec:
  controller: k8s.io/gce # Arbitrary since GCE does not support IngressClass.
EOF
trace kubectl patch ingress example -p '{"spec":{"ingressClassName":"gce"}}'
(i=0 && while [[ $i -lt 5 ]] && ! test -n "$(trace kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')"; do (( i++ )); sleep 5; done \
  && i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "$(kubectl get ingress example -ojsonpath='{.status.loadBalancer.ingress[*].ip}')/.well-known/test"; do (( i++ )); sleep 5; done) || echo "FAIL"
kubectl delete ingress example
kubectl delete ingressclass gce
Skipper

Skipper uses the Pod hostPort feature in order to expose the Ingress. I
previously wrote a blog post on how hostPort works:
https://maelvls.dev/avoid-gke-lb-with-hostport/. Unlike k3s' service-lb and
Akrobateo (Akrobateo is now defunct), Skipper does not set the
status.loadBalancer.ingress[].ip field.
Since Skipper has the "default controller" behavior, I had to add the argument
--kubernetes-ingress-class=skipper.
Since k3d already offers the hostPort feature through its service-lb, we won't
be using Skipper's hostPort (although I did not remove the hostPort field from
Skipper's Deployment). The Service skipper-ingress that Skipper comes with in
service.yaml (which we don't use) is a type: ClusterIP but service-lb only
supports type: LoadBalancer, so we create our own service.
Also, because Kubernetes worker nodes usually don't have the label
kubernetes.io/role (such as on k3s), I patched the Deployment to remove the
selector. Also, I run k3s with a single node and Skipper's Deployment is
configured with 3.
I removed the hostPort because it was preventing creating the pod twice on the
same node, since I only had one node.
k3d cluster create --k3s-arg=--disable="traefik@server:*" --image=docker.io/rancher/k3s:v1.20.14-k3s2 --port 8080:80@loadbalancer
curl -sSL https://raw.githubusercontent.com/zalando/skipper/v0.13.174/docs/kubernetes/deploy/deployment/deployment.yaml \
  | kubectl apply -f - --dry-run=client -ojson \
  | jq '.spec.strategy |= null' \
  | jq '.spec.replicas |= 1' \
  | jq 'del(.spec.template.spec.nodeSelector)' \
  | jq '.spec.template.spec.containers[0].args += ["-kubernetes-ingress-class=skipper"]' \
  | jq 'del(.spec.template.spec.containers[0].ports[0].hostPort)' \
  | jq 'del(.spec.template.spec.hostNetwork)' \
  | kubectl apply -f-
kubectl apply -f https://raw.githubusercontent.com/zalando/skipper/v0.13.174/docs/kubernetes/deploy/deployment/rbac.yaml
kubectl apply -f- <<EOF
kind: Service
apiVersion: v1
metadata:
  name: skipper-ingress
  namespace: kube-system
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 9999
      protocol: TCP
  selector:
    application: skipper-ingress
EOF
kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  labels:
    app: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: k8s.gcr.io/echoserver:1.4
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  type: ClusterIP
  ports:
  - name: web
    port: 80
    targetPort: 8080
  selector:
    app: echoserver
EOF

yel="\033[33m"
gray="\033[90m"
end='\033[0m'
color() {
  while read -r line; do
    printf "${1}%s${end}\n" "$line"
  done
}
uncolor() {
  sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g"
}
# https://superuser.com/questions/184307/bash-create-anonymous-fifo
PIPE=$(mktemp -u); mkfifo "$PIPE"; exec 3<>"$PIPE"; rm "$PIPE"; exec 3>/dev/stderr
trace() { # Usage: trace ls /usr/local
  printf "${yel}%s${end} " "$1"
  LANG=C perl -e 'print join(" ", map { $_ =~ / / ? "\"".$_."\"" : $_} @ARGV)' -- "${@:2}" $'\n'

  # (1) First, if stdin is attached, display stdin.
  # (2) Then, run the command and print stdout/stderr.
  if ! [ -t 0 ]; then
    tee >(cat >&3) | command "$@" 2> >(uncolor | color "$gray" >&3) > >(tee >(uncolor | color "$gray" >&3))
    # <-------------(1)-------> <------------------------------------------(2)-------------------------------------------------------->
  else
    command "$@" 2> >(uncolor | color "$gray" >&3) > >(tee >(uncolor | color "$gray" >&3))
    # <--------------------(2)------------------------------------------------------------->
  fi
}
v1beta1 Ingress with just annotation

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: skipper
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          serviceName: echoserver
          servicePort: 80
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
trace kubectl delete ingress example
-> 200
v1 Ingress with spec.IngressClassName but no IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: skipper
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          serviceName: echoserver
          servicePort: 80
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
trace kubectl delete ingress example
-> 200 WTF with ingressClassName: foo??? Skipper seems to be picking up any
Ingress without or without the annotation and with or without the
ingressClassName. This is very much unexpected: the ingressClassName should
only work for skipper or "", and the annotation should only work for
skipper or no annotation. Even with the argument
--kubernetes-ingress-class=skipper, Skipper seems to be picking up everything.
Update: after more testing and looking at the
code, it
considers the value of that field as a regex, meaning that when it is set to
"skipper", it matches "skipper2" for example.
v1 Ingress with spec.ingressClassName and IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: skipper
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: skipper
spec:
  controller: k8s.io/foo
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
kubectl delete ingress example
kubectl delete ingressclass skipper
-> 200 but same bug as above.
v1 Ingress with just annotation and no IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: skipper
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
kubectl delete ingress example
-> 200 but same bug as above.
v1 Ingress with just annotation and IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: skipper
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: skipper
spec:
  controller: k8s.io/skipper # Arbitrary since skipper does not look at the IngressClass.
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
kubectl delete ingress example
kubectl delete ingressclass skipper
-> 200 but same bug as above.
v1 Ingress with both annotation and spec.ingressClassName (not compliant with spec, but possible to create by applying over the top of an old ingress)

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: skipper
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: skipper
spec:
  controller: k8s.io/skipper # Arbitrary since skipper does not look at the IngressClass.
EOF
trace kubectl patch ingress example -p '{"spec":{"ingressClassName":"skipper"}}'
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
kubectl delete ingress example
kubectl delete ingressclass skipper
-> 200 but same bug as above.
NGINX Inc Ingress Controller

Since Skipper has the "default controller" behavior, I had to add the argument
--kubernetes-ingress-class=skipper.
I use k3s' hostPort feature through its service-lb in order to expose NGINX'
Service using type: LoadBalancer.
k3d cluster create --k3s-arg=--disable="traefik@server:*" --image=docker.io/rancher/k3s:v1.20.14-k3s2 --port 8080:80@loadbalancer
helm repo add nginx-stable https://helm.nginx.com/stable
helm upgrade --install nginx nginx-stable/nginx-ingress --set controller.service.type=LoadBalancer --version 0.12.0 # NGINX Ingress v2.1.0
kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  labels:
    app: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: k8s.gcr.io/echoserver:1.4
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  type: ClusterIP
  ports:
  - name: web
    port: 80
    targetPort: 8080
  selector:
    app: echoserver
EOF

yel="\033[33m"
gray="\033[90m"
end='\033[0m'
color() {
  while read -r line; do
    printf "${1}%s${end}\n" "$line"
  done
}
uncolor() {
  sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g"
}
# https://superuser.com/questions/184307/bash-create-anonymous-fifo
PIPE=$(mktemp -u); mkfifo "$PIPE"; exec 3<>"$PIPE"; rm "$PIPE"; exec 3>/dev/stderr
trace() { # Usage: trace ls /usr/local
  printf "${yel}%s${end} " "$1"
  LANG=C perl -e 'print join(" ", map { $_ =~ / / ? "\"".$_."\"" : $_} @ARGV)' -- "${@:2}" $'\n'

  # (1) First, if stdin is attached, display stdin.
  # (2) Then, run the command and print stdout/stderr.
  if ! [ -t 0 ]; then
    tee >(cat >&3) | command "$@" 2> >(uncolor | color "$gray" >&3) > >(tee >(uncolor | color "$gray" >&3))
    # <-------------(1)-------> <------------------------------------------(2)-------------------------------------------------------->
  else
    command "$@" 2> >(uncolor | color "$gray" >&3) > >(tee >(uncolor | color "$gray" >&3))
    # <--------------------(2)------------------------------------------------------------->
  fi
}
Works as a “default controller”

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
spec:
  rules:
  - host: localhost
    http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
trace kubectl delete ingress example
-> 200
v1 Ingress with spec.IngressClassName but no IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - host: *
        path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          serviceName: echoserver
          servicePort: 80
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
trace kubectl delete ingress example
-> 200 WTF with ingressClassName: foo??? nginx seems to be picking up any
Ingress without or without the annotation and with or without the
ingressClassName. This is very much unexpected: the ingressClassName should
only work for nginx or "", and the annotation should only work for
nginx or no annotation. Even with the argument
--kubernetes-ingress-class=nginx, nginx seems to be picking up everything.
Update: after more testing and looking at the
code, it
considers the value of that field as a regex, meaning that when it is set to
"nginx", it matches "nginx2" for example.
v1 Ingress with spec.ingressClassName and IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/foo
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
kubectl delete ingress example
kubectl delete ingressclass nginx
-> 200 but same bug as above.
v1 Ingress with just annotation and no IngressClass object

trace kubectl create -f- <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/test
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver
            port:
              number: 80
EOF
i=0 && while [[ $i -lt 5 ]] && ! trace curl -isS -o /dev/null -w "%{http_code}\n" --show-error --fail --max-time 10 "localhost:8080/.well-known/test"; do (( i++ )); sleep 5; done
kubectl delete ingress example