Skip to content

Instantly share code, notes, and snippets.

@tuxerrante
Last active February 19, 2021 08:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tuxerrante/3b23b75642d1778a21903e309e6fa1c7 to your computer and use it in GitHub Desktop.
Save tuxerrante/3b23b75642d1778a21903e309e6fa1c7 to your computer and use it in GitHub Desktop.
knative troubleshooting
$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
argocd argocd-application-controller-0 1/1 Running 0 3d20h 172.17.0.9 minikube <none> <none>
argocd argocd-dex-server-5fbb579948-jzn9t 1/1 Running 0 3d20h 172.17.0.5 minikube <none> <none>
argocd argocd-redis-6fb68d9df5-rfh7c 1/1 Running 0 3d20h 172.17.0.7 minikube <none> <none>
argocd argocd-repo-server-b4c6dc8f9-lqcln 1/1 Running 0 3d20h 172.17.0.6 minikube <none> <none>
argocd argocd-server-56ffccb4cd-fbkxq 1/1 Running 0 3d20h 172.17.0.8 minikube <none> <none>
default dnsutils 1/1 Running 36 37h 172.17.0.17 minikube <none> <none>
istio-system istio-ingressgateway-6d8565884d-4tqlr 1/1 Running 0 39h 172.17.0.14 minikube <none> <none>
istio-system istiod-788578f785-7h64p 1/1 Running 0 39h 172.17.0.13 minikube <none> <none>
knative-serving activator-85cd6f6f9-lxmz2 1/1 Running 0 39h 172.17.0.11 minikube <none> <none>
knative-serving autoscaler-7959969587-hbdt9 1/1 Running 0 39h 172.17.0.2 minikube <none> <none>
knative-serving controller-577558f799-8tqkc 1/1 Running 0 39h 172.17.0.10 minikube <none> <none>
knative-serving istio-webhook-7647dc9498-c4c69 1/1 Running 0 39h 172.17.0.16 minikube <none> <none>
knative-serving networking-istio-7d685c46d6-4mkgq 1/1 Running 0 39h 172.17.0.15 minikube <none> <none>
knative-serving webhook-78f446786-fvxxg 1/1 Running 0 39h 172.17.0.12 minikube <none> <none>
kube-system coredns-74ff55c5b-q5t4q 1/1 Running 8 9d 172.17.0.3 minikube <none> <none>
kube-system etcd-minikube 1/1 Running 8 9d 192.168.49.2 minikube <none> <none>
kube-system kube-apiserver-minikube 1/1 Running 8 9d 192.168.49.2 minikube <none> <none>
kube-system kube-controller-manager-minikube 1/1 Running 8 9d 192.168.49.2 minikube <none> <none>
kube-system kube-proxy-9jbdh 1/1 Running 7 9d 192.168.49.2 minikube <none> <none>
kube-system kube-scheduler-minikube 1/1 Running 7 9d 192.168.49.2 minikube <none> <none>
kube-system storage-provisioner 1/1 Running 21 9d 192.168.49.2 minikube <none> <none>
$ kubectl describe pod -n istio-system istio-ingressgateway-6d8565884d-4tqlr
Name: istio-ingressgateway-6d8565884d-4tqlr
Namespace: istio-system
Priority: 0
Node: minikube/192.168.49.2
Start Time: Wed, 17 Feb 2021 16:43:56 +0100
Labels: app=istio-ingressgateway
chart=gateways
heritage=Tiller
install.operator.istio.io/owning-resource=unknown
istio=ingressgateway
istio.io/rev=default
operator.istio.io/component=IngressGateways
pod-template-hash=6d8565884d
release=istio
service.istio.io/canonical-name=istio-ingressgateway
service.istio.io/canonical-revision=latest
sidecar.istio.io/inject=false
Annotations: prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
sidecar.istio.io/inject: false
Status: Running
IP: 172.17.0.14
IPs:
IP: 172.17.0.14
Controlled By: ReplicaSet/istio-ingressgateway-6d8565884d
Containers:
istio-proxy:
Container ID: docker://042f3aab1200e7c39a04b167bf1703a5722fdbf414ef2266d1540bc548a24dd4
Image: docker.io/istio/proxyv2:1.9.0
Image ID: docker-pullable://istio/proxyv2@sha256:286b821197d7a9233d1d889119f090cd9a9394468d3a312f66ea24f6e16b2294
Ports: 15021/TCP, 8080/TCP, 8443/TCP, 15012/TCP, 15443/TCP, 15090/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Args:
proxy
router
--domain
$(POD_NAMESPACE).svc.cluster.local
--proxyLogLevel=warning
--proxyComponentLogLevel=misc:error
--log_output_level=default:info
--serviceCluster
istio-ingressgateway
State: Running
Started: Wed, 17 Feb 2021 16:44:43 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Readiness: http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
Environment:
JWT_POLICY: first-party-jwt
PILOT_CERT_PROVIDER: istiod
CA_ADDR: istiod.istio-system.svc:15012
NODE_NAME: (v1:spec.nodeName)
POD_NAME: istio-ingressgateway-6d8565884d-4tqlr (v1:metadata.name)
POD_NAMESPACE: istio-system (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
HOST_IP: (v1:status.hostIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
CANONICAL_SERVICE: (v1:metadata.labels['service.istio.io/canonical-name'])
CANONICAL_REVISION: (v1:metadata.labels['service.istio.io/canonical-revision'])
ISTIO_META_WORKLOAD_NAME: istio-ingressgateway
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
ISTIO_META_UNPRIVILEGED_POD: true
ISTIO_META_ROUTER_MODE: standard
ISTIO_META_CLUSTER_ID: Kubernetes
Mounts:
/etc/istio/config from config-volume (rw)
/etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
/etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
/etc/istio/pod from podinfo (rw)
/etc/istio/proxy from istio-envoy (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/istio from istiod-ca-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-lpvqb (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
limits.cpu -> cpu-limit
requests.cpu -> cpu-request
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio
Optional: true
ingressgateway-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio-ingressgateway-certs
Optional: true
ingressgateway-ca-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio-ingressgateway-ca-certs
Optional: true
istio-ingressgateway-service-account-token-lpvqb:
Type: Secret (a volume populated by a Secret)
SecretName: istio-ingressgateway-service-account-token-lpvqb
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
=======================================================================================
=======================================================================================
$ kubectl -n knative-serving describe configmaps config-domain
Name: config-domain
Namespace: knative-serving
Labels: serving.knative.dev/release=v0.20.0
Annotations: knative.dev/example-checksum: 74c3fc6a
Data
====
192-168-49-2.kubernetes.default.svc.cluster.local:
----
Events: <none>
=======================================================================================
=======================================================================================
$ kn service list
NAME URL LATEST AGE CONDITIONS READY REASON
my-echo http://my-echo.default.192-168-49-2.kubernetes.default.svc.cluster.local 14h 0 OK / 3 False RevisionMissing : Configuration "my-echo" does not have any ready Revision.
=======================================================================================
=======================================================================================
$ kn service create my-echo --image=docker.io/alessandroaffinito/echo:0.1
Creating service 'my-echo' in namespace 'default':
0.047s The Configuration is still working to reflect the latest desired specification.
0.094s The Route is still working to reflect the latest desired specification.
0.145s Configuration "my-echo" is waiting for a Revision to become ready.
10.119s Revision "my-echo-nxxwm-1" failed with message: Unable to fetch image "docker.io/alessandroaffinito/echo:0.1": failed to resolve image to digest: Get "https://index.docker.io/v2/": context deadline exceeded.
10.135s Configuration "my-echo" does not have any ready Revision.
Error: RevisionFailed: Revision "my-echo-nxxwm-1" failed with message: Unable to fetch image "docker.io/alessandroaffinito/echo:0.1": failed to resolve image to digest: Get "https://index.docker.io/v2/": context deadline exceeded.
Run 'kn --help' for usage
=======================================================================================
=======================================================================================
$ kubectl get events --sort-by='.lastTimestamp'
LAST SEEN TYPE REASON OBJECT MESSAGE
37m Normal Pulling pod/dnsutils Pulling image "tutum/dnsutils"
37m Normal Created pod/dnsutils Created container dnsutils
37m Normal Started pod/dnsutils Started container dnsutils
37m Normal Pulled pod/dnsutils Successfully pulled image "tutum/dnsutils" in 18.056334146s
3m2s Warning InternalError revision/my-echo-tnysk-1 Unable to fetch image "alessandroaffinito/echo": failed to resolve image to digest: Get "https://index.docker.io/v2/": context deadline exceeded
86s Warning InternalError route/my-echo failed to remove route annotation to /, Kind= "my-echo": configurations.serving.knative.dev "my-echo" not found
86s Normal FinalizerUpdate route/my-echo Updated "my-echo" finalizers
68s Normal Created service/my-echo Created Configuration "my-echo"
68s Normal Created configuration/my-echo Created Revision "my-echo-nxxwm-1"
68s Normal Created service/my-echo Created Route "my-echo"
68s Normal FinalizerUpdate route/my-echo Updated "my-echo" finalizers
58s Warning LatestCreatedFailed configuration/my-echo Latest created revision "my-echo-nxxwm-1" has failed
8s Warning InternalError revision/my-echo-nxxwm-1 Unable to fetch image "docker.io/alessandroaffinito/echo:0.1": failed to resolve image to digest: Get "https://index.docker.io/v2/": context deadline exceeded
$ minikube addons list
|-----------------------------|----------|--------------|
| ADDON NAME | PROFILE | STATUS |
|-----------------------------|----------|--------------|
| ambassador | minikube | disabled |
| csi-hostpath-driver | minikube | disabled |
| dashboard | minikube | enabled ✅ |
| default-storageclass | minikube | enabled ✅ |
| efk | minikube | disabled |
| freshpod | minikube | disabled |
| gcp-auth | minikube | disabled |
| gvisor | minikube | disabled |
| helm-tiller | minikube | disabled |
| ingress | minikube | disabled |
| ingress-dns | minikube | disabled |
| istio | minikube | disabled |
| istio-provisioner | minikube | disabled |
| kubevirt | minikube | disabled |
| logviewer | minikube | disabled |
| metallb | minikube | disabled |
| metrics-server | minikube | enabled ✅ |
| nvidia-driver-installer | minikube | disabled |
| nvidia-gpu-device-plugin | minikube | disabled |
| olm | minikube | disabled |
| pod-security-policy | minikube | disabled |
| registry | minikube | disabled |
| registry-aliases | minikube | disabled |
| registry-creds | minikube | disabled |
| storage-provisioner | minikube | enabled ✅ |
| storage-provisioner-gluster | minikube | disabled |
| volumesnapshots | minikube | disabled |
|-----------------------------|----------|--------------|
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment