Skip to content

Instantly share code, notes, and snippets.

Keybase proof

I hereby claim:

  • I am fraenkel on github.
  • I am fraenkel (https://keybase.io/fraenkel) on keybase.
  • I have a public key ASAfg6r6tqcHW0jctRn5XWXsSxCZXAKMtZ6WO_Ufl0zn7go

To claim this, I am signing this object:

@fraenkel
fraenkel / gist:882898500b8308eff33c71685bb08839
Created December 4, 2017 23:18
prometheus failures due to master bounce
Prometheus is installed prior to openshift logging
Here are the snippets from the journald of another run since its always the same:
Dec 04 22:38:43 node-ip origin-node[10449]: I1204 22:38:43.872692 10519 kubelet.go:1888] SyncLoop (PLEG): "prometheus-0_monitoring(d3d0ccd2-d943-11e7-b00c-0abf80211996)", event: &pleg.PodLifecycleEvent{ID:"d3d0ccd2-d943-11e7-b00c-0abf80211996", Type:"ContainerStarted", Data:"836c022014c3f97b59a43abdfda78a09aa680aaf14812f4ef8f917b08e17e503"}
Dec 04 22:38:43 node-ip dockerd-current[4378]: I1204 22:38:43.872692 10519 kubelet.go:1888] SyncLoop (PLEG): "prometheus-0_monitoring(d3d0ccd2-d943-11e7-b00c-0abf80211996)", event: &pleg.PodLifecycleEvent{ID:"d3d0ccd2-d943-11e7-b00c-0abf80211996", Type:"ContainerStarted", Data:"836c022014c3f97b59a43abdfda78a09aa680aaf14812f4ef8f917b08e17e503"}
Dec 04 22:38:44 node-ip dockerd-current[4378]: level=info ts=2017-12-04T22:38:44.115903702Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0
@fraenkel
fraenkel / gist:21b8ce1b72097017a03ee6b5ee986a6b
Created December 4, 2017 15:54
openshift logging install failures
[ec2-user terraform-k8s]$ oc get pods -n logging
NAME READY STATUS RESTARTS AGE
logging-curator-1-deploy 0/1 Error 0 1m
logging-es-data-master-huyqxuc2-1-whcfz 2/2 Running 0 2m
logging-fluentd-6c5z4 1/1 Running 0 1m
logging-fluentd-xhjnn 1/1 Running 1 1m
logging-kibana-1-deploy 0/1 Error 0 1m
[ec2-user terraform-k8s]$ oc describe pod -n logging logging-kibana-1-deploy
....