readinessprobes.md

Liveness

Readiness

Stackanetes ReadinessProbe
search:
https://github.com/stackanetes/stackanetes/search?utf8=%E2%9C%93&q=readinessProbe%3A

http:

socket:

memcached
cinder
glance
nova
neutron
horizon

exec:

etcd
rabbitmq
mariadb

Openstack-helm ReadinessProbe
search:
https://github.com/sapcc/openstack-helm/search?utf8=%E2%9C%93&q=livenessProbe%3A

http:

(project, url, port)

barbican - / - barbican_api_port_internal
ironic - / - ironic_api_port_internal
keystone - /v3 - 5000
manilla - / - manila_api_port_internal
nova-api - / - nova_api_port_internal
cinder - / - cinder_api_port_internal
neutron - / neutron_api_port_internal
glance - / - glance_api_port_internal

socket:

rabbitmq - rabbitmq_port_public

Kolla-kubernetes (2016-11) ReadinessProbe
search:
https://github.com/openstack/kolla-kubernetes/search?utf8=%E2%9C%93&q=readinessProbe%3A

http:

glance-api - /healthcheck - glance_api_port
glance-registry (via common tpl) -
nova-api (via common tpl)

Service

MariaDB

• stackanetes

  Trap pysql response when trying to connect to db from script

    readinessProbe:
        exec:
          command:
            - python
            - mariadb-readiness.py
  

Script: https://github.com/stackanetes/stackanetes/blob/3030c57418224b9c5792d7af5657fd1c32fa788d/mariadb/templates/configmaps/mariadb-readiness.py.yaml.j2

Keystone

• Stackanetes Socket connection https://github.com/stackanetes/stackanetes/blob/ab4a6174e09e176d3494b48d2e1db785b62fc8fe/keystone/templates/deployment.yaml.j2

  readinessProbe:
      tcpSocket:
        port: {{ network.port.public }}
  

• openstack-helm http https://github.com/sapcc/openstack-helm/blob/master/keystone/templates/deployment.yaml

  livenessProbe:
    httpGet:
      path: /v3
      port: 5000
    initialDelaySeconds: 15
    timeoutSeconds: 15
  

Nova

• Stackanetes Socket connection

  • nova-api https://github.com/stackanetes/stackanetes/blob/ab4a6174e09e176d3494b48d2e1db785b62fc8fe/nova/templates/api/deployment.yaml.j2

    readinessProbe: tcpSocket: port: {{ network.port.api }}

  • nova-novncproxy https://github.com/stackanetes/stackanetes/blob/ab4a6174e09e176d3494b48d2e1db785b62fc8fe/nova/templates/novncproxy/deployment.yaml.j2

    readinessProbe:
      tcpSocket:
        port: {{ network.port.novncproxy }}
    

• openstack-helm https://github.com/sapcc/openstack-helm/blob/master/nova/templates/api-deployment.yaml

    livenessProbe:
      httpGet:
        path: /
        port: {{.Values.global.nova_api_port_internal}}
      initialDelaySeconds: 15
      timeoutSeconds: 5
  

Rabbit

• Stackanetes https://github.com/stackanetes/stackanetes/blob/ae909e841333e71f9f3e26cd397ea71432550535/rabbitmq/templates/daemonset.yaml.j2

  readinessProbe:
      timeoutSeconds: {{ rabbitmq.probe_timeout }}
      exec:
        command:
        - bash
        - -x
        - /scripts/readiness.sh
  

Script: https://github.com/stackanetes/stackanetes/blob/ab4a6174e09e176d3494b48d2e1db785b62fc8fe/rabbitmq/templates/configmaps/rabbitmq-scripts.sh.yaml.j2

This is an awesome overview, cheers!

It's also worth noting with these methods that the primarily rely on the k8s API, which is great but has the potential for a few issues as they cannot account (AFAIK) for situations where:

• the service may be operating correctly but not available to be consumed where needed: eg MariaDB is up, but a pod is unable to reach it, the simplest examples being because of a network issue or security group rules. This is much more relevant in the case of OpenStack than most applications launched on k8s as many deployments will be partitioning services for security and performance issues rather than just using a single flat L2 topology. For example, I'm using a CNI networking plugin[1] that creates a neutron (backed by OVN) network for each namespace, and which builds SG rules based upon the ports exposed in the Pod definitions to provide enhanced security, this approach also allows QoS policies to be applied to each pod and allows seamless interaction between OpenStack and k8s managed services/VMs/containers etc.

• A deployment is spread over multiple isolated k8s clusters, or some services are operating outside of k8s - which is especially relevant for existing OpenStack deployments who may wish to gradually migrate to k8s.

To counter these issues I think it is important that the environment check for a pod actually directly tests the service that it is attempting to consume, in addition (or instead) of checking just the k8s API - which from my perspective is a better indicator about whether we should be launching/deploying a service than if the individual pod is ready to run.

This is the methodology I took when building Harbor (https://github.com/portdirect/harbor), using simple bash scripts in init-containers, that I have found operate very well in practice:

• https://github.com/portdirect/harbor/blob/latest/docker/openstack/openstack-keystone/openstack-keystone-init/assets/start.sh#L35
• https://github.com/portdirect/harbor/blob/latest/docker/common-assets/usr/bin/mysql-test#L86
• https://github.com/portdirect/harbor/blob/latest/kubernetes/templates/keystone/controllers.yaml#L301

[1] https://github.com/portdirect/harbor/tree/latest/kubernetes/templates/raven