What is Kubernetes Liveness Probe-DecodingDevOps

When the application is running but unable to process the request for some reasons we need to restart the container. We can automate this process in Kubernetes. The kubelet service uses a liveness probe to make sure that the application is working correctly or not. If the application is not working properly it will restart the container itself. in this post we are going to discuss about kubernetes livenessprobe tutorial complete guide and it use cases.

Before sending the traffic to the container, we need to make sure that the container is working properly and able to serve requests. Kubelet uses a readiness probe to know when the container is ready to serve the traffic. A pod may contain more than one container when all the containers are u and running the pods is said to be in the ready state. When the pods are not ready they are removed from the load balancers.

Sometimes the liveness and readiness can create a disaster to the application which will be quite hard to debug. It will check for liveness and readiness before the application is up which will kill the containers before they can even start which will cause crashloopbackoff error. To avoid this, Kubelet uses something called a startup probe which will know that the container has started which will disable the liveness and readiness checks until the containers are started.

Setting up liveness command

   livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

In this example, we are trying to open file /tmp/healthy. If it will find the file liveness will pass and the pods will be considered healthy. When it won’t find this file the pods will be considered failed and kubelet will restart the pod immediately. Here, we have explicitly defined initialDelaySeconds: 5 which will wait for 5 seconds after the container is starting and periodSeconds: 5 makes sure that there are 5 seconds of delay between two checks.

Liveness using HTTP request

In order to setup liveness through HTTP request your application must need a health check API. For example if the application is taking more time than the defined threshold it will consider that the application is not working properly.

http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    duration := time.Now().Sub(started)
    if duration.Seconds() > 10 {
        w.WriteHeader(500)
        w.Write([]byte(fmt.Sprintf("error: %v", duration.Seconds())))
    } else {
        w.WriteHeader(200)
        w.Write([]byte("ok"))
    }
})

The application will try to connect to /healthz on port 8080 3 seconds after the container starts and keep 3 seconds of gap between two consecutive checks. If the response will take more than 10 seconds it will restart the pod.(based on our scenario).

   livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3

Defining startup probe

As described earlier you can use a startup probe if the container is taking a long time to start and the application is not able to start because of constant failure of the liveness probe. The rule of thumb for setting up the container startup probe is failureThreshold * periodSeconds should be a bit more than the worst startup time.

You can define startup probe in the manifest file as below.

ports:
- name: liveness-port
  containerPort: 8080
  hostPort: 8080

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 1
  periodSeconds: 10

startupProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 30
  periodSeconds: 10

This will let the application start for 5 minutes. It will check on every 10 seconds interval whether the application is running or not till it fails 30 times. After which the container will follow whatever you have defined in the restartPolicy.

Parameters which you can override in the liveness probe

initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.

periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.

timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.

successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. Minimum value is 1.

failureThreshold: When a Pod starts and the probe fails, Kubernetes will try failureThreshold times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1.

HTTP probes have additional fields that can be set on httpGet:

host: Host name to connect to, defaults to the pod IP. You probably want to set “Host” in httpHeaders instead.

scheme: Scheme to use for connecting to the host (HTTP or HTTPS). Defaults to HTTP.

path: Path to access on the HTTP server.

httpHeaders: Custom headers to set in the request. HTTP allows repeated headers.

port: Name or number of the port to access on the container. Number must be in the range 1 to 65535.