Istio is often installed on a Kubernetes cluster to leverage canary deployments. However, not everyone who installs Istio employs canary routing straight away. Some people opt for its security features, it also hosts a variety of other resilience an testing features.

Some Downtime Deployments

If you’re not utilising canary deploys, you may notice that zero-downtime deployments stop being zero-downtime once you’ve enabled Istio sidecars in your Pods.

This has been raised numerous times on GitHub (7136, 7665, 10112, 11298, 12183, and 12342).

There’s too much information in those issues to digest, but suffice to say the problem arises because Istio’s sidecar is not able to shutdown Envoy in a graceful manner. Upon termination of the Pod, SIGTERM is passed to Envoy and it shuts down immediately. This causes any inbound connections with the application container(s) to be terminated, and any in-flight or subsequent outbound calls to fail.

Confirming the issue

I was able to verify this behaviour on a range of Istio versions, from 1.3.x to 1.9.x. A mundane Kubernetes deployment experienced no failed requests during a vanilla Kubernetes rollout. As soon as sidecar injection was enabled, we started to experience failed requests as each Pod was terminated.

Making Kubernetes Rollouts more robust

One solution to this, outlined in a few of the above issues, is to add a custom script to the proxyv2 image and employ a pre-stop handler to invoke this in the sidecar container. This technique is also mentioned on Flagger’s documentation for zero-downtime deployments.

The script looks something like this:

#!/bin/bash
set -e
if ! pidof envoy &>/dev/null; then
  exit 0
fi

if ! pidof pilot-agent &>/dev/null; then
  exit 0
fi

while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do
  sleep 1;
done

exit 0

The meat of it is, the script waits for all listening sockets which are not owned by the proxy (Envoy) to terminate before allowing the sidecar to be sent SIGTERM, at which point the abrupt shutdown behaviour is inconsequential.

However, I ran into an issue with this script. The script only filters the envoy socket. In the Istio versions I tested, pilot-agent also binds a TCP socket, meaning the script never terminated (until it was called with SIGKILL!).
Perhaps in older versions pilot-agent didn’t bind a TCP port.

Let’s revise that block:

while [ $(netstat -plunt | grep tcp | grep -v envoy | grep -v "pilot-agent" | wc -l | xargs) -ne 0 ]; do
  sleep 1;
done

I’ve not yet worked out what function xargs performs in this case. Removing it seemed to work for me!

Pre-stop vs Entrypoint

The original script was intended to be used as a pre-stop hook. This requires configuration in addition to overriding with the custom proxyv2 image. An alternative is to override the entrypoint of the custom image. The new entrypoint defers to the original (/usr/local/bin/pilot-agent "$@" &), and traps SIGTERM, forwarding onto pilot-agent once graceful shutdown has completed.

Putting it all together

To utilise the new image we need to do two things.

1. Build our custom container, targeting a specific Istio version, and push to an accessible registry
2. Update Istio’s sidecar injector configuration to reference the new images. In particular, overwrite proxy.image and tag.

Once we’ve restarted the Istio components and recycled our Pods, we can once again perform zero-downtime deployments with Istio and vanilla Kubernetes rollouts.