Ephemeral App Mindset
Containers are meant to be ephemeral. We as developers should expect our container to be terminated at any time, for reasons such as rolling deployments, downscaling, node termination, or resource usage limits.
This fact becomes even more crucial now that cloud providers offer cost-effective computing through spot instances. With spot instances, your containers will move around nodes, meaning they’ll be provisioned and terminated a lot.
There’s a couple of things to keep in mind to fully embrace the ephemeral app mindset.
Listen for SIGTERM signals. Kubernetes will send a SIGTERM signal to your container to let it know that it’s going to be terminated soon. This is the point where you should finish up any remaining work. Here’s what you should care about:
- Save the current state.
- For job processors: finish up your active jobs and stop pulling any new jobs.
- Close any long-lived database connections or websockets.
How much time do you have to wrap things up? By default, 30 seconds. This is determined by terminationGracePeriodSeconds.
The
SIGTERMsignal is a generic signal used to cause program termination. UnlikeSIGKILL, this signal can be blocked, handled, and ignored. It is the normal way to politely ask a program to terminate. Source
Configure your health checks (readiness and liveness probes). For readiness, probe dependencies that your app can’t serve requests without. For liveness, you mostly just need to know when your app is in a deadlock so Kubernetes can restart it, so can be more lenient and probe something simple like a controller.
Horizontal scaling and stateless apps. We’ll spawn and terminate instances on the spot to meet the current demand. Containers should be stateless to be scale horizontally.