Our notification service ran out of memory at two in the morning and stayed half alive until nine

One Sunday in July no dispatch emails went out for seven hours. Parcels left the warehouse as usual, and customers who expected to be told about them rang support instead. The service that sends those emails was running the whole time, answering health checks and serving its admin pages. The notifier runs as a single pod because it owns a schedule: every minute a task picks up newly dispatched orders and sends their emails. The same JVM serves a small admin API, and at ten past two someone ran