A worker process that ignores shutdown signals doesn't fail loudly, it fails quietly, mid-job, every time a deploy or an autoscaler decides it's time for that process to go. The job it was holding either gets lost or, worse, ends up half-applied with no clean way to tell. Here's how to make a worker process shut down in a way that actually protects the work it's doing, and what to check before assuming it works.
Photo by Anna Shvets on Pexels
Step 1: Listen for the signals your platform actually sends
Most container platforms and process managers send SIGTERM first, giving the process a grace period before sending SIGKILL, which cannot be intercepted or delayed. The grace period length varies by platform, commonly 10 to 30 seconds, so your shutdown logic needs to complete comfortably within that window.
process.on('SIGTERM', () => {
console.log('Received SIGTERM, starting graceful shutdown');
shutdown();
});
If you don't know your platform's grace period, find out before you rely on it. Kubernetes, for example, defaults to a 30-second terminationGracePeriodSeconds before sending SIGKILL, but that value is configurable per deployment, and a shutdown routine that needs 45 seconds on a cluster configured for 10 will never get the chance to finish.
Step 2: Stop accepting new work immediately
The first thing the shutdown handler should do is flip a flag that stops the worker's polling loop from claiming any new jobs. Finishing in-flight work is the goal; starting new work during shutdown just adds more that has to finish before the process can safely exit.
let shuttingDown = false;
function shutdown() {
shuttingDown = true;
}
async function pollLoop() {
while (!shuttingDown) {
const job = await claimNextJob();
if (job) await processJob(job);
}
}
Step 3: Let in-flight jobs finish, with a hard deadline
Track any jobs currently being processed and wait for them to complete before exiting, but don't wait indefinitely. Set a deadline tied to your platform's grace period, and if a job hasn't finished by then, let its visibility timeout expire naturally so another worker can pick it back up rather than blocking shutdown forever.
async function shutdown() {
shuttingDown = true;
const deadline = Date.now() + 8000; // leave margin before SIGKILL
while (activeJobs.size > 0 && Date.now() < deadline) {
await sleep(200);
}
await closeDbConnections();
process.exit(0);
}
Step 4: Release claims on jobs you can't finish in time
If a job is still in flight when the deadline hits, explicitly release its claim rather than letting the process just disappear and forcing the visibility timeout to expire on its own. This is the difference between the next worker picking the job up in seconds versus waiting out the full timeout window, which matters more than it sounds like it should when that window is measured in minutes.
for (const job of activeJobs) {
if (!job.completed) {
await releaseClaim(job.id);
}
}
Step 5: Coordinate with your orchestrator, not just the process signals
If you're running under an orchestrator like Kubernetes, there's a subtlety worth handling explicitly: the orchestrator typically removes a pod from service (stops routing new requests to it) at roughly the same time it sends SIGTERM, but "roughly the same time" isn't "guaranteed simultaneous." A short readiness-probe failure window, deliberately returning unhealthy from a readiness check the moment shutdown starts, gives the orchestrator a beat to stop routing new work to the pod before the shutdown clock starts in earnest, which reduces the odds of a new job landing on a worker that's already mid-shutdown.
Step 6: Test it the way it actually happens in production
Local testing rarely exercises SIGTERM handling, since most developers stop a local process with Ctrl+C, which sends SIGINT, or kill it outright. Explicitly send SIGTERM to a running worker (kill -TERM <pid> on the Node.js process) during a manual test, with a job deliberately in flight, and confirm the job either completes or its claim is released cleanly. This is the scenario that will happen constantly in production and almost never happens by accident in local development, which is exactly why it's worth a dedicated test rather than trusting that it "probably works."
What to watch for in production after you ship this
Shutdown handling that looks correct in a manual test can still misbehave under real deploy conditions, so it's worth adding two specific metrics once this is live: a count of jobs whose claims were explicitly released during shutdown (this should be rare, and a sudden spike usually means jobs are taking longer than your deadline allows), and the gap between when a job's claim was released and when the next worker picked it back up (this should be seconds, and if it's minutes, something in your claim-checking query or worker polling interval needs attention). Neither metric is exotic, but neither shows up automatically either; they need to be added deliberately once the shutdown logic itself is in place.
The mistake we see most often
The most common mistake isn't skipping shutdown handling entirely, it's handling SIGTERM for the HTTP server (stop accepting new connections, drain existing ones) while forgetting that a job worker process needs the same discipline applied to its own unit of work, a job, rather than a request. Teams that run job workers as a separate process from their web server sometimes copy the web server's shutdown logic verbatim, without noticing that "in-flight work" means something different for a job that might legitimately take several minutes versus a request that resolves in milliseconds.
A pitfall worth naming: shutdown logic that itself needs the database
One subtlety that catches teams off guard: the shutdown handler often needs to talk to the database (to release claims, to write final status updates), and that database connection needs to still be open when the handler runs. If your connection pool has its own idle-timeout or shutdown logic that fires independently of your job-processing shutdown handler, you can end up in a situation where the code trying to release a job's claim can't reach the database to do it, because the connection pool already tore itself down first. The fix is ordering: make sure database connection teardown happens strictly after job-related shutdown work completes, not on its own independent timer.
Why this matters more for job workers than for web servers
Graceful shutdown gets more attention for HTTP servers, where an abrupt kill drops in-flight requests users are actively waiting on. Job workers get less attention, but the stakes are often higher: a dropped request is usually retried automatically by the client, but a dropped job might be a payment reconciliation or a data sync step that nothing else is going to retry unless the queue itself is designed to notice and recover it.
We go deeper on the durable-queue side of this problem, visibility timeouts, idempotent retries, and dead-letter handling for jobs that never succeed, in a full guide on building a job queue that survives a server restart. Graceful shutdown and a durable queue are two halves of the same problem: one handles the process going away cleanly, the other handles what happens if it doesn't.
Reviewing worker shutdown and job-queue durability together is a common part of the production-readiness audits this web development agency runs for teams shipping background job infrastructure for the first time. The two almost always need to be fixed together, since fixing one without the other just moves where the lost work shows up.