An alert reported that more than fifteen messages had been waiting in a queue for over ten minutes.

At first, it looked like a typical capacity issue: perhaps a worker service could not keep up with incoming tasks. The real cause was more subtle. The autoscaling component was still considered healthy by the infrastructure, but it had stopped performing its core job.

I investigated the incident, identified the stalled control loop, and used a restart to restore processing while defining the safeguards needed to prevent recurrence.

What the alert actually meant

Message queues distinguish between messages already delivered to a worker and messages still waiting for a worker to pick them up.

In this incident, messages were waiting in the queue and there were no active consumers.

That detail mattered. The tasks were not stuck inside a slow worker. There was simply no worker available to start processing them.

The desired scaling state required a worker, but the actual replica count remained at zero.

  1. Tasks arrive in the queue.
  2. The autoscaler should start a worker.
  3. Scaling does not happen.
  4. No active worker is available.
  5. Messages accumulate.

Where the failure happened

The task-processing application did not run continuously. When new work appeared, an autoscaling controller was expected to start the required number of application instances.

During the incident, the application remained scaled down to zero instances. No worker was connected to the queue, so tasks continued accumulating.

The conclusion did not come from a single symptom. It was confirmed by several independent signals:

  • the queue was correctly retaining pending messages;
  • the scaling state already indicated that a worker should be started;
  • the autoscaler container was still running;
  • its connection to the state store was still healthy;
  • metrics from its main control loop had stopped updating.

This pointed to a specific failure mode: the process had not crashed, but its main work loop had stopped making progress.

That is why the usual availability checks did not catch the issue. The platform could see a running container, while the container was no longer managing application capacity.

The technical root cause

The controller checked several applications sequentially through an external API responsible for reading and applying application scale.

Simplified pseudocode:

while True:
    for service in services:
        current = get_current_scale(service)
        apply_required_scale(service, current)

The request used to retrieve the current number of instances did not have a timeout. If one request became stuck, the entire sequential loop stopped.

As a result, the controller could remain running, respond to availability checks, receive fresh state updates, and still fail to apply those updates or start workers.

This is a particularly dangerous production failure mode because it creates a false sense of health: the infrastructure dashboard is green, but user workloads are no longer being processed.

The corrective direction was straightforward: enforce bounded external calls and isolate failures so that one stalled service cannot block the entire control loop.

for service in services:
    try:
        current = get_current_scale(service, timeout=5)
        apply_required_scale(service, current, timeout=5)
    except TimeoutError:
        record_scaling_error(service)
        continue

Why a restart helped

After the controller was restarted, it read the current scaling state again, detected that a worker was needed, and started an application instance.

The worker connected to the queue, began consuming messages, and the backlog cleared.

The restart was the right immediate mitigation because it restored processing quickly. But it did not remove the underlying defect.

How to prevent a repeat

Several small improvements can turn this failure mode into a recoverable one:

  • set timeouts for every external API call;
  • treat a controller as unhealthy when its main work loop has not completed within an expected period;
  • alert when key operational metrics stop updating, even if the container remains Running;
  • add logs and metrics that show which operation or application caused the process to stall;
  • keep a clear runbook that separates immediate recovery actions from permanent fixes.

The lesson for production operations

A running container does not necessarily mean a working service.

Production monitoring should not only answer, “Is the process alive?” It should also answer, “Is the process still doing its intended work?”

For an autoscaling controller, that means monitoring real progress: completed control loops, updated metrics, and replica changes when workload appears.

That shift in thinking helps teams catch silent degradation early — before a healthy-looking platform turns into an unavailable service for users.