// The Deadlock That Was Not a Deadlock

the code

A service endpoint. The client library is async all the way down, as every modern one is. The handler is synchronous because it was written before the library was, and somebody bridged the two with a one-line facade. It has been in production for a year.

// The I/O. Async all the way down, as every modern client library is.
static async Task<int> QueryAsync(int id)
{
    await Task.Delay(20);                       // stands in for the database round trip
    return id * 2;
}

// The synchronous facade somebody added so that callers did not have to change.
static int Query(int id) => QueryAsync(id).GetAwaiter().GetResult();

// The request handler. Runs on a thread-pool thread, exactly as in ASP.NET Core.
static int Handle(int id) => Query(id) + 1;

// The same handler, written the other way.
static async Task<int> HandleAsync(int id) => await QueryAsync(id) + 1;

Three hundred requests arrive at once. Each needs 20 ms of I/O and nothing else. The full file — this code plus the driver and a sampler on their own OS thread — is bench/threads-and-scheduling/threadpool-starvation.cs, run as dotnet run bench/threads-and-scheduling/threadpool-starvation.cs blocking.

find it

the failure

Same file, both modes, sampled at a fixed number of evenly spaced checks — never a clock reading, only counts:

mode=blocking   requests=300   cores=16
  check  1   pool threads   1   queued    1   done   0/300
  check  2   pool threads  27   queued  216   done  72/300
  check  3   pool threads  35   queued  118   done 169/300
  check  4   pool threads  39   queued   81   done 219/300
  pool threads at end 42   (4 checks to drain the queue)

mode=async   requests=300   cores=16
  check  1   pool threads   1   queued    1   done   0/300
  pool threads at end 16   (1 checks to drain the queue)

The async run finishes before the pool has grown past its starting size at all — it never needed to. The blocking run is still climbing at the last check, having already created many times more workers than there are cores, and the queue is still not empty. Nothing in this service is broken; almost nothing in it can run, either.

Now shrink it until every step fits in a table. bench/threads-and-scheduling/starvation-interleaving.cs clamps the pool to two workers and sends three requests, so that injection cannot paper over what is happening. A watchdog has to end the run, because this bug does not resolve on its own:

  step  1  tid   1  [pool threads 2, queued 1]  all 3 requests queued
  step  2  tid   5  [pool threads 2, queued 1]  request 0: handler starts
  step  3  tid   7  [pool threads 2, queued 1]  request 1: handler starts
  step  4  tid   5  [pool threads 2, queued 1]  request 0: blocking this pool thread in .GetAwaiter().GetResult()
  step  5  tid   7  [pool threads 2, queued 1]  request 1: blocking this pool thread in .GetAwaiter().GetResult()
  step  6  tid   5  [pool threads 2, queued 1]  request 0: QueryAsync starts, awaiting the I/O
  step  7  tid   7  [pool threads 2, queued 1]  request 1: QueryAsync starts, awaiting the I/O
  step  8  tid   4  [pool threads 2, queued 3]  WATCHDOG: no progress — printing the log and exiting
  never finished: 3 requests, pool threads 2, queued 3

That is the whole bug, in seven steps. As a table — every row is a step of that run:

The last row is observed, not reasoned: the queue held one item while the workers were parking and three items by the time the watchdog gave up, and it never shrank in between. Neither timer, nor the kernel, nor the runtime is stuck: the queue is full, every item in it is runnable, and there is no thread left to run one. A circular wait with no lock in it:

   worker tid 5                        worker tid 7
   ┌──────────────────────┐            ┌──────────────────────┐
   │ handling request 0    │            │ handling request 1    │
   │ parked in GetResult() │            │ parked in GetResult() │
   └───────────┬───────────┘            └───────────┬───────────┘
               │ needs                               │ needs
               ▼                                     ▼
         continuation 0                        continuation 1
               │                                     │
               └───────────────────┬─────────────────┘
                                    ▼
                 pool's queue: [request 2, continuation 0, continuation 1]
                                    │
                                    ▼
                 needs a free worker to run any of these — but
                 the only two workers that exist are tid 5 and
                 tid 7, and both are parked waiting for exactly
                 the items sitting in this queue.

No lock is held anywhere in this picture. It is a cycle made entirely of “I am waiting for a work item that only a worker like me can run” — every worker that would run the next step is the thing that’s blocked.

And the OS’s view of the same shape, from bench/threads-and-scheduling/blocked-threads.cs — twenty blocking work items, then a census of every thread in the process with the kernel function it is parked in, straight out of /proc/self/task:

pool threads 21, work items still queued 0
   21 x  .NET TP Worker     state S   parked in futex_do_wait
    2 x  blocked-thr-ust    state S   parked in futex_do_wait
    1 x  blocked-threads    state R   parked in 0
    1 x  .NET SynchManag    state S   parked in poll_schedule_timeout.constprop.0
    1 x  .NET EventPipe     state S   parked in poll_schedule_timeout.constprop.0
    1 x  .NET DebugPipe     state S   parked in wait_for_partner
    1 x  .NET Debugger      state S   parked in futex_do_wait
    1 x  .NET Finalizer     state S   parked in futex_do_wait
    1 x  .NET Tiered Com    state S   parked in futex_do_wait
    1 x  .NET TP Gate       state S   parked in futex_do_wait
    1 x  .NET Timer         state S   parked in futex_do_wait
    1 x  .NET SigHandler    state S   parked in anon_pipe_read

Twenty-one workers, all in state S, all sitting in futex_do_wait — the kernel primitive underneath every managed wait, and the subject of how a lock is built. state S is why the CPU graph is flat during an outage: a blocked thread burns nothing at all.

why it breaks

.GetAwaiter().GetResult() does not wait for a thread to finish. It waits for a work item to be scheduled. That is the step everybody skips. QueryAsync returns at its first await; there is no thread inside it. What completes its Task is a continuation, and a continuation is a work item queued to the very pool whose thread you are currently occupying. So each blocked request holds one worker hostage and simultaneously joins the queue for another one.

Every one of the 300 handlers that gets a worker immediately blocks that worker inside GetResult(), waiting for a continuation. The blocked handlers are, as a group, occupying workers and creating demand for workers at the same time. The 20 ms timers eventually fire and post their continuations into the same global queue the still-unstarted handlers are sitting in — behind them. Nothing can drain until the pool decides to add a thread, one round of injection at a time.

The only thing that breaks the cycle is the pool deciding to add a thread — and how it decides that is on the topic page: cautiously, and more so the less the pool can tell about why a worker is idle. The blocking run above needed four separate rounds of injection and dozens of extra workers to finish what the await version finished with none. The service does not deadlock at this scale; it recovers, slowly, against a load its starting pool size was never going to satisfy on its own. Clamp the pool hard enough — as the interleaving demo above does — and there is no injection left to rescue it at all, and the graded collapse becomes a permanent hang.

Two details are worth keeping separate, because both get called “the async deadlock”:

Real services hit the graded version far more often, which is worse in one specific way — it looks like a performance problem, so people go looking for a slow query.

the fix

Delete the facade. That is the whole fix, and every other option is a way of paying for keeping it.

// The I/O. Unchanged — it was never the problem.
static async Task<int> QueryAsync(int id)
{
    await Task.Delay(20);                       // stands in for the database round trip
    return id * 2;
}

// The handler, all the way async. No thread is held while the I/O is in flight:
// the method returns at the await, and its state lives in a small heap object
// until a continuation resumes it.
static async Task<int> HandleAsync(int id) => await QueryAsync(id) + 1;

Same 300 requests: the pool never grows past its starting size, because at no point does any request hold a worker hostage waiting on another worker.

Two fixes people reach for first, and what is wrong with each:

“Raise ThreadPool.SetMinThreads.” This works, in the sense that a tourniquet works. It removes the injection delay for the first N threads, so the collapse becomes a fast, expensive service instead of a stalled one. You are still paying a full OS thread — reserved address space and resident memory per thread, both far larger than a pending await’s (the exact figures are on the topic page) — per concurrent request, and you have chosen a concurrency ceiling by hand that the pool was supposed to discover. It is the right emergency action at 3 a.m. and the wrong permanent state.

“Add ConfigureAwait(false) everywhere.” This fixes a different bug. ConfigureAwait(false) stops the continuation being posted back to a captured SynchronizationContext — and ASP.NET Core does not install one, so under ASP.NET Core it changes nothing here at all. The continuation was already going to the pool. The pool is what has no threads. (In a WinForms or legacy-ASP.NET app it does fix the hang, which is exactly why the advice travels to places it does not belong.)

The third one, Task.Run(() => Query(id)).Result, is worse than the bug: it consumes a pool thread for the blocking call and a pool thread for the caller waiting on it, so it doubles the pool threads consumed per request. Moving blocking code onto the pool does not remove the blocking; it just makes it somebody else’s thread.

what this looks like in prod

The pager says the service is up and the requests are timing out. CPU is 15%, memory is flat, GC is quiet, and the logs contain nothing but the timeouts themselves. Restarting fixes it for as long as it takes the traffic to come back.

What to look at, in order:

1. threadpool-queue-length in dotnet-counters. A queue climbing while CPU is low is starvation, and there is no other thing it can be. threadpool-thread-count climbing a little at a time next to it is the confirmation.
2. A dump. dotnet-dump analyze, then clrstack -all. Dozens of pool threads whose stacks bottom out in a wait, and directly above each wait, one of your own frames calling .Result, .Wait(), or .GetAwaiter().GetResult(). That frame is the fix.
3. On Linux, without any managed tooling at all, cat /proc/PID/task/*/wchan and count the threads sitting in futex_do_wait while top shows the process near idle.

The usual sources, in the order they turn up: a sync facade over an async client like the one above; HttpClient or a database call reached through a legacy interface; async void handlers that a caller “awaits” by blocking; a SemaphoreSlim.Wait() (the synchronous one) instead of await WaitAsync(); constructors and property getters, which cannot be async and so become the place people hide a .Result; and logging or telemetry that flushes synchronously.

The prevention that actually holds: an analyzer rule banning .Result, .Wait() and GetAwaiter().GetResult() outside Main, and a load test with concurrency well above your core count — because at low concurrency this bug is invisible. At a handful of concurrent requests, the code above is fine.

the same idea in other languages

common bugs

• Testing at a concurrency below your core count. A handful of concurrent requests through the broken handler above look perfect — a worker per request, no queue. The bug only exists when demand exceeds the pool, so a functional test, a local debug session and a light smoke test all pass. Load-test above Environment.ProcessorCount or you have tested nothing.
• Reading the flat CPU graph as “not our code”. A blocked thread is in state S and burns zero CPU, so the more starved the service is, the healthier the CPU graph looks. The utilization number is measuring the wrong thing; the queue length is measuring the right one.
• Blaming the dependency. Every request’s latency includes the queue wait, so the traces show a slow database call when the database itself did the trivial 20 ms round trip it was asked to do. Compare the client-side span with the server-side one: if the gap is on your side of the wire, you are queuing, not waiting.
• Fixing it with ConfigureAwait(false). It addresses a different mechanism — context capture, not thread supply — and ASP.NET Core has no context to capture. It is not harmful; it is just not related, and shipping it lets everyone believe the bug is fixed.
• Fixing it by raising min threads and stopping there. The incident goes away and the design problem stays: one OS thread per concurrent request — reserved address space and resident memory far larger than a pending await’s, per the topic page — and a hand-picked ceiling that is now load-bearing.
• Assuming an async method is safe to block on because it is async. The asyncness is what makes it unsafe: an async method’s completion is a work item on the pool, so blocking a pool thread on it is a request for a resource you are holding.