gRPC Deadlines in Go: Passing One Request Budget Through Three ServicesgRPC Deadlines in Go: Passing One Request Budget Through Three Services

In my RxJS search-bar tutorial, changing the search term replaces the previous request. But after the caller drops the call, does the backend stop calculating !?

Consider an API, an aggregator, and a backend handling one request. The caller allows 240 ms. Every service appears to have a timeout, yet the backend can keep working after the caller receives DeadlineExceeded. The delays and timings below are illustrative.

One request, one shrinking budget

The API spends 60 ms preparing a search, the aggregator spends another 60 ms, and the backend does about 400 ms of CPU work. Only about 120 ms remains when the backend starts, so it cannot finish before the caller's deadline.

caller (240 ms) -> API (60 ms) -> aggregator (60 ms) -> backend (400 ms)

This example omits a browser and gateway. An HTTP handler would have to connect its own request lifetime to the outgoing RPC context.

A timeout can restart the clock

Suppose both outgoing hops get a fresh 700 ms timeout based on context.Background(). That severs the incoming cancellation tree. When the caller's 240 ms expires, the API's existing call to the aggregator can continue, and the aggregator can leave the backend running.

outCtx, cancel := context.WithTimeout(context.Background(), 700*time.Millisecond)
defer cancel()

Everyone has a deadline. Whose clock are you using?

Instead, derive the outgoing context from each handler's incoming ctx:

outCtx, cancel := context.WithTimeout(ctx, 700*time.Millisecond)
defer cancel()
resp, err := next.Check(outCtx, req)

The effective deadline is the earlier of the parent's deadline and the new 700 ms cap. You can shorten a child's budget but not replenish it. If the service needs no tighter limit, pass ctx directly. Go's context contract defines that relationship. gRPC propagates deadlines by default in Go when the outgoing call uses the derived context; it cannot recover a parent you discarded.

The RPC stopped. Did the work?

Imagine the backend doing CPU work in batches. A cooperative loop checks for cancellation between batches:

for time.Now().Before(end) {
    if err := ctx.Err(); err != nil {
        return err
    }
    doBatch()
}

Here, doBatch() stands for a bounded piece of work. The table illustrates what changes when the backend ignores that check. Budgets are approximate milliseconds remaining at each service; return and exit times are shown relative to the caller's start:

In the first row, the backend has a fresh context, so its cancellation check sees no reason to stop. In the third, the RPC is canceled around 240 ms, but the loop continues for roughly another 280 ms. gRPC's cancellation guide leaves stopping application work to the handler. The scheduler can preempt a goroutine, but it does not kill it when a context is canceled.

The caller might also cancel before its deadline. With propagated contexts and cooperative work, the backend can stop while time remains. Copying only the deadline into an unrelated context would miss that earlier signal. I would choose the loop's check frequency around the longest acceptable stop delay. Blocking calls need their own context-aware APIs; an outer check cannot interrupt them.

A timeout cannot undo a write

Now replace search with a payment or inventory reservation. The backend may commit before the response is lost. The caller can receive DeadlineExceeded even though the mutation succeeded; gRPC documents this possibility. Checking the context immediately before the write cannot close the race with the commit.

That needs a separate recovery contract: a stable operation key across retries and an outcome lookup for ambiguous completion. For a local write, deduplicate atomically with the mutation; for an external charge, use a downstream idempotency key and reconciliation. My idempotency-key article develops that case. Work intentionally continued after the caller leaves, such as an accepted export, needs an explicit owner and bounded lifetime.

Conclusion

For interactive requests, I'd derive each outgoing context from the incoming one, retain any tighter local cap, and make backend work observe cancellation. Then I'd measure two intervals: caller cancellation to server notification, and notification to worker exit. A long first interval points to propagation or transport; a long second points to work that ignores the signal. Finally, I'd inspect each write for a way to recover its outcome when the caller never receives it.