Inference is a memory problem, too

Three buckets

Weights, intermediate activations, and the KV cache all compete for device memory. Serving more concurrent requests changes the balance.

Next experiment

Record memory use while varying sequence length and concurrency, one variable at a time.