AI inference
AI inference routing by workload
The cheapest GPU hour is not automatically the cheapest successful request. Model fit, warm capacity, queue time, token throughput, data boundary, and failure behavior determine the route.
The short answer
Use community capacity for interruptible batch work, spot capacity for fault-tolerant jobs with explicit retry, and reserved or managed endpoints when predictable latency and availability are hard requirements.
Compare the options
Compare the options that fit this decision
| Route | Use when | Price example | What we do not know |
|---|---|---|---|
| Community GPU | Batch can retry or queue | Salad RTX 4090 from $0.16/hr at its lowest priority | Capacity and interruption |
| Cloud spot | Checkpointed fault-tolerant job | Azure spot A100 80 GB (NC24ads A100 v4, East US) from $0.67877/hr | Regional spot availability |
| Managed endpoint | Stable latency is required | Price depends on model and scale floor | Throughput and idle minimum |
What changes the answer
What changes the answer
- 01Reject GPUs that cannot hold the model and runtime overhead.
- 02Separate batch queue tolerance from interactive latency.
- 03Compare cost per successful request, not only cost per GPU hour.
- 04Keep a fallback route when capacity is not guaranteed.
Bill and sources
At these inputs: the same formula, assumptions and sources every visitor sees.
Use community capacity for interruptible batch work, spot capacity for fault-tolerant jobs with explicit retry, and reserved or managed endpoints when predictable latency and availability are hard requirements.
Formula
Cost per successful request = (GPU hours × hourly price + queue and retry cost + endpoint overhead) ÷ successful requests
Assumptions
- One GPU minimum
- Published hourly input
- No reserved discount
Explicit unknowns
- Measured model throughput
- Capacity at request time
- Retry and queue cost