Inference route
AI inference routing by workload
The cheapest GPU hour is not automatically the cheapest successful request. Model fit, warm capacity, queue time, token throughput, data boundary, and failure behavior determine the route.
Standing answer
Use community capacity for interruptible batch work, spot capacity for fault-tolerant jobs with explicit retry, and reserved or managed endpoints when predictable latency and availability are hard requirements.
Working comparison
Compare the options that fit this decision
| Route | Use when | Inline price signal | Primary unknown |
|---|---|---|---|
| Community GPU | Batch can retry or queue | RTX 4090 from $0.204/hr | Capacity and interruption |
| Cloud spot | Checkpointed fault-tolerant job | A100 80 GB from $0.67877/hr | Regional spot availability |
| Managed endpoint | Stable latency is required | Price depends on model and scale floor | Throughput and idle minimum |
Decision test
What changes the answer
- 01Reject GPUs that cannot hold the model and runtime overhead.
- 02Separate batch queue tolerance from interactive latency.
- 03Compare cost per successful request, not only cost per GPU hour.
- 04Keep a fallback route when capacity is not guaranteed.
Shareable result receipt
Fixed for this page: the same formula, assumptions and sources every visitor sees.
Use community capacity for interruptible batch work, spot capacity for fault-tolerant jobs with explicit retry, and reserved or managed endpoints when predictable latency and availability are hard requirements.
Formula
Cost per successful request = (GPU hours × hourly price + queue and retry cost + endpoint overhead) ÷ successful requests
Assumptions
- One GPU minimum
- Published hourly input
- No reserved discount
Explicit unknowns
- Measured model throughput
- Capacity at request time
- Retry and queue cost
Sources and freshness
- GPURentalPrices comparison data retrieved 2026-07-24