LLM Inference Infrastructure Cost

Translate LLM usage forecasts into infrastructure spend. Combine daily request volume, tokens per interaction, GPU throughput, and hourly pricing to estimate monthly compute cost along with the GPU hours you need to provision.

Projected prompt or session count per day.
Average prompt + completion tokens returned.
Sustained tokens generated per second by one GPU.
Blended price per GPU-hour including orchestration.
Optional. Defaults to 0.10; reduces served tokens by the hit rate.
Optional. Defaults to 30 days per month.

Infrastructure sizing aid; confirm with your cloud cost management tooling before budgeting.

Examples

  • 5,000 requests, 1,800 tokens, 900 tokens/s, $2.75/hr, 15% cache, 30 days ⇒ $194.79 per month | GPU hours: 70.83 h
  • 12,000 requests, 2,200 tokens, 1,400 tokens/s, $3.20/hr, blank cache, 31 days ⇒ $467.66 per month | GPU hours: 146.14 h

FAQ

What if I use multiple GPU types?

Blend their hourly rates and throughput into a single weighted average before entering the values.

Can I model autoscaling pools?

Yes. Multiply your expected peak requests by the share of time spent at peak to derive an average daily request count.

Does the cost include networking or storage?

No. Add your own surcharge to the GPU hourly cost to cover vector databases, bandwidth, or observability tools.

Additional Information

  • Cache hit rate trims the number of tokens you must serve, lowering both GPU hours and spend.
  • Throughput should reflect steady-state performance per GPU after batching and KV cache optimisations.
  • Monthly GPU hours help estimate how many dedicated or spot instances you need to reserve.

Related calculators