The best ways to host LiteLLM in 2026

The LiteLLM proxy is a small service with an outsized position: it sits between your application and every model provider you use. That placement is what makes hosting it interesting. A background job that's slow is annoying. A gateway that's slow makes every LLM call in your product slow, and a gateway that's down makes your product down.

So this isn't really a "where can I deploy a Python service" question, even though that's what it looks like. It's a question about latency overhead, failure behaviour, and where your provider API keys live. Here's how the realistic options compare.

What you're actually deploying

The proxy is a Python service — FastAPI under the hood — that exposes an OpenAI-compatible API and routes to whatever providers you configure. Two deployment shapes exist and they have quite different requirements.

• Config-only: a YAML file listing models and provider credentials, no database. Stateless, trivially horizontally scalable, restarts cleanly. Fine when your app is the only client and you manage keys yourself.
• With Postgres: virtual API keys, per-key budgets and rate limits, spend tracking, teams, and the admin UI. This is what most people actually want, and it makes the proxy stateful in the way that matters — losing that database means losing every issued key.

Many production deployments add Redis as well, so that rate limits and budgets are enforced across replicas rather than per-process. If you run more than one instance and care about limits being real, you need that shared state — without it, three replicas means three times your intended rate limit.

What actually matters for a gateway

Four properties, roughly in order of how much they'll bother you.

Added latency

The proxy adds a hop. In absolute terms it's small — model calls take hundreds of milliseconds to many seconds, and a well-placed proxy adds single-digit to low-tens of milliseconds. The way that gets bad is geography: proxy in one region, provider in another, app in a third. Each hop crosses the internet and you can turn 10ms of overhead into 150ms without doing anything obviously wrong. Put the proxy near the app, and prefer a region with good connectivity to your providers.

Cold starts are disqualifying here

This is the one that rules out a whole category. Scale-to-zero is excellent for a lot of workloads and it's a poor fit for a gateway on the critical path of every model call. If your proxy sleeps after idle, the first call after a quiet period pays the wake, and that's exactly when a user is waiting.

I say this as someone who sells scale-to-zero infrastructure: don't put your LLM gateway on it unless it's a development environment. Keep a gateway warm. It's a small always-on service and it's the right thing to spend continuous money on.

Streaming and long connections

Almost all LLM traffic is streamed, so the proxy holds server-sent-event connections open for the duration of a generation — sometimes minutes. Any host with an aggressive request timeout, or one that buffers responses instead of streaming them through, will break this in ways that look like random truncation. Verify streaming end-to-end on a long generation before committing to a platform, not with a two-token test.

Where the keys live

The proxy holds provider API keys, which are among the most valuable secrets in an AI stack — they're directly monetizable by whoever steals them. Wherever you host it, those keys should arrive as runtime secrets rather than being baked into an image, and the admin UI should not be publicly reachable. A surprising number of self-hosted proxies have their admin interface exposed to the internet.

The options

1. A VM you manage

Docker on a single instance, or two behind a load balancer. Total control over region, timeouts, and networking, and no platform surprises around streaming. You own patching, TLS, and restarts. For a service this small and this critical, plenty of teams find that acceptable — it's a container and a systemd unit.

2. Container PaaS

Deploy the published image, attach a managed Postgres, set secrets, done. This is the sweet spot for most teams: it's a container-shaped workload with one database, which is precisely what a PaaS is good at.

Two things to check before committing. First, that the platform streams responses rather than buffering them, and permits long-lived connections. Second, that you can pin a minimum instance count above zero, since many PaaS products default to scaling down.

3. Kubernetes

There's a Helm chart, and if you already run Kubernetes this is straightforward — it's a stateless deployment, a Postgres connection, and a Redis. The usual caveat: worth it if the cluster exists, not worth adopting for this.

4. Serverless functions — mostly no

It's tempting: the proxy is a web service, functions are cheap. In practice, cold starts on the critical path, execution timeouts that fight minutes-long streaming generations, and per-invocation connection churn against Postgres all work against you. Some platforms handle streaming well enough to make it viable. Test it hard before you rely on it.

5. MicroVM platforms

Where our own fits. You get git-driven deploys, hardware-isolated tenancy, and managed Postgres for the key store, which covers the gateway's needs cleanly. Isolation is a genuine argument here — a service holding every provider credential you own benefits from a boundary stronger than a shared kernel.

The honest caveat is the one above: don't run it with idle sleep enabled. Configure it as a persistent app so it stays warm. Scale-to-zero is the wrong feature for this particular workload, and it's ours, so I'd rather you use it where it helps.

# Whatever you pick, verify streaming survives a real generation.
# A short prompt won't reveal buffering or timeout problems.
curl -N -X POST https://your-proxy.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [{"role":"user","content":"Count slowly from 1 to 200."}]
  }'

# Watch for: tokens arriving incrementally (not all at once at the end),
# and the connection surviving to completion without a gateway timeout.

Scaling notes

• Horizontal scaling is easy — the proxy is stateless — but shared Redis is what makes rate limits and budgets correct across replicas. Without it, limits multiply by replica count.
• Postgres connections add up. Each replica opens a pool; several replicas against a small managed instance can exhaust connections. Use a pooler if you scale past a few instances.
• Health checks should hit an endpoint that doesn't call a provider. Checking liveness via a real model call means a provider outage takes your gateway out of rotation too.
• Run at least two instances if the gateway is on a user-facing path. It's a cheap service to duplicate and it removes a single point of failure from every LLM call you make.

The short version

LiteLLM hosting is mostly about respecting where the proxy sits. Keep it warm, keep it near your application, make sure your platform streams properly, guard the keys it holds, and put a real database behind it if you use virtual keys.

It's a small service that deserves boring, well-provisioned infrastructure — the opposite of where you'd spend your cleverness. Save that for the workloads behind it, which is where isolation, bursting, and scale-to-zero actually pay off.