LiteLLM Proxy
API gatewaylitellm
Self-hosted OpenAI-compatible gateway that fronts many model providers and tracks spend per key.
- Provider
- default
- Kind
- API gateway
- Category
- Edge
- Official docs
- https://docs.litellm.ai/docs/routing
| property | type | values | required | kind default | this component |
|---|---|---|---|---|---|
| auth | enum | none | api-key | jwt | oauth2 | mtls | no | none | ▸ api-key |
| concurrencyLimit | boolean | — | no | false | ▸ false |
| fallbacks | boolean | — | no | false | inherited |
| idempotencyKeys | boolean | — | no | false | inherited |
| instances | integer | — | no | — | inherited |
| keyTtlHours | integer | — | no | — | inherited |
| managed | boolean | — | no | false | ▸ false |
| protocol | enum | http | grpc | websocket | fix | ouch | no | http | ▸ http |
| rateLimiting | boolean | — | no | false | ▸ false |
| requiresDatabase | boolean | — | no | false | inherited |
| routingStrategy | enum | simple-shuffle | least-busy | usage-based | latency-based | cost-based | no | simple-shuffle | inherited |
| spendTracking | boolean | — | no | true | inherited |
| tls | boolean | — | no | true | ▸ false |
| virtualKeys | boolean | — | no | false | inherited |
Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.
sync-chain-too-deep
Every consecutive synchronous hop from this entry point spends part of the stated p95 latency budget, and the chain's failures multiply. More than three hops is a budget nobody is tracking.
- — all of these hold:
- — the design requirement `latency.p95Ms` has a value
- — the longest chain of synchronous calls starting at the node is more than 3 hops long
nodes:
- { id: "litellm", component: "litellm" }API Gatewayalibaba-api-gatewayAmazon API Gatewayaws-api-gatewayAWS AppSync GraphQL APIaws-appsyncAzure API Managementazure-api-managementAI Gatewaycloudflare-ai-gatewayCloudflare API Shieldcloudflare-api-shieldMCP Server Portalcloudflare-mcp-portalCloudflare Privacy Gatewaycloudflare-privacy-gatewayGeneric API gatewayapi-gatewayApache APISIXapisixApollo GraphOSapollo-graphosEasegresseasegressEmissary-ingressemissaryEnvoy Gatewayenvoy-gatewayGloo GatewayglooHasurahasuraHelicone AI Gatewayhelicone-gatewayHigresshigressKong GatewaykongKong AI Gatewaykong-ai-gatewayKong Konnectkong-konnectKrakenDkrakendMartian GatewaymartianNot Diamondnot-diamondPortkey AI GatewayportkeyRequestyrequestySpring Cloud Gatewayspring-cloud-gatewayTyk GatewaytykUnifyunify-aiVercel AI Gatewayvercel-ai-gatewayWSO2 API Managerwso2-api-managerZuulzuulApigeeapigeeAgent Gatewaygoogle-agent-gatewayAPI Gatewaygoogle-api-gatewayApigee hybridgoogle-apigee-hybridCloud Endpointsgoogle-cloud-endpointsFirebase Data Connectgoogle-firebase-data-connectIBM API Connectibm-api-connectOCI API Gatewayoci-api-gateway
# yaml-language-server: $schema=../../schema/component.schema.json # LiteLLM Proxy (LLM Gateway): self-hosted OpenAI-compatible gateway that fronts 100+ model providers, # tracks spend and applies budgets per virtual key. Per "Virtual Keys", a master key starting with sk- # is required to start the proxy and virtual keys need a Postgres database; per "Router", the default # routing strategy is simple-shuffle and retries and fallbacks must be configured explicitly. schemaVersion: "1" type: "component" id: "litellm" kind: "api-gateway" provider: "default" docs: "https://docs.litellm.ai/docs/routing" defaults: managed: false auth: "api-key" rateLimiting: false tls: false protocol: "http" concurrencyLimit: false properties: routingStrategy: type: "enum" values: ["simple-shuffle", "least-busy", "usage-based", "latency-based", "cost-based"] default: "simple-shuffle" virtualKeys: type: "boolean" default: false requiresDatabase: type: "boolean" default: false fallbacks: type: "boolean" default: false spendTracking: type: "boolean" default: true