---
id: "ml-inference"
type: "kind"
category: "compute"
url: "https://localhost:3000/catalogue/kinds/ml-inference"
---

# ML inference

Serves model predictions online.

| | |
|---|---|
| Category | Compute |
| Properties | 8 |
| Components | 38 |
| Rules | 1 |

## Properties

| property | type | values | required | default |
|---|---|---|---|---|
| `accelerator` | enum | `none`, `gpu`, `asic` | no | "none" |
| `acceleratorModel` (advanced) | enum | `nvidia-h100`, `nvidia-h200`, `nvidia-b200`, `nvidia-l4`, `nvidia-a100`, `amd-mi300x`, `aws-trainium`, `aws-inferentia`, `google-tpu`, `intel-gaudi`, `groq-lpu`, `cerebras-wse` | no | — |
| `embeddingSyncMinutes` (advanced) | integer | — | no | — |
| `inputs` | enum | `live`, `historical`, `both` | no | "live" |
| `instances` | integer | — | no | — |
| `managed` | boolean | — | no | false |
| `snapshotToObjectStore` (advanced) | boolean | — | no | false |
| `training` | enum | `batch`, `online` | no | "batch" |

## Rules for this kind · 1

### Stateful node without zone redundancy `stateful-node-without-zone-redundancy` (error)

- Page: https://localhost:3000/catalogue/rules/stateful-node-without-zone-redundancy.md
- Why: A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.
- Applies to: every node in category compute
- Message: Stateful node '{{node.id}}' lives in exactly one availability zone. A zone outage takes its state with it.

It fires when:

- all of these hold:
  - the node's `stateless` is false
  - the node sits inside exactly 1 availability-zone group
  - this does not hold:
    - the node sits inside at least one cell group

## Components · 38

### vendor-neutral

- [Generic ML inference](https://localhost:3000/catalogue/components/ml-inference.md) `ml-inference` — the kind itself

### aws

- [SageMaker endpoint](https://localhost:3000/catalogue/components/sagemaker-endpoint.md) `sagemaker-endpoint` — 3 defaults

### azure

- [Azure ML endpoint](https://localhost:3000/catalogue/components/azure-ml-endpoint.md) `azure-ml-endpoint` — 3 defaults

### cloudflare

- [Workers AI](https://localhost:3000/catalogue/components/cloudflare-workers-ai.md) `cloudflare-workers-ai` — 4 defaults

### default

- [Anyscale Endpoints](https://localhost:3000/catalogue/components/anyscale-endpoints.md) `anyscale-endpoints` — 4 defaults
- [Banana](https://localhost:3000/catalogue/components/banana-dev.md) `banana-dev` — 4 defaults
- [Baseten](https://localhost:3000/catalogue/components/baseten.md) `baseten` — 4 defaults
- [Beam](https://localhost:3000/catalogue/components/beam-cloud.md) `beam-cloud` — 3 defaults
- [BentoML](https://localhost:3000/catalogue/components/bentoml.md) `bentoml` — 3 defaults
- [Hugging Face Inference Endpoints](https://localhost:3000/catalogue/components/huggingface-inference.md) `huggingface-inference` — 4 defaults
- [Infinity](https://localhost:3000/catalogue/components/infinity-embeddings.md) `infinity-embeddings` — 3 defaults
- [Jan](https://localhost:3000/catalogue/components/jan.md) `jan` — 3 defaults
- [KServe](https://localhost:3000/catalogue/components/kserve.md) `kserve` — 3 defaults
- [llama.cpp server](https://localhost:3000/catalogue/components/llama-cpp-server.md) `llama-cpp-server` — 3 defaults
- [llm-d](https://localhost:3000/catalogue/components/llm-d.md) `llm-d` — 3 defaults
- [LM Studio](https://localhost:3000/catalogue/components/lm-studio.md) `lm-studio` — 3 defaults
- [LMDeploy](https://localhost:3000/catalogue/components/lmdeploy.md) `lmdeploy` — 4 defaults
- [LocalAI](https://localhost:3000/catalogue/components/localai.md) `localai` — 4 defaults
- [MLC LLM](https://localhost:3000/catalogue/components/mlc-llm.md) `mlc-llm` — 4 defaults
- [Modal](https://localhost:3000/catalogue/components/modal.md) `modal` — 3 defaults
- [Mystic AI](https://localhost:3000/catalogue/components/mystic-ai.md) `mystic-ai` — 4 defaults
- [NVIDIA NIM (self-hosted)](https://localhost:3000/catalogue/components/nvidia-nim.md) `nvidia-nim` — 4 defaults
- [OctoAI](https://localhost:3000/catalogue/components/octoai.md) `octoai` — 4 defaults
- [Ollama](https://localhost:3000/catalogue/components/ollama.md) `ollama` — 3 defaults
- [Ray Serve](https://localhost:3000/catalogue/components/ray-serve.md) `ray-serve` — 4 defaults
- [Seldon Core](https://localhost:3000/catalogue/components/seldon-core.md) `seldon-core` — 3 defaults
- [SGLang](https://localhost:3000/catalogue/components/sglang.md) `sglang` — 4 defaults
- [TensorFlow Serving](https://localhost:3000/catalogue/components/tensorflow-serving.md) `tensorflow-serving` — 2 defaults
- [TensorRT-LLM](https://localhost:3000/catalogue/components/tensorrt-llm.md) `tensorrt-llm` — 4 defaults
- [Text Embeddings Inference](https://localhost:3000/catalogue/components/text-embeddings-inference.md) `text-embeddings-inference` — 4 defaults
- [Text Generation Inference](https://localhost:3000/catalogue/components/text-generation-inference.md) `text-generation-inference` — 4 defaults
- [TorchServe](https://localhost:3000/catalogue/components/torchserve.md) `torchserve` — 3 defaults
- [NVIDIA Triton](https://localhost:3000/catalogue/components/triton.md) `triton` — 3 defaults
- [vLLM](https://localhost:3000/catalogue/components/vllm.md) `vllm` — 4 defaults
- [Xinference](https://localhost:3000/catalogue/components/xinference.md) `xinference` — 3 defaults

### gcp

- [Vertex AI endpoint](https://localhost:3000/catalogue/components/vertex-ai-endpoint.md) `vertex-ai-endpoint` — 3 defaults

### oci

- [OCI Data Science Model Deployment](https://localhost:3000/catalogue/components/oci-data-science-model-deployment.md) `oci-data-science-model-deployment` — 4 defaults
- [OCI Generative AI Dedicated AI Cluster](https://localhost:3000/catalogue/components/oci-generative-ai-cluster.md) `oci-generative-ai-cluster` — 5 defaults

## Source (`kind` YAML, verbatim)

```yaml
# yaml-language-server: $schema=../schema/kind.schema.json
# Abstract kind: Serves model predictions online (ranking, dispatch, fraud, spam, TTL prediction).
schemaVersion: "1"
type: "kind"
id: "ml-inference"
category: "compute"
properties:
  training:
    type: "enum"
    values: ["batch", "online"]
    default: "batch"
  inputs:
    type: "enum"
    values: ["live", "historical", "both"]
    default: "live"
  embeddingSyncMinutes:
    type: "integer"
    advanced: true
  snapshotToObjectStore:
    type: "boolean"
    default: false
    advanced: true
  accelerator:
    type: "enum"
    values: ["none", "gpu", "asic"]
    default: "none"
  acceleratorModel:
    type: "enum"
    values: ["nvidia-h100", "nvidia-h200", "nvidia-b200", "nvidia-l4", "nvidia-a100", "amd-mi300x", "aws-trainium", "aws-inferentia", "google-tpu", "intel-gaudi", "groq-lpu", "cerebras-wse"]
    advanced: true
  instances:
    type: "integer"
  managed:
    type: "boolean"
    default: false
```

Human page: https://localhost:3000/catalogue/kinds/ml-inference