ML inference

computeml-inference

Serves model predictions online.

Category: Compute
Properties: 8
Components: 38
Rules: 1

Properties · 8 defined by this kind

A node of this kind resolves each property from its own props, then the component's defaults, then the default below.

property	type	values	required	default
accelerator	enum	none \| gpu \| asic	no	none
acceleratorModel · advanced	enum	nvidia-h100 \| nvidia-h200 \| nvidia-b200 \| nvidia-l4 \| nvidia-a100 \| amd-mi300x \| aws-trainium \| aws-inferentia \| google-tpu \| intel-gaudi \| groq-lpu \| cerebras-wse	no	—
embeddingSyncMinutes · advanced	integer	—	no	—
inputs	enum	live \| historical \| both	no	live
instances	integer	—	no	—
managed	boolean	—	no	false
snapshotToObjectStore · advanced	boolean	—	no	false
training	enum	batch \| online	no	batch

Rules for this kind · 1

Stateful node without zone redundancyerror

stateful-node-without-zone-redundancy

A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.

— all of these hold:
— the node's `stateless` is false
— the node sits inside exactly 1 availability-zone group
— this does not hold:
— the node sits inside at least one cell group

Components · 38

vendor-neutral

Generic ML inferenceml-inferencegenericthe kind itself

aws

SageMaker endpointsagemaker-endpoint3 defaults

azure

Azure ML endpointazure-ml-endpoint3 defaults

cloudflare

Workers AIcloudflare-workers-ai4 defaults

default

Anyscale Endpointsanyscale-endpoints4 defaults Bananabanana-dev4 defaults Basetenbaseten4 defaults Beambeam-cloud3 defaults BentoMLbentoml3 defaults Hugging Face Inference Endpointshuggingface-inference4 defaults Infinityinfinity-embeddings3 defaults Janjan3 defaults KServekserve3 defaults llama.cpp serverllama-cpp-server3 defaults llm-dllm-d3 defaults LM Studiolm-studio3 defaults LMDeploylmdeploy4 defaults LocalAIlocalai4 defaults MLC LLMmlc-llm4 defaults Modalmodal3 defaults Mystic AImystic-ai4 defaults NVIDIA NIM (self-hosted)nvidia-nim4 defaults OctoAIoctoai4 defaults Ollamaollama3 defaults Ray Serveray-serve4 defaults Seldon Coreseldon-core3 defaults SGLangsglang4 defaults TensorFlow Servingtensorflow-serving2 defaults TensorRT-LLMtensorrt-llm4 defaults Text Embeddings Inferencetext-embeddings-inference4 defaults Text Generation Inferencetext-generation-inference4 defaults TorchServetorchserve3 defaults NVIDIA Tritontriton3 defaults vLLMvllm4 defaults Xinferencexinference3 defaults

gcp

Vertex AI endpointvertex-ai-endpoint3 defaults

oci

OCI Data Science Model Deploymentoci-data-science-model-deployment4 defaults OCI Generative AI Dedicated AI Clusteroci-generative-ai-cluster5 defaults

Source

1# yaml-language-server: $schema=../schema/kind.schema.json
2# Abstract kind: Serves model predictions online (ranking, dispatch, fraud, spam, TTL prediction).
3schemaVersion: "1"
4type: "kind"
5id: "ml-inference"
6category: "compute"
7properties:
8  training:
9    type: "enum"
10    values: ["batch", "online"]
11    default: "batch"
12  inputs:
13    type: "enum"
14    values: ["live", "historical", "both"]
15    default: "live"
16  embeddingSyncMinutes:
17    type: "integer"
18    advanced: true
19  snapshotToObjectStore:
20    type: "boolean"
21    default: false
22    advanced: true
23  accelerator:
24    type: "enum"
25    values: ["none", "gpu", "asic"]
26    default: "none"
27  acceleratorModel:
28    type: "enum"
29    values: ["nvidia-h100", "nvidia-h200", "nvidia-b200", "nvidia-l4", "nvidia-a100", "amd-mi300x", "aws-trainium", "aws-inferentia", "google-tpu", "intel-gaudi", "groq-lpu", "cerebras-wse"]
30    advanced: true
31  instances:
32    type: "integer"
33  managed:
34    type: "boolean"
35    default: false

The same page as markdown, for an agent: /catalogue/kinds/ml-inference.md