---
id: "worker"
type: "kind"
category: "compute"
url: "https://localhost:3000/catalogue/kinds/worker"
---

# Worker

Background process triggered by a queue, schedule, stream or batch.

| | |
|---|---|
| Category | Compute |
| Properties | 10 |
| Components | 83 |
| Rules | 3 |

## Properties

| property | type | values | required | default |
|---|---|---|---|---|
| `accelerator` | enum | `none`, `gpu`, `asic` | no | "none" |
| `concurrency` | integer | — | no | — |
| `fanoutStrategy` (advanced) | enum | `none`, `push`, `pull`, `hybrid` | no | "none" |
| `fanoutThreshold` (advanced) | integer | — | no | — |
| `idempotent` | boolean | — | no | false |
| `instances` | integer | — | no | — |
| `managed` | boolean | — | no | false |
| `rateControl` (advanced) | enum | `none`, `fixed`, `aimd` | no | "none" |
| `schedule` | string | — | no | — |
| `trigger` | enum | `queue`, `stream`, `schedule`, `batch`, `manual` | yes | — |

## Rules for this kind · 3

### Push fan-out without a celebrity threshold `push-fanout-without-celebrity-path` (warning)

- Page: https://localhost:3000/catalogue/rules/push-fanout-without-celebrity-path.md
- Why: Pure push fan-out over a social graph is brutal at the tail: one post from an account with millions of followers becomes millions of cache writes. A follower threshold above which readers merge at read time bounds that write amplification.
- Applies to: every node of kind worker
- Message: Worker '{{node.id}}' pushes fan-out over a social graph with no fanoutThreshold. Declare a follower threshold above which timelines are merged at read time.

It fires when:

- all of these hold:
  - the node's `fanoutStrategy` is "push"
  - the node's `fanoutThreshold` has no value
  - the design contains a node of kind graph-database

### Shared worker pool across queues `shared-worker-pool-across-queues` (warning)

- Page: https://localhost:3000/catalogue/rules/shared-worker-pool-across-queues.md
- Why: One worker pool consuming several queues lets a slow or poisoned queue starve the others. That is precisely what the design's per-queue isolation requirement forbids.
- Applies to: every node of kind worker
- Message: Worker '{{node.id}}' consumes more than one queue while the design requires per-queue isolation. Give each queue its own worker pool.

It fires when:

- all of these hold:
  - the node has more than 1 incoming edge from a queue
  - the design requirement `isolation.perQueue` is true

### Stateful node without zone redundancy `stateful-node-without-zone-redundancy` (error)

- Page: https://localhost:3000/catalogue/rules/stateful-node-without-zone-redundancy.md
- Why: A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.
- Applies to: every node in category compute
- Message: Stateful node '{{node.id}}' lives in exactly one availability zone. A zone outage takes its state with it.

It fires when:

- all of these hold:
  - the node's `stateless` is false
  - the node sits inside exactly 1 availability-zone group
  - this does not hold:
    - the node sits inside at least one cell group

## Components · 83

### vendor-neutral

- [Generic worker](https://localhost:3000/catalogue/components/worker.md) `worker` — the kind itself

### alibabacloud

- [SchedulerX](https://localhost:3000/catalogue/components/alibaba-schedulerx.md) `alibaba-schedulerx` — 2 defaults

### aws

- [Amazon AppFlow](https://localhost:3000/catalogue/components/aws-appflow.md) `aws-appflow` — 5 defaults
- [AWS B2B Data Interchange](https://localhost:3000/catalogue/components/aws-b2b-data-interchange.md) `aws-b2b-data-interchange` — 5 defaults
- [AWS Batch](https://localhost:3000/catalogue/components/aws-batch.md) `aws-batch` — 2 defaults
- [AWS Glue DataBrew](https://localhost:3000/catalogue/components/aws-databrew.md) `aws-databrew` — 6 defaults
- [AWS DataSync](https://localhost:3000/catalogue/components/aws-datasync.md) `aws-datasync` — 3 defaults
- [AWS Deadline Cloud](https://localhost:3000/catalogue/components/aws-deadline-cloud.md) `aws-deadline-cloud` — 2 defaults
- [AWS Entity Resolution](https://localhost:3000/catalogue/components/aws-entity-resolution.md) `aws-entity-resolution` — 5 defaults
- [Amazon EventBridge Scheduler](https://localhost:3000/catalogue/components/aws-eventbridge-scheduler.md) `aws-eventbridge-scheduler` — 5 defaults
- [AWS Fault Injection Service](https://localhost:3000/catalogue/components/aws-fis.md) `aws-fis` — 2 defaults
- [AWS Glue crawler](https://localhost:3000/catalogue/components/aws-glue-crawler.md) `aws-glue-crawler` — 5 defaults
- [AWS Glue for Ray job](https://localhost:3000/catalogue/components/aws-glue-ray-job.md) `aws-glue-ray-job` — 5 defaults
- [AWS Application Migration Service](https://localhost:3000/catalogue/components/aws-mgn.md) `aws-mgn` — 2 defaults
- [Amazon MSK Connect](https://localhost:3000/catalogue/components/aws-msk-connect.md) `aws-msk-connect` — 4 defaults
- [Amazon SageMaker AI Training Job](https://localhost:3000/catalogue/components/aws-sagemaker-training-job.md) `aws-sagemaker-training-job` — 2 defaults
- [AWS Systems Manager](https://localhost:3000/catalogue/components/aws-systems-manager.md) `aws-systems-manager` — 2 defaults
- [AWS Thinkbox Deadline 10](https://localhost:3000/catalogue/components/aws-thinkbox-deadline.md) `aws-thinkbox-deadline` — 1 defaults
- [AWS Glue job](https://localhost:3000/catalogue/components/glue-job.md) `glue-job` — 2 defaults

### azure

- [Azure Automation](https://localhost:3000/catalogue/components/azure-automation.md) `azure-automation` — 2 defaults
- [Azure Batch](https://localhost:3000/catalogue/components/azure-batch.md) `azure-batch` — 3 defaults
- [Azure Chaos Studio](https://localhost:3000/catalogue/components/azure-chaos-studio.md) `azure-chaos-studio` — 2 defaults
- [Azure Data Factory](https://localhost:3000/catalogue/components/azure-data-factory.md) `azure-data-factory` — 2 defaults
- [Azure Data Share](https://localhost:3000/catalogue/components/azure-data-share.md) `azure-data-share` — 2 defaults
- [Azure Database Migration Service](https://localhost:3000/catalogue/components/azure-database-migration-service.md) `azure-database-migration-service` — 2 defaults
- [Microsoft Entra Connect Sync](https://localhost:3000/catalogue/components/azure-entra-connect-sync.md) `azure-entra-connect-sync` — 2 defaults
- [Microsoft Graph Data Connect](https://localhost:3000/catalogue/components/azure-graph-data-connect.md) `azure-graph-data-connect` — 2 defaults
- [Azure Storage Mover](https://localhost:3000/catalogue/components/azure-storage-mover.md) `azure-storage-mover` — 2 defaults
- [Azure Update Manager](https://localhost:3000/catalogue/components/azure-update-manager.md) `azure-update-manager` — 2 defaults

### cloudflare

- [Browser Run](https://localhost:3000/catalogue/components/cloudflare-browser-run.md) `cloudflare-browser-run` — 4 defaults
- [Cloudflare Snippets](https://localhost:3000/catalogue/components/cloudflare-snippets.md) `cloudflare-snippets` — 4 defaults
- [Cloudflare Zaraz](https://localhost:3000/catalogue/components/cloudflare-zaraz.md) `cloudflare-zaraz` — 2 defaults

### default

- [Airbyte](https://localhost:3000/catalogue/components/airbyte.md) `airbyte` — 3 defaults
- [Airflow DAG](https://localhost:3000/catalogue/components/airflow-dag.md) `airflow-dag` — 2 defaults
- [Chaos Mesh](https://localhost:3000/catalogue/components/chaos-mesh.md) `chaos-mesh` — 3 defaults
- [Chunkr](https://localhost:3000/catalogue/components/chunkr.md) `chunkr` — 3 defaults
- [Container worker](https://localhost:3000/catalogue/components/container-worker.md) `container-worker` — 2 defaults
- [Cron job](https://localhost:3000/catalogue/components/cron-job.md) `cron-job` — 1 defaults
- [Daytona](https://localhost:3000/catalogue/components/daytona.md) `daytona` — 3 defaults
- [E2B](https://localhost:3000/catalogue/components/e2b.md) `e2b` — 4 defaults
- [Embulk](https://localhost:3000/catalogue/components/embulk.md) `embulk` — 3 defaults
- [Fan-out daemon](https://localhost:3000/catalogue/components/fanout-daemon.md) `fanout-daemon` — 2 defaults
- [Fivetran](https://localhost:3000/catalogue/components/fivetran.md) `fivetran` — 3 defaults
- [Litmus](https://localhost:3000/catalogue/components/litmus.md) `litmus` — 3 defaults
- [LlamaParse](https://localhost:3000/catalogue/components/llamaparse.md) `llamaparse` — 4 defaults
- [Meltano](https://localhost:3000/catalogue/components/meltano.md) `meltano` — 2 defaults
- [Northflank Sandboxes](https://localhost:3000/catalogue/components/northflank-sandbox.md) `northflank-sandbox` — 3 defaults
- [Reducto](https://localhost:3000/catalogue/components/reducto.md) `reducto` — 3 defaults
- [Runloop](https://localhost:3000/catalogue/components/runloop.md) `runloop` — 4 defaults
- [Spark job](https://localhost:3000/catalogue/components/spark-job.md) `spark-job` — 1 defaults
- [Stitch](https://localhost:3000/catalogue/components/stitch.md) `stitch` — 2 defaults
- [Unstructured API](https://localhost:3000/catalogue/components/unstructured-io.md) `unstructured-io` — 3 defaults
- [Vercel Sandbox](https://localhost:3000/catalogue/components/vercel-sandbox.md) `vercel-sandbox` — 4 defaults

### digitalocean

- [DigitalOcean Functions](https://localhost:3000/catalogue/components/digitalocean-functions.md) `digitalocean-functions` — 3 defaults

### gcp

- [Dataproc job](https://localhost:3000/catalogue/components/dataproc-job.md) `dataproc-job` — 2 defaults
- [Batch](https://localhost:3000/catalogue/components/google-batch.md) `google-batch` — 5 defaults
- [BigQuery Data Transfer Service](https://localhost:3000/catalogue/components/google-bigquery-data-transfer.md) `google-bigquery-data-transfer` — 3 defaults
- [Cloud Data Fusion](https://localhost:3000/catalogue/components/google-cloud-data-fusion.md) `google-cloud-data-fusion` — 2 defaults
- [Cloud Run job](https://localhost:3000/catalogue/components/google-cloud-run-job.md) `google-cloud-run-job` — 4 defaults
- [Cloud Run worker pool](https://localhost:3000/catalogue/components/google-cloud-run-worker-pool.md) `google-cloud-run-worker-pool` — 4 defaults
- [Cloud Scheduler](https://localhost:3000/catalogue/components/google-cloud-scheduler.md) `google-cloud-scheduler` — 3 defaults
- [Dual Run](https://localhost:3000/catalogue/components/google-dual-run.md) `google-dual-run` — 2 defaults
- [Migrate to Containers](https://localhost:3000/catalogue/components/google-migrate-to-containers.md) `google-migrate-to-containers` — 2 defaults
- [Migrate to Virtual Machines](https://localhost:3000/catalogue/components/google-migrate-to-virtual-machines.md) `google-migrate-to-virtual-machines` — 3 defaults
- [Storage Transfer Service](https://localhost:3000/catalogue/components/google-storage-transfer-service.md) `google-storage-transfer-service` — 3 defaults
- [Vertex AI Training](https://localhost:3000/catalogue/components/google-vertex-ai-training.md) `google-vertex-ai-training` — 5 defaults

### ibm

- [IBM DataStage as a Service](https://localhost:3000/catalogue/components/ibm-datastage.md) `ibm-datastage` — 2 defaults

### k8s

- [CronJob](https://localhost:3000/catalogue/components/k8s-cronjob.md) `k8s-cronjob` — 2 defaults
- [Job](https://localhost:3000/catalogue/components/k8s-job.md) `k8s-job` — 3 defaults

### oci

- [OCI Batch](https://localhost:3000/catalogue/components/oci-batch.md) `oci-batch` — 2 defaults
- [Oracle Cloud Migrations](https://localhost:3000/catalogue/components/oci-cloud-migrations.md) `oci-cloud-migrations` — 2 defaults
- [OCI Data Integration](https://localhost:3000/catalogue/components/oci-data-integration.md) `oci-data-integration` — 2 defaults
- [Oracle Data Integrator on OCI](https://localhost:3000/catalogue/components/oci-data-integrator.md) `oci-data-integrator` — 1 defaults
- [OCI Data Science Job](https://localhost:3000/catalogue/components/oci-data-science-job.md) `oci-data-science-job` — 6 defaults
- [Oracle Data Transforms](https://localhost:3000/catalogue/components/oci-data-transforms.md) `oci-data-transforms` — 2 defaults
- [OCI Database Migration](https://localhost:3000/catalogue/components/oci-database-migration.md) `oci-database-migration` — 2 defaults
- [Exadata Fleet Update](https://localhost:3000/catalogue/components/oci-exadata-fleet-update.md) `oci-exadata-fleet-update` — 2 defaults
- [OCI Fleet Application Management](https://localhost:3000/catalogue/components/oci-fleet-application-management.md) `oci-fleet-application-management` — 2 defaults
- [OCI Guarded Data Pipelines](https://localhost:3000/catalogue/components/oci-guarded-data-pipelines.md) `oci-guarded-data-pipelines` — 1 defaults
- [Oracle Managed File Transfer](https://localhost:3000/catalogue/components/oci-managed-file-transfer.md) `oci-managed-file-transfer` — 2 defaults
- [OCI OS Management Hub](https://localhost:3000/catalogue/components/oci-os-management-hub.md) `oci-os-management-hub` — 2 defaults
- [OCI Resource Scheduler](https://localhost:3000/catalogue/components/oci-resource-scheduler.md) `oci-resource-scheduler` — 2 defaults

### openstack

- [OpenStack Storlets](https://localhost:3000/catalogue/components/openstack-storlets.md) `openstack-storlets` — 2 defaults

## Source (`kind` YAML, verbatim)

```yaml
# yaml-language-server: $schema=../schema/kind.schema.json
# Background process with no request path: queue consumer, scheduled job, batch or ETL job, fan-out daemon.
# trigger replaces the separate scheduled-job and etl-job candidates.
schemaVersion: "1"
type: "kind"
id: "worker"
category: "compute"
properties:
  trigger:
    type: "enum"
    values: ["queue", "stream", "schedule", "batch", "manual"]
    required: true
  instances:
    type: "integer"
  idempotent:
    type: "boolean"
    default: false
  concurrency:
    type: "integer"
  schedule:
    type: "string"
  accelerator:
    type: "enum"
    values: ["none", "gpu", "asic"]
    default: "none"
  rateControl:
    type: "enum"
    values: ["none", "fixed", "aimd"]
    default: "none"
    advanced: true
  fanoutStrategy:
    type: "enum"
    values: ["none", "push", "pull", "hybrid"]
    default: "none"
    advanced: true
  fanoutThreshold:
    type: "integer"
    advanced: true
  managed:
    type: "boolean"
    default: false
```

Human page: https://localhost:3000/catalogue/kinds/worker