Worker
computeworker
Background process triggered by a queue, schedule, stream or batch.
- Category
- Compute
- Properties
- 10
- Components
- 83
- Rules
- 3
A node of this kind resolves each property from its own props, then the component's defaults, then the default below.
| property | type | values | required | default |
|---|---|---|---|---|
| accelerator | enum | none | gpu | asic | no | none |
| concurrency | integer | — | no | — |
| fanoutStrategy | enum | none | push | pull | hybrid | no | none |
| fanoutThreshold | integer | — | no | — |
| idempotent | boolean | — | no | false |
| instances | integer | — | no | — |
| managed | boolean | — | no | false |
| rateControl | enum | none | fixed | aimd | no | none |
| schedule | string | — | no | — |
| trigger | enum | queue | stream | schedule | batch | manual | yes | — |
push-fanout-without-celebrity-path
Pure push fan-out over a social graph is brutal at the tail: one post from an account with millions of followers becomes millions of cache writes. A follower threshold above which readers merge at read time bounds that write amplification.
- — all of these hold:
- — the node's `fanoutStrategy` is "push"
- — the node's `fanoutThreshold` has no value
- — the design contains a node of kind graph-database
shared-worker-pool-across-queues
One worker pool consuming several queues lets a slow or poisoned queue starve the others. That is precisely what the design's per-queue isolation requirement forbids.
- — all of these hold:
- — the node has more than 1 incoming edge from a queue
- — the design requirement `isolation.perQueue` is true
stateful-node-without-zone-redundancy
A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.
- — all of these hold:
- — the node's `stateless` is false
- — the node sits inside exactly 1 availability-zone group
- — this does not hold:
- — the node sits inside at least one cell group
vendor-neutral
alibabacloud
aws
Amazon AppFlowaws-appflowAWS B2B Data Interchangeaws-b2b-data-interchangeAWS Batchaws-batchAWS Glue DataBrewaws-databrewAWS DataSyncaws-datasyncAWS Deadline Cloudaws-deadline-cloudAWS Entity Resolutionaws-entity-resolutionAmazon EventBridge Scheduleraws-eventbridge-schedulerAWS Fault Injection Serviceaws-fisAWS Glue crawleraws-glue-crawlerAWS Glue for Ray jobaws-glue-ray-jobAWS Application Migration Serviceaws-mgnAmazon MSK Connectaws-msk-connectAmazon SageMaker AI Training Jobaws-sagemaker-training-jobAWS Systems Manageraws-systems-managerAWS Thinkbox Deadline 10aws-thinkbox-deadlineAWS Glue jobglue-job
azure
Azure Automationazure-automationAzure Batchazure-batchAzure Chaos Studioazure-chaos-studioAzure Data Factoryazure-data-factoryAzure Data Shareazure-data-shareAzure Database Migration Serviceazure-database-migration-serviceMicrosoft Entra Connect Syncazure-entra-connect-syncMicrosoft Graph Data Connectazure-graph-data-connectAzure Storage Moverazure-storage-moverAzure Update Managerazure-update-manager
cloudflare
default
AirbyteairbyteAirflow DAGairflow-dagChaos Meshchaos-meshChunkrchunkrContainer workercontainer-workerCron jobcron-jobDaytonadaytonaE2Be2bEmbulkembulkFan-out daemonfanout-daemonFivetranfivetranLitmuslitmusLlamaParsellamaparseMeltanomeltanoNorthflank Sandboxesnorthflank-sandboxReductoreductoRunlooprunloopSpark jobspark-jobStitchstitchUnstructured APIunstructured-ioVercel Sandboxvercel-sandbox
digitalocean
gcp
Dataproc jobdataproc-jobBatchgoogle-batchBigQuery Data Transfer Servicegoogle-bigquery-data-transferCloud Data Fusiongoogle-cloud-data-fusionCloud Run jobgoogle-cloud-run-jobCloud Run worker poolgoogle-cloud-run-worker-poolCloud Schedulergoogle-cloud-schedulerDual Rungoogle-dual-runMigrate to Containersgoogle-migrate-to-containersMigrate to Virtual Machinesgoogle-migrate-to-virtual-machinesStorage Transfer Servicegoogle-storage-transfer-serviceVertex AI Traininggoogle-vertex-ai-training
oci
OCI Batchoci-batchOracle Cloud Migrationsoci-cloud-migrationsOCI Data Integrationoci-data-integrationOracle Data Integrator on OCIoci-data-integratorOCI Data Science Joboci-data-science-jobOracle Data Transformsoci-data-transformsOCI Database Migrationoci-database-migrationExadata Fleet Updateoci-exadata-fleet-updateOCI Fleet Application Managementoci-fleet-application-managementOCI Guarded Data Pipelinesoci-guarded-data-pipelinesOracle Managed File Transferoci-managed-file-transferOCI OS Management Huboci-os-management-hubOCI Resource Scheduleroci-resource-scheduler
openstack
# yaml-language-server: $schema=../schema/kind.schema.json # Background process with no request path: queue consumer, scheduled job, batch or ETL job, fan-out daemon. # trigger replaces the separate scheduled-job and etl-job candidates. schemaVersion: "1" type: "kind" id: "worker" category: "compute" properties: trigger: type: "enum" values: ["queue", "stream", "schedule", "batch", "manual"] required: true instances: type: "integer" idempotent: type: "boolean" default: false concurrency: type: "integer" schedule: type: "string" accelerator: type: "enum" values: ["none", "gpu", "asic"] default: "none" rateControl: type: "enum" values: ["none", "fixed", "aimd"] default: "none" advanced: true fanoutStrategy: type: "enum" values: ["none", "push", "pull", "hybrid"] default: "none" advanced: true fanoutThreshold: type: "integer" advanced: true managed: type: "boolean" default: false