Batch
Managed batch scheduler that provisions and tears down Compute Engine VMs for each job.
- Provider
- gcp
- Kind
- Worker
- Category
- Compute
- Official docs
- https://cloud.google.com/batch/docs/get-started
| property | type | values | required | kind default | this component |
|---|---|---|---|---|---|
| accelerator | enum | none | gpu | asic | no | none | ▸ none |
| bootDiskType | enum | pd-balanced | pd-standard | pd-ssd | no | pd-balanced | inherited |
| concurrency | integer | — | no | — | ▸ 1 |
| fanoutStrategy | enum | none | push | pull | hybrid | no | none | inherited |
| fanoutThreshold | integer | — | no | — | inherited |
| idempotent | boolean | — | no | false | inherited |
| instances | integer | — | no | — | inherited |
| managed | boolean | — | no | false | ▸ true |
| maxRetryCount | integer | — | no | 0 | inherited |
| provisioningModel | enum | standard | spot | no | standard | inherited |
| rateControl | enum | none | fixed | aimd | no | none | ▸ none |
| schedule | string | — | no | — | inherited |
| taskCount | integer | — | no | 1 | inherited |
| trigger | enum | queue | stream | schedule | batch | manual | yes | — | ▸ batch |
Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.
push-fanout-without-celebrity-path
Pure push fan-out over a social graph is brutal at the tail: one post from an account with millions of followers becomes millions of cache writes. A follower threshold above which readers merge at read time bounds that write amplification.
- — all of these hold:
- — the node's `fanoutStrategy` is "push"
- — the node's `fanoutThreshold` has no value
- — the design contains a node of kind graph-database
shared-worker-pool-across-queues
One worker pool consuming several queues lets a slow or poisoned queue starve the others. That is precisely what the design's per-queue isolation requirement forbids.
- — all of these hold:
- — the node has more than 1 incoming edge from a queue
- — the design requirement `isolation.perQueue` is true
stateful-node-without-zone-redundancy
A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.
- — all of these hold:
- — the node's `stateless` is false
- — the node sits inside exactly 1 availability-zone group
- — this does not hold:
- — the node sits inside at least one cell group
nodes:
- { id: "batch", component: "google-batch" }SchedulerXalibaba-schedulerxAmazon AppFlowaws-appflowAWS B2B Data Interchangeaws-b2b-data-interchangeAWS Batchaws-batchAWS Glue DataBrewaws-databrewAWS DataSyncaws-datasyncAWS Deadline Cloudaws-deadline-cloudAWS Entity Resolutionaws-entity-resolutionAmazon EventBridge Scheduleraws-eventbridge-schedulerAWS Fault Injection Serviceaws-fisAWS Glue crawleraws-glue-crawlerAWS Glue for Ray jobaws-glue-ray-jobAWS Application Migration Serviceaws-mgnAmazon MSK Connectaws-msk-connectAmazon SageMaker AI Training Jobaws-sagemaker-training-jobAWS Systems Manageraws-systems-managerAWS Thinkbox Deadline 10aws-thinkbox-deadlineAWS Glue jobglue-jobAzure Automationazure-automationAzure Batchazure-batchAzure Chaos Studioazure-chaos-studioAzure Data Factoryazure-data-factoryAzure Data Shareazure-data-shareAzure Database Migration Serviceazure-database-migration-serviceMicrosoft Entra Connect Syncazure-entra-connect-syncMicrosoft Graph Data Connectazure-graph-data-connectAzure Storage Moverazure-storage-moverAzure Update Managerazure-update-managerBrowser Runcloudflare-browser-runCloudflare Snippetscloudflare-snippetsCloudflare Zarazcloudflare-zarazAirbyteairbyteAirflow DAGairflow-dagChaos Meshchaos-meshChunkrchunkrContainer workercontainer-workerCron jobcron-jobDaytonadaytonaE2Be2bEmbulkembulk
and 42 more → Worker page# yaml-language-server: $schema=../../schema/component.schema.json # Batch job: a fully managed batch scheduler that creates and deletes Compute Engine VMs in a regional # managed instance group for the life of the job ("Get started with Batch"). Unset job fields default # to taskCount 1, parallelism 1, maxRetryCount 0, 2 vCPU and 2 GB per task; boot disks default to the # Batch Debian image (COS for container-only jobs) on pd-balanced ("VM OS environment overview"). schemaVersion: "1" type: "component" id: "google-batch" kind: "worker" provider: "gcp" docs: "https://cloud.google.com/batch/docs/get-started" defaults: managed: true trigger: "batch" concurrency: 1 accelerator: "none" rateControl: "none" properties: taskCount: type: "integer" default: 1 maxRetryCount: type: "integer" default: 0 provisioningModel: type: "enum" values: ["standard", "spot"] default: "standard" bootDiskType: type: "enum" values: ["pd-balanced", "pd-standard", "pd-ssd"] default: "pd-balanced"