Apache Airflow

Workflow engineairflow

Open source workflow platform whose scheduler, dag processor and API server run Python DAGs through a local executor against a metadata database.

Provider: default
Kind: Workflow engine
Category: Compute
Official docs: https://airflow.apache.org/docs/apache-airflow/stable/index.html

Properties · 4 set by this component

property	type	values	required	kind default	this component
apiPort · component property	integer	—	no	8080	inherited
compensation	boolean	—	no	false	▸ false
durable	boolean	—	no	true	▸ true
executor · component property	enum	local \| celery \| kubernetes \| edge	no	local	inherited
humanReview	boolean	—	no	false	inherited
managed	boolean	—	no	false	▸ false
maxActiveTasksPerDag · component property	integer	—	no	16	inherited
metadataDatabase · component property	enum	sqlite \| postgres \| mysql	no	sqlite	inherited
parallelism · component property	integer	—	no	32	inherited
pattern	enum	saga \| tcc \| two-phase-commit \| dag	no	dag	▸ dag
stepsIdempotent	boolean	—	no	false	inherited

Rules that apply · 1

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Stateful node without zone redundancyerror

stateful-node-without-zone-redundancy

A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.

— all of these hold:
— the node's `stateless` is false
— the node sits inside exactly 1 availability-zone group
— this does not hold:
— the node sits inside at least one cell group

Use it in a design

nodes:
  - { id: "airflow", component: "airflow" }

Other Workflow engine products · 44

CloudFlowalibaba-cloudflowalibabacloudworkflow-engine CloudOps Orchestration Servicealibaba-oosalibabacloudworkflow-engine Amazon Augmented AI (A2I)aws-augmented-aiawsworkflow-engine Amazon Managed Workflows for Apache Airflowaws-mwaaawsworkflow-engine AWS Systems Manager Incident Manager and Change Manageraws-ssm-incident-and-change-managerawsworkflow-engine AWS Step Functionsaws-step-functionsawsworkflow-engine Amazon Simple Workflow Serviceaws-swfawsworkflow-engine Azure Durable Functionsazure-durable-functionsazureworkflow-engine Azure Logic Appsazure-logic-appsazureworkflow-engine Cloudflare Workflowscloudflare-workflowscloudflareworkflow-engine Argo Workflowsargo-workflowsworkflow-engine Astronomer Astroastronomerworkflow-engine Cadencecadenceworkflow-engine Camunda 8camundaworkflow-engine Conductorconductorworkflow-engine Dagsterdagsterworkflow-engine Digdagdigdagworkflow-engine Flowableflowableworkflow-engine Inngestinngestworkflow-engine jBPMjbpmworkflow-engine Kestrakestraworkflow-engine Luigiluigiworkflow-engine n8nn8nworkflow-engine Node-REDnode-redworkflow-engine Prefectprefectworkflow-engine Prefect Cloudprefect-cloudworkflow-engine Restaterestateworkflow-engine Temporaltemporalworkflow-engine Temporal Cloudtemporal-cloudworkflow-engine Trigger.devtrigger-devworkflow-engine Windmillwindmillworkflow-engine Workatoworkatoworkflow-engine Generic workflow engineworkflow-engineworkflow-engine Zapierzapierworkflow-engine Application Integrationgoogle-application-integrationgcpworkflow-engine Cloud Composer (Managed Service for Apache Airflow)google-cloud-composergcpworkflow-engine Workflowsgoogle-cloud-workflowsgcpworkflow-engine Orchestration Pipelinesgoogle-orchestration-pipelinesgcpworkflow-engine IBM App Connectibm-app-connectibmworkflow-engine Oracle Integration 3oci-integrationociworkflow-engine

and 4 more → Workflow engine page

Source

1# yaml-language-server: $schema=../../schema/component.schema.json
2# Apache Airflow: open source platform for developing, scheduling and monitoring workflows written
3# as Python DAGs ("What is Airflow?"). "Architecture Overview" makes the scheduler, dag processor,
4# dag bundle, API server and metadata database the required parts and the triggerer and workers
5# optional. "Executor" gives LocalExecutor as the shipped executor; "Set up a Database Backend"
6# gives SQLite as the backend on first install and says it must never be used for production.
7# "Configuration Reference" gives parallelism 32, max_active_tasks_per_dag 16 and API port 8080.
8schemaVersion: "1"
9type: "component"
10id: "airflow"
11kind: "workflow-engine"
12provider: "default"
13docs: "https://airflow.apache.org/docs/apache-airflow/stable/index.html"
14defaults:
15  durable: true
16  pattern: "dag"
17  compensation: false
18  managed: false
19properties:
20  executor:
21    type: "enum"
22    values: ["local", "celery", "kubernetes", "edge"]
23    default: "local"
24  metadataDatabase:
25    type: "enum"
26    values: ["sqlite", "postgres", "mysql"]
27    default: "sqlite"
28  parallelism:
29    type: "integer"
30    default: 32
31  maxActiveTasksPerDag:
32    type: "integer"
33    default: 16
34  apiPort:
35    type: "integer"
36    default: 8080

The same page as markdown, for an agent: /catalogue/components/airflow.md