Apache Beam
Stream processorbeam
Unified batch and streaming programming model executed by a pluggable runner.
- Provider
- default
- Kind
- Stream processor
- Category
- Messaging
- Official docs
- https://beam.apache.org/documentation/basics/
| property | type | values | required | kind default | this component |
|---|---|---|---|---|---|
| checkpointIntervalSeconds | integer | — | no | — | inherited |
| exactlyOnce | boolean | — | no | false | ▸ false |
| instances | integer | — | no | — | inherited |
| lateDataPolicy | enum | drop | shift | sideline | no | — | ▸ drop |
| managed | boolean | — | no | false | ▸ false |
| runner | enum | direct | flink | spark | samza | dataflow | no | direct | inherited |
| sdk | enum | java | python | go | typescript | no | java | inherited |
| stateBackend | enum | memory | rocksdb | external | no | memory | inherited |
| watermarkSeconds | integer | — | no | — | inherited |
| windowing | enum | none | event-time | processing-time | no | none | ▸ none |
Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.
windowed-aggregation-without-watermark
Event-time windows need a watermark to know when a window is complete. Without one the processor either waits forever or closes windows early and silently drops late events.
- — all of these hold:
- — the node's `windowing` is "event-time"
- — the node's `watermarkSeconds` has no value
nodes:
- { id: "beam", component: "beam" }Realtime Compute for Apache Flinkalibaba-flinkAmazon EventBridge Pipesaws-eventbridge-pipesAmazon Data Firehoseaws-firehoseAmazon Managed Service for Apache Flinkaws-managed-flinkAmazon OpenSearch Ingestionaws-opensearch-ingestionAzure Stream Analyticsazure-stream-analyticsCloudflare Logpushcloudflare-logpushCloudflare Pipelinescloudflare-pipelinesArroyoarroyoBytewaxbytewaxDrasidrasiApache FlinkflinkKafka Streamskafka-streamsksqlDBksqldbLogstashlogstashMaterializematerializeApache NiFinifiNorikranorikraRedpanda Connectredpanda-connectRisingWaverisingwaveApache SamzasamzaSnowplowsnowplowSpark Structured Streamingspark-streamingApache StormstormGeneric stream processorstream-processorTremortremorGoogle Cloud Dataflowgoogle-dataflowManufacturing Data Enginegoogle-manufacturing-data-engineActivity Tracker Event Routingibm-activity-tracker-event-routingIBM Cloud Logs Routingibm-logs-routingIBM Cloud Metrics Routingibm-metrics-routingOCI Connector Huboci-connector-hubOCI GoldenGate Stream Analyticsoci-goldengate-stream-analytics
# yaml-language-server: $schema=../../schema/component.schema.json # Apache Beam: unified batch and streaming model executed by a runner (Dataflow, Flink, Spark). # "By default, all data in a PCollection is assigned to the single global window, and late data is # discarded", and the default trigger is event-time based, firing when the watermark estimates all # data has arrived ("Apache Beam, Basics of the Beam model"). Checkpointing and state come from the runner. schemaVersion: "1" type: "component" id: "beam" kind: "stream-processor" provider: "default" docs: "https://beam.apache.org/documentation/basics/" defaults: managed: false windowing: "none" lateDataPolicy: "drop" exactlyOnce: false properties: runner: type: "enum" values: ["direct", "flink", "spark", "samza", "dataflow"] default: "direct" sdk: type: "enum" values: ["java", "python", "go", "typescript"] default: "java"