Stream processor

messagingstream-processor

Continuous computation over streams with windows and state.

Category: Messaging
Properties: 8
Components: 34
Rules: 1

Properties · 8 defined by this kind

A node of this kind resolves each property from its own props, then the component's defaults, then the default below.

property	type	values	required	default
checkpointIntervalSeconds	integer	—	no	—
exactlyOnce	boolean	—	no	false
instances	integer	—	no	—
lateDataPolicy	enum	drop \| shift \| sideline	no	—
managed	boolean	—	no	false
stateBackend	enum	memory \| rocksdb \| external	no	memory
watermarkSeconds	integer	—	no	—
windowing	enum	none \| event-time \| processing-time	no	none

Rules for this kind · 1

Event-time windows without watermarkwarning

windowed-aggregation-without-watermark

Event-time windows need a watermark to know when a window is complete. Without one the processor either waits forever or closes windows early and silently drops late events.

— all of these hold:
— the node's `windowing` is "event-time"
— the node's `watermarkSeconds` has no value

Components · 34

vendor-neutral

Generic stream processorstream-processorgenericthe kind itself

alibabacloud

Realtime Compute for Apache Flinkalibaba-flink2 defaults

aws

Amazon EventBridge Pipesaws-eventbridge-pipes3 defaults Amazon Data Firehoseaws-firehose4 defaults Amazon Managed Service for Apache Flinkaws-managed-flink4 defaults Amazon OpenSearch Ingestionaws-opensearch-ingestion4 defaults

azure

Azure Stream Analyticsazure-stream-analytics5 defaults

cloudflare

Cloudflare Logpushcloudflare-logpush1 defaults Cloudflare Pipelinescloudflare-pipelines3 defaults

default

Arroyoarroyo4 defaults Apache Beambeam4 defaults Bytewaxbytewax4 defaults Drasidrasi4 defaults Apache Flinkflink4 defaults Kafka Streamskafka-streams4 defaults ksqlDBksqldb5 defaults Logstashlogstash4 defaults Materializematerialize3 defaults Apache NiFinifi4 defaults Norikranorikra3 defaults Redpanda Connectredpanda-connect4 defaults RisingWaverisingwave4 defaults Apache Samzasamza4 defaults Snowplowsnowplow2 defaults Spark Structured Streamingspark-streaming3 defaults Apache Stormstorm4 defaults Tremortremor4 defaults

gcp

Google Cloud Dataflowgoogle-dataflow5 defaults Manufacturing Data Enginegoogle-manufacturing-data-engine2 defaults

ibm

Activity Tracker Event Routingibm-activity-tracker-event-routing1 defaults IBM Cloud Logs Routingibm-logs-routing1 defaults IBM Cloud Metrics Routingibm-metrics-routing1 defaults

oci

OCI Connector Huboci-connector-hub3 defaults OCI GoldenGate Stream Analyticsoci-goldengate-stream-analytics1 defaults

Source

1# yaml-language-server: $schema=../schema/kind.schema.json
2# Continuous computation over one or more streams (Flink, Samza, Kafka Streams, Dataflow).
3# watermarkSeconds is optional with no default so windowed-aggregation-without-watermark can fire.
4schemaVersion: "1"
5type: "kind"
6id: "stream-processor"
7category: "messaging"
8properties:
9  windowing:
10    type: "enum"
11    values: ["none", "event-time", "processing-time"]
12    default: "none"
13  watermarkSeconds:
14    type: "integer"
15  checkpointIntervalSeconds:
16    type: "integer"
17  exactlyOnce:
18    type: "boolean"
19    default: false
20  stateBackend:
21    type: "enum"
22    values: ["memory", "rocksdb", "external"]
23    default: "memory"
24  lateDataPolicy:
25    type: "enum"
26    values: ["drop", "shift", "sideline"]
27  instances:
28    type: "integer"
29  managed:
30    type: "boolean"
31    default: false

The same page as markdown, for an agent: /catalogue/kinds/stream-processor.md