Amazon Managed Service for Apache Flink

aws Stream processoraws-managed-flink

Managed Apache Flink cluster that runs stateful stream processing applications with checkpointing and automatic scaling.

Provider: aws
Kind: Stream processor
Category: Messaging
Official docs: https://docs.aws.amazon.com/managed-flink/latest/java/how-fault.html
Also known as: Kinesis Data Analytics

Properties · 4 set by this component

property	type	values	required	kind default	this component
apiFlavour · component property	enum	datastream \| table \| sql \| studio-notebook	no	datastream	inherited
autoScaling · component property	boolean	—	no	true	inherited
checkpointIntervalSeconds	integer	—	no	—	▸ 60
exactlyOnce	boolean	—	no	false	▸ true
instances	integer	—	no	—	▸ 1
lateDataPolicy	enum	drop \| shift \| sideline	no	—	inherited
managed	boolean	—	no	false	▸ true
minPauseBetweenCheckpointsMs · component property	integer	—	no	5000	inherited
parallelismPerKpu · component property	integer	—	no	1	inherited
stateBackend	enum	memory \| rocksdb \| external	no	memory	inherited
watermarkSeconds	integer	—	no	—	inherited
windowing	enum	none \| event-time \| processing-time	no	none	inherited

Rules that apply · 1

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Event-time windows without watermarkwarning

windowed-aggregation-without-watermark

Event-time windows need a watermark to know when a window is complete. Without one the processor either waits forever or closes windows early and silently drops late events.

— all of these hold:
— the node's `windowing` is "event-time"
— the node's `watermarkSeconds` has no value

Use it in a design

nodes:
  - { id: "managed-flink", component: "aws-managed-flink" }

Other Stream processor products · 33

Realtime Compute for Apache Flinkalibaba-flinkalibabacloudstream-processor Amazon EventBridge Pipesaws-eventbridge-pipesawsstream-processor Amazon Data Firehoseaws-firehoseawsstream-processor Amazon OpenSearch Ingestionaws-opensearch-ingestionawsstream-processor Azure Stream Analyticsazure-stream-analyticsazurestream-processor Cloudflare Logpushcloudflare-logpushcloudflarestream-processor Cloudflare Pipelinescloudflare-pipelinescloudflarestream-processor Arroyoarroyostream-processor Apache Beambeamstream-processor Bytewaxbytewaxstream-processor Drasidrasistream-processor Apache Flinkflinkstream-processor Kafka Streamskafka-streamsstream-processor ksqlDBksqldbstream-processor Logstashlogstashstream-processor Materializematerializestream-processor Apache NiFinifistream-processor Norikranorikrastream-processor Redpanda Connectredpanda-connectstream-processor RisingWaverisingwavestream-processor Apache Samzasamzastream-processor Snowplowsnowplowstream-processor Spark Structured Streamingspark-streamingstream-processor Apache Stormstormstream-processor Generic stream processorstream-processorstream-processor Tremortremorstream-processor Google Cloud Dataflowgoogle-dataflowgcpstream-processor Manufacturing Data Enginegoogle-manufacturing-data-enginegcpstream-processor Activity Tracker Event Routingibm-activity-tracker-event-routingibmstream-processor IBM Cloud Logs Routingibm-logs-routingibmstream-processor IBM Cloud Metrics Routingibm-metrics-routingibmstream-processor OCI Connector Huboci-connector-hubocistream-processor OCI GoldenGate Stream Analyticsoci-goldengate-stream-analyticsocistream-processor

Source

1# yaml-language-server: $schema=../../schema/component.schema.json
2# Amazon Managed Service for Apache Flink: single-tenant Flink cluster run by AWS across
3# Availability Zones. "Implement fault tolerance" gives the DEFAULT checkpoint configuration
4# (enabled, 60000 ms interval, 5000 ms minimum pause); "Implement application scaling" gives
5# Parallelism 1, ParallelismPerKPU 1 and automatic scaling enabled by default.
6schemaVersion: "1"
7type: "component"
8id: "aws-managed-flink"
9kind: "stream-processor"
10provider: "aws"
11docs: "https://docs.aws.amazon.com/managed-flink/latest/java/how-fault.html"
12aliases: ["Kinesis Data Analytics"]
13defaults:
14  managed: true
15  checkpointIntervalSeconds: 60
16  exactlyOnce: true
17  instances: 1
18properties:
19  parallelismPerKpu:
20    type: "integer"
21    default: 1
22  autoScaling:
23    type: "boolean"
24    default: true
25  minPauseBetweenCheckpointsMs:
26    type: "integer"
27    default: 5000
28  apiFlavour:
29    type: "enum"
30    values: ["datastream", "table", "sql", "studio-notebook"]
31    default: "datastream"

The same page as markdown, for an agent: /catalogue/components/aws-managed-flink.md