---
id: "cloudflare-pipelines"
type: "component"
kind: "stream-processor"
provider: "cloudflare"
category: "messaging"
docs: "https://developers.cloudflare.com/pipelines/"
url: "https://localhost:3000/catalogue/components/cloudflare-pipelines"
---

# Cloudflare Pipelines

Streaming ingestion with SQL transformations and exactly-once delivery into R2.

| | |
|---|---|
| Provider | [cloudflare](https://localhost:3000/catalogue?provider=cloudflare) |
| Kind | [Stream processor](https://localhost:3000/catalogue/kinds/stream-processor) |
| Category | Messaging |
| Official docs | [https://developers.cloudflare.com/pipelines/](https://developers.cloudflare.com/pipelines/) |

## Properties

| property | type | values | required | kind default | this component |
|---|---|---|---|---|---|
| `checkpointIntervalSeconds` | integer | — | no | — | inherited |
| `exactlyOnce` | boolean | — | no | false | **true** |
| `ingestSource` | enum | `http-endpoint`, `worker-binding` | no | "http-endpoint" | inherited |
| `instances` | integer | — | no | — | inherited |
| `lateDataPolicy` | enum | `drop`, `shift`, `sideline` | no | — | inherited |
| `managed` | boolean | — | no | false | **true** |
| `sinkFormat` | enum | `apache-iceberg`, `parquet`, `json` | no | — | inherited |
| `sqlTransformations` | boolean | — | no | true | inherited |
| `stateBackend` | enum | `memory`, `rocksdb`, `external` | no | "memory" | inherited |
| `watermarkSeconds` | integer | — | no | — | inherited |
| `windowing` | enum | `none`, `event-time`, `processing-time` | no | "none" | **"none"** |

Bold values are the ones this component pins.

## Rules that apply · 1

### Event-time windows without watermark `windowed-aggregation-without-watermark` (warning)

- Page: https://localhost:3000/catalogue/rules/windowed-aggregation-without-watermark.md
- Why: Event-time windows need a watermark to know when a window is complete. Without one the processor either waits forever or closes windows early and silently drops late events.
- Applies to: every node of kind stream-processor
- Message: Stream processor '{{node.id}}' windows by event time but declares no watermark. Late data handling is undefined.

It fires when:

- all of these hold:
  - the node's `windowing` is "event-time"
  - the node's `watermarkSeconds` has no value

## Other products of the same kind · 33

- [Realtime Compute for Apache Flink](https://localhost:3000/catalogue/components/alibaba-flink.md) `alibaba-flink` — alibabacloud
- [Amazon EventBridge Pipes](https://localhost:3000/catalogue/components/aws-eventbridge-pipes.md) `aws-eventbridge-pipes` — aws
- [Amazon Data Firehose](https://localhost:3000/catalogue/components/aws-firehose.md) `aws-firehose` — aws
- [Amazon Managed Service for Apache Flink](https://localhost:3000/catalogue/components/aws-managed-flink.md) `aws-managed-flink` — aws
- [Amazon OpenSearch Ingestion](https://localhost:3000/catalogue/components/aws-opensearch-ingestion.md) `aws-opensearch-ingestion` — aws
- [Azure Stream Analytics](https://localhost:3000/catalogue/components/azure-stream-analytics.md) `azure-stream-analytics` — azure
- [Cloudflare Logpush](https://localhost:3000/catalogue/components/cloudflare-logpush.md) `cloudflare-logpush` — cloudflare
- [Arroyo](https://localhost:3000/catalogue/components/arroyo.md) `arroyo` — default
- [Apache Beam](https://localhost:3000/catalogue/components/beam.md) `beam` — default
- [Bytewax](https://localhost:3000/catalogue/components/bytewax.md) `bytewax` — default
- [Drasi](https://localhost:3000/catalogue/components/drasi.md) `drasi` — default
- [Apache Flink](https://localhost:3000/catalogue/components/flink.md) `flink` — default
- [Kafka Streams](https://localhost:3000/catalogue/components/kafka-streams.md) `kafka-streams` — default
- [ksqlDB](https://localhost:3000/catalogue/components/ksqldb.md) `ksqldb` — default
- [Logstash](https://localhost:3000/catalogue/components/logstash.md) `logstash` — default
- [Materialize](https://localhost:3000/catalogue/components/materialize.md) `materialize` — default
- [Apache NiFi](https://localhost:3000/catalogue/components/nifi.md) `nifi` — default
- [Norikra](https://localhost:3000/catalogue/components/norikra.md) `norikra` — default
- [Redpanda Connect](https://localhost:3000/catalogue/components/redpanda-connect.md) `redpanda-connect` — default
- [RisingWave](https://localhost:3000/catalogue/components/risingwave.md) `risingwave` — default
- [Apache Samza](https://localhost:3000/catalogue/components/samza.md) `samza` — default
- [Snowplow](https://localhost:3000/catalogue/components/snowplow.md) `snowplow` — default
- [Spark Structured Streaming](https://localhost:3000/catalogue/components/spark-streaming.md) `spark-streaming` — default
- [Apache Storm](https://localhost:3000/catalogue/components/storm.md) `storm` — default
- [Generic stream processor](https://localhost:3000/catalogue/components/stream-processor.md) `stream-processor` — default
- [Tremor](https://localhost:3000/catalogue/components/tremor.md) `tremor` — default
- [Google Cloud Dataflow](https://localhost:3000/catalogue/components/google-dataflow.md) `google-dataflow` — gcp
- [Manufacturing Data Engine](https://localhost:3000/catalogue/components/google-manufacturing-data-engine.md) `google-manufacturing-data-engine` — gcp
- [Activity Tracker Event Routing](https://localhost:3000/catalogue/components/ibm-activity-tracker-event-routing.md) `ibm-activity-tracker-event-routing` — ibm
- [IBM Cloud Logs Routing](https://localhost:3000/catalogue/components/ibm-logs-routing.md) `ibm-logs-routing` — ibm
- [IBM Cloud Metrics Routing](https://localhost:3000/catalogue/components/ibm-metrics-routing.md) `ibm-metrics-routing` — ibm
- [OCI Connector Hub](https://localhost:3000/catalogue/components/oci-connector-hub.md) `oci-connector-hub` — oci
- [OCI GoldenGate Stream Analytics](https://localhost:3000/catalogue/components/oci-goldengate-stream-analytics.md) `oci-goldengate-stream-analytics` — oci

## Use it in a design

```yaml
nodes:
  - { id: "cloudflare-pipelines", component: "cloudflare-pipelines" }
```

## Source (`component` YAML, verbatim)

```yaml
# yaml-language-server: $schema=../../schema/component.schema.json
# Cloudflare Pipelines: streams buffer events from an HTTP endpoint or a Worker binding, a SQL
# pipeline transforms them, and a sink writes Iceberg tables or Parquet and JSON files to R2.
# "Pipelines" states it provides exactly-once delivery to R2. Currently open beta, Workers Paid only.
schemaVersion: "1"
type: "component"
id: "cloudflare-pipelines"
kind: "stream-processor"
provider: "cloudflare"
docs: "https://developers.cloudflare.com/pipelines/"
defaults:
  managed: true
  exactlyOnce: true
  windowing: "none"
properties:
  ingestSource:
    type: "enum"
    values: ["http-endpoint", "worker-binding"]
    default: "http-endpoint"
  sinkFormat:
    type: "enum"
    values: ["apache-iceberg", "parquet", "json"]
  sqlTransformations:
    type: "boolean"
    default: true
```

Human page: https://localhost:3000/catalogue/components/cloudflare-pipelines