---
id: "oci-data-flow"
type: "component"
kind: "big-data-cluster"
provider: "oci"
category: "compute"
docs: "https://docs.oracle.com/en-us/iaas/data-flow/using/home.htm"
url: "https://localhost:3000/catalogue/components/oci-data-flow"
---

# OCI Data Flow

Serverless Apache Spark service that runs applications without provisioning or managing a cluster.

| | |
|---|---|
| Provider | [oci](https://localhost:3000/catalogue?provider=oci) |
| Kind | [Big data cluster](https://localhost:3000/catalogue/kinds/big-data-cluster) |
| Category | Compute |
| Official docs | [https://docs.oracle.com/en-us/iaas/data-flow/using/home.htm](https://docs.oracle.com/en-us/iaas/data-flow/using/home.htm) |

## Properties

| property | type | values | required | kind default | this component |
|---|---|---|---|---|---|
| `autoscaling` | boolean | — | no | false | inherited |
| `engine` | enum | `spark`, `hadoop`, `presto`, `flink`, `hpc`, `ray`, `dask`, `kubernetes-batch` | no | — | **"spark"** |
| `executors` | integer | — | no | — | inherited |
| `language` | enum | `java`, `scala`, `python`, `sql`, `spark-submit` | no | — | inherited |
| `managed` | boolean | — | no | false | **true** |
| `poolBacked` | boolean | — | no | false | inherited |
| `runtimeVersion` | string | — | no | — | inherited |
| `spotInstances` | boolean | — | no | false | inherited |
| `workerNodes` | integer | — | no | — | inherited |
| `zoneRedundant` | boolean | — | no | — | inherited |

Bold values are the ones this component pins.

## Rules that apply · 1

### Stateful node without zone redundancy `stateful-node-without-zone-redundancy` (error)

- Page: https://localhost:3000/catalogue/rules/stateful-node-without-zone-redundancy.md
- Why: A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.
- Applies to: every node in category compute
- Message: Stateful node '{{node.id}}' lives in exactly one availability zone. A zone outage takes its state with it.

It fires when:

- all of these hold:
  - the node's `stateless` is false
  - the node sits inside exactly 1 availability-zone group
  - this does not hold:
    - the node sits inside at least one cell group

## Other products of the same kind · 25

- [Elastic High Performance Computing](https://localhost:3000/catalogue/components/alibaba-e-hpc.md) `alibaba-e-hpc` — alibabacloud
- [E-MapReduce](https://localhost:3000/catalogue/components/alibaba-e-mapreduce.md) `alibaba-e-mapreduce` — alibabacloud
- [AWS Clean Rooms](https://localhost:3000/catalogue/components/aws-clean-rooms.md) `aws-clean-rooms` — aws
- [Amazon EMR](https://localhost:3000/catalogue/components/aws-emr.md) `aws-emr` — aws
- [Amazon EMR Serverless](https://localhost:3000/catalogue/components/aws-emr-serverless.md) `aws-emr-serverless` — aws
- [AWS Parallel Computing Service](https://localhost:3000/catalogue/components/aws-pcs.md) `aws-pcs` — aws
- [Amazon SageMaker HyperPod](https://localhost:3000/catalogue/components/aws-sagemaker-hyperpod.md) `aws-sagemaker-hyperpod` — aws
- [Azure CycleCloud](https://localhost:3000/catalogue/components/azure-cyclecloud.md) `azure-cyclecloud` — azure
- [Azure Databricks](https://localhost:3000/catalogue/components/azure-databricks.md) `azure-databricks` — azure
- [Azure HDInsight](https://localhost:3000/catalogue/components/azure-hdinsight.md) `azure-hdinsight` — azure
- [Armada](https://localhost:3000/catalogue/components/armada.md) `armada` — default
- [Generic big data cluster](https://localhost:3000/catalogue/components/big-data-cluster.md) `big-data-cluster` — default
- [Dask](https://localhost:3000/catalogue/components/dask.md) `dask` — default
- [Databricks](https://localhost:3000/catalogue/components/databricks.md) `databricks` — default
- [Apache Hadoop YARN](https://localhost:3000/catalogue/components/hadoop-yarn.md) `hadoop-yarn` — default
- [Ray](https://localhost:3000/catalogue/components/ray.md) `ray` — default
- [Slurm](https://localhost:3000/catalogue/components/slurm.md) `slurm` — default
- [Apache Spark cluster](https://localhost:3000/catalogue/components/spark-cluster.md) `spark-cluster` — default
- [Volcano](https://localhost:3000/catalogue/components/volcano.md) `volcano` — default
- [Cluster Director](https://localhost:3000/catalogue/components/google-cluster-director.md) `google-cluster-director` — gcp
- [Dataproc](https://localhost:3000/catalogue/components/google-dataproc.md) `google-dataproc` — gcp
- [Managed Service for Apache Spark (serverless)](https://localhost:3000/catalogue/components/google-dataproc-serverless.md) `google-dataproc-serverless` — gcp
- [IBM Analytics Engine](https://localhost:3000/catalogue/components/ibm-analytics-engine.md) `ibm-analytics-engine` — ibm
- [OCI AI Data Platform](https://localhost:3000/catalogue/components/oci-ai-data-platform.md) `oci-ai-data-platform` — oci
- [OCI Big Data Service](https://localhost:3000/catalogue/components/oci-big-data-service.md) `oci-big-data-service` — oci

## Use it in a design

```yaml
nodes:
  - { id: "oci-data-flow", component: "oci-data-flow" }
```

## Source (`component` YAML, verbatim)

```yaml
# yaml-language-server: $schema=../../schema/component.schema.json
# OCI Data Flow: "a fully managed service for running Apache Spark applications" where "you don't need to
# spend any time on the underlying infrastructure, cluster provisioning, or software installation"
# ("Data Flow overview"). Each run gets its own Spark cluster driven entirely by REST APIs, and both the
# applications and the data they read and write live in Object Storage ("Data Flow service overview").
schemaVersion: "1"
type: "component"
id: "oci-data-flow"
kind: "big-data-cluster"
provider: "oci"
docs: "https://docs.oracle.com/en-us/iaas/data-flow/using/home.htm"
defaults:
  managed: true
  engine: "spark"
properties:
  language:
    type: "enum"
    values: ["java", "scala", "python", "sql", "spark-submit"]
  executors:
    type: "integer"
  poolBacked:
    type: "boolean"
    default: false
```

Human page: https://localhost:3000/catalogue/components/oci-data-flow