Big data cluster

computebig-data-cluster

A multi-node analytics cluster running Spark, Hadoop, Presto or HPC jobs.

Category: Compute
Properties: 7
Components: 26
Rules: 1

Properties · 7 defined by this kind

A node of this kind resolves each property from its own props, then the component's defaults, then the default below.

property	type	values	required	default
autoscaling	boolean	—	no	false
engine	enum	spark \| hadoop \| presto \| flink \| hpc \| ray \| dask \| kubernetes-batch	no	—
managed	boolean	—	no	false
runtimeVersion	string	—	no	—
spotInstances	boolean	—	no	false
workerNodes	integer	—	no	—
zoneRedundant	boolean	—	no	—

Rules for this kind · 1

Stateful node without zone redundancyerror

stateful-node-without-zone-redundancy

A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.

— all of these hold:
— the node's `stateless` is false
— the node sits inside exactly 1 availability-zone group
— this does not hold:
— the node sits inside at least one cell group

Components · 26

vendor-neutral

Generic big data clusterbig-data-clustergenericthe kind itself

alibabacloud

Elastic High Performance Computingalibaba-e-hpc2 defaults E-MapReducealibaba-e-mapreduce1 defaults

aws

AWS Clean Roomsaws-clean-rooms3 defaults Amazon EMRaws-emr4 defaults Amazon EMR Serverlessaws-emr-serverless4 defaults AWS Parallel Computing Serviceaws-pcs2 defaults Amazon SageMaker HyperPodaws-sagemaker-hyperpod2 defaults

azure

Azure CycleCloudazure-cyclecloud3 defaults Azure Databricksazure-databricks4 defaults Azure HDInsightazure-hdinsight2 defaults

default

Armadaarmada4 defaults Daskdask4 defaults Databricksdatabricks4 defaults Apache Hadoop YARNhadoop-yarn4 defaults Rayray4 defaults Slurmslurm4 defaults Apache Spark clusterspark-cluster4 defaults Volcanovolcano4 defaults

gcp

Cluster Directorgoogle-cluster-director4 defaults Dataprocgoogle-dataproc6 defaults Managed Service for Apache Spark (serverless)google-dataproc-serverless4 defaults

ibm

IBM Analytics Engineibm-analytics-engine3 defaults

oci

OCI AI Data Platformoci-ai-data-platform2 defaults OCI Big Data Serviceoci-big-data-service2 defaults OCI Data Flowoci-data-flow2 defaults

Source

1# yaml-language-server: $schema=../schema/kind.schema.json
2# A sized cluster that accepts job submissions, where worker is a single job runner and data-warehouse is query-only storage. workerNodes and zoneRedundant have no default so sizing and resilience rules can catch an undeclared cluster.
3schemaVersion: "1"
4type: "kind"
5id: "big-data-cluster"
6category: "compute"
7properties:
8  engine:
9    type: "enum"
10    values: ["spark", "hadoop", "presto", "flink", "hpc", "ray", "dask", "kubernetes-batch"]
11  workerNodes:
12    type: "integer"
13  runtimeVersion:
14    type: "string"
15  autoscaling:
16    type: "boolean"
17    default: false
18  spotInstances:
19    type: "boolean"
20    default: false
21  zoneRedundant:
22    type: "boolean"
23  managed:
24    type: "boolean"
25    default: false

The same page as markdown, for an agent: /catalogue/kinds/big-data-cluster.md