Managed Service for Apache Spark (serverless)

gcp Big data clustergoogle-dataproc-serverless

Serverless Spark batches and sessions with no cluster to provision.

Provider: gcp
Kind: Big data cluster
Category: Compute
Official docs: https://docs.cloud.google.com/dataproc-serverless/docs/concepts/properties

Properties · 4 set by this component

property	type	values	required	kind default	this component
autoscaling	boolean	—	no	false	▸ true
driverCores · component property	integer	—	no	4	inherited
engine	enum	spark \| hadoop \| presto \| flink \| hpc \| ray \| dask \| kubernetes-batch	no	—	▸ spark
executorCores · component property	integer	—	no	4	inherited
managed	boolean	—	no	false	▸ true
maxExecutors · component property	integer	—	no	2000	inherited
minExecutors · component property	integer	—	no	2	inherited
runtimeVersion	string	—	no	—	inherited
spotInstances	boolean	—	no	false	▸ false
tier · component property	enum	standard \| premium	no	standard	inherited
workerNodes	integer	—	no	—	inherited
zoneRedundant	boolean	—	no	—	inherited

Rules that apply · 1

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Stateful node without zone redundancyerror

stateful-node-without-zone-redundancy

A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.

— all of these hold:
— the node's `stateless` is false
— the node sits inside exactly 1 availability-zone group
— this does not hold:
— the node sits inside at least one cell group

Use it in a design

nodes:
  - { id: "dataproc-serverless", component: "google-dataproc-serverless" }

Other Big data cluster products · 25

Elastic High Performance Computingalibaba-e-hpcalibabacloudbig-data-cluster E-MapReducealibaba-e-mapreducealibabacloudbig-data-cluster AWS Clean Roomsaws-clean-roomsawsbig-data-cluster Amazon EMRaws-emrawsbig-data-cluster Amazon EMR Serverlessaws-emr-serverlessawsbig-data-cluster AWS Parallel Computing Serviceaws-pcsawsbig-data-cluster Amazon SageMaker HyperPodaws-sagemaker-hyperpodawsbig-data-cluster Azure CycleCloudazure-cyclecloudazurebig-data-cluster Azure Databricksazure-databricksazurebig-data-cluster Azure HDInsightazure-hdinsightazurebig-data-cluster Armadaarmadabig-data-cluster Generic big data clusterbig-data-clusterbig-data-cluster Daskdaskbig-data-cluster Databricksdatabricksbig-data-cluster Apache Hadoop YARNhadoop-yarnbig-data-cluster Rayraybig-data-cluster Slurmslurmbig-data-cluster Apache Spark clusterspark-clusterbig-data-cluster Volcanovolcanobig-data-cluster Cluster Directorgoogle-cluster-directorgcpbig-data-cluster Dataprocgoogle-dataprocgcpbig-data-cluster IBM Analytics Engineibm-analytics-engineibmbig-data-cluster OCI AI Data Platformoci-ai-data-platformocibig-data-cluster OCI Big Data Serviceoci-big-data-serviceocibig-data-cluster OCI Data Flowoci-data-flowocibig-data-cluster

Source

1# yaml-language-server: $schema=../../schema/component.schema.json
2# Managed Service for Apache Spark serverless (Dataproc Serverless) batch: Spark workloads with no
3# cluster to provision. "Serverless Spark overview" and "Spark properties": the service autoscales
4# the workload, the standard tier is the default, and driver and executor default to four cores each
5# with 2 to 2,000 executors.
6schemaVersion: "1"
7type: "component"
8id: "google-dataproc-serverless"
9kind: "big-data-cluster"
10provider: "gcp"
11docs: "https://docs.cloud.google.com/dataproc-serverless/docs/concepts/properties"
12defaults:
13  managed: true
14  engine: "spark"
15  autoscaling: true
16  spotInstances: false
17properties:
18  tier:
19    type: "enum"
20    values: ["standard", "premium"]
21    default: "standard"
22  driverCores:
23    type: "integer"
24    default: 4
25  executorCores:
26    type: "integer"
27    default: 4
28  minExecutors:
29    type: "integer"
30    default: 2
31  maxExecutors:
32    type: "integer"
33    default: 2000

The same page as markdown, for an agent: /catalogue/components/google-dataproc-serverless.md