Zeema
New design
Catalogue / big-data-cluster

Databricks

Big data clusterdatabricks

Managed lakehouse platform running Spark compute clusters and jobs.

Provider
default
Kind
Big data cluster
Category
Compute
Official docs
https://docs.databricks.com/aws/en/compute/configure
Properties · 4 set by this component
propertytypevaluesrequiredkind defaultthis component
accessMode · component propertyenumauto | standard | dedicatednoautoinherited
autoscalingboolean—nofalse▸ false
computeType · component propertyenumall-purpose | job | serverlessno—inherited
engineenumspark | hadoop | presto | flink | hpc | ray | dask | kubernetes-batchno—▸ spark
managedboolean—nofalse▸ true
mosaicAi · component propertyboolean—nofalseinherited
photon · component propertyboolean—notrueinherited
runtimeVersionstring—no—inherited
singleNode · component propertyboolean—nofalseinherited
spotInstancesboolean—nofalse▸ false
workerNodesinteger—no—inherited
zoneRedundantboolean—no—inherited
Rules that apply · 1

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Stateful node without zone redundancyerror
stateful-node-without-zone-redundancy

A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.

  • — all of these hold:
  • — the node's `stateless` is false
  • — the node sits inside exactly 1 availability-zone group
  • — this does not hold:
  • — the node sits inside at least one cell group
Use it in a design
nodes:
  - { id: "databricks", component: "databricks" }
Other Big data cluster products · 25
Elastic High Performance Computingalibaba-e-hpcalibabacloudbig-data-clusterE-MapReducealibaba-e-mapreducealibabacloudbig-data-clusterAWS Clean Roomsaws-clean-roomsawsbig-data-clusterAmazon EMRaws-emrawsbig-data-clusterAmazon EMR Serverlessaws-emr-serverlessawsbig-data-clusterAWS Parallel Computing Serviceaws-pcsawsbig-data-clusterAmazon SageMaker HyperPodaws-sagemaker-hyperpodawsbig-data-clusterAzure CycleCloudazure-cyclecloudazurebig-data-clusterAzure Databricksazure-databricksazurebig-data-clusterAzure HDInsightazure-hdinsightazurebig-data-clusterArmadaarmadabig-data-clusterGeneric big data clusterbig-data-clusterbig-data-clusterDaskdaskbig-data-clusterApache Hadoop YARNhadoop-yarnbig-data-clusterRayraybig-data-clusterSlurmslurmbig-data-clusterApache Spark clusterspark-clusterbig-data-clusterVolcanovolcanobig-data-clusterCluster Directorgoogle-cluster-directorgcpbig-data-clusterDataprocgoogle-dataprocgcpbig-data-clusterManaged Service for Apache Spark (serverless)google-dataproc-serverlessgcpbig-data-clusterIBM Analytics Engineibm-analytics-engineibmbig-data-clusterOCI AI Data Platformoci-ai-data-platformocibig-data-clusterOCI Big Data Serviceoci-big-data-serviceocibig-data-clusterOCI Data Flowoci-data-flowocibig-data-cluster
Source
# yaml-language-server: $schema=../../schema/component.schema.json
# Databricks: managed lakehouse platform whose classic compute is a Spark cluster of driver and workers.
# Defaults from "Compute" and "Configure compute": access mode Auto, multi-node, autoscaling and
# spot instances off until enabled, Photon on from Databricks Runtime 9.1 LTS. mosaicAi says the workspace
# uses Mosaic AI (Model Serving, Agent Framework, Vector Search, AutoML), Databricks' AI layer; off until stated.
schemaVersion: "1"
type: "component"
id: "databricks"
kind: "big-data-cluster"
provider: "default"
docs: "https://docs.databricks.com/aws/en/compute/configure"
defaults:
  engine: "spark"
  autoscaling: false
  spotInstances: false
  managed: true
properties:
  computeType:
    type: "enum"
    values: ["all-purpose", "job", "serverless"]
  accessMode:
    type: "enum"
    values: ["auto", "standard", "dedicated"]
    default: "auto"
  photon:
    type: "boolean"
    default: true
  singleNode:
    type: "boolean"
    default: false
  mosaicAi:
    type: "boolean"
    default: false

The same page as markdown, for an agent: /catalogue/components/databricks.md