Azure Data Lake Storage

azure Object storeazure-data-lake-storage

Blob Storage with a hierarchical namespace and POSIX ACLs, reached over the ABFS driver for analytics.

Provider: azure
Kind: Object store
Category: Data
Official docs: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction
Also known as: ADLS

Properties · 6 set by this component

property	type	values	required	kind default	this component
accessControl · component property	enum	rbac \| posix-acl \| both	no	both	inherited
checksumOnWrite · advanced	boolean	—	no	false	inherited
durabilityNines · advanced	integer	—	no	—	▸ 11
encryptionAtRest	boolean	—	no	false	▸ true
endpoint · component property	enum	dfs \| blob	no	dfs	inherited
hierarchicalNamespace · component property	boolean	—	no	true	inherited
managed	boolean	—	no	false	▸ true
objectLock	boolean	—	no	false	inherited
redundancy · component property	enum	lrs \| zrs \| grs \| gzrs \| ra-grs \| ra-gzrs	no	lrs	inherited
replication	enum	none \| primary-secondary \| quorum \| erasure-coded	no	none	▸ quorum
strongReadAfterWrite · advanced	boolean	—	no	false	▸ true
tier	enum	hot \| warm \| cold	no	hot	inherited
versioning	boolean	—	no	false	inherited
zones	integer	—	no	—	▸ 1

Rules that apply · 1

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Object store in fewer than three zones with a durability targeterror

object-store-single-zone-with-durability-target

Eleven nines of durability assumes redundancy spread over at least three failure domains. An object store confined to fewer zones, or declaring none, cannot reach the target the design states.

— all of these hold:
— the design requirement `durability.annualNines` is a number greater than 10
— any of these hold:
— the node's `zones` has no value
— the node's `zones` is a number less than 3

Use it in a design

nodes:
  - { id: "data-lake-storage", component: "azure-data-lake-storage" }

Other Object store products · 32

Object Storage Servicealibaba-ossalibabacloudobject-store AWS HealthImagingaws-healthimagingawsobject-store AWS HealthOmicsaws-healthomicsawsobject-store Amazon S3 Express One Zoneaws-s3-express-one-zoneawsobject-store Amazon S3 Glacieraws-s3-glacierawsobject-store Amazon S3 Tablesaws-s3-tablesawsobject-store Amazon Security Lakeaws-security-lakeawsobject-store Amazon S3s3awsobject-store Azure Blob Storageazure-blob-storageazureobject-store Microsoft Planetary Computer Proazure-planetary-computerazureobject-store Cache Reservecloudflare-cache-reservecloudflareobject-store Cloudflare Imagescloudflare-imagescloudflareobject-store Cloudflare R2cloudflare-r2cloudflareobject-store Backblaze B2 Cloud Storagebackblaze-b2object-store Blob storeblob-storeobject-store Ceph RADOS Gatewayceph-rgwobject-store Cloudinarycloudinaryobject-store Garagegarageobject-store lakeFSlakefsobject-store MinIOminioobject-store Generic object storeobject-storeobject-store Apache Ozoneozoneobject-store SeaweedFSseaweedfsobject-store Storjstorjobject-store OpenStack Swiftswiftobject-store Wasabi Hot Cloud Storagewasabiobject-store Spaces Object Storagedigitalocean-spacesdigitaloceanobject-store Google Cloud Storagegcsgcpobject-store IBM Cloud Object Storageibm-cloud-object-storageibmobject-store OCI Archive Storageoci-archive-storageociobject-store OCI Object Storageoci-object-storageociobject-store OUTSCALE Object Storage (OOS)outscale-oosoutscaleobject-store

Source

1# yaml-language-server: $schema=../../schema/component.schema.json
2# Azure Data Lake Storage: Blob Storage with the hierarchical namespace enabled, reached over the
3# ABFS driver at the dfs endpoint ("Azure Data Lake Storage overview"). It inherits the storage
4# account defaults: LRS, three synchronous copies in one datacenter, encrypted at rest.
5schemaVersion: "1"
6type: "component"
7id: "azure-data-lake-storage"
8kind: "object-store"
9provider: "azure"
10docs: "https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction"
11aliases: ["ADLS"]
12defaults:
13  zones: 1
14  durabilityNines: 11
15  replication: "quorum"
16  strongReadAfterWrite: true
17  encryptionAtRest: true
18  managed: true
19properties:
20  hierarchicalNamespace:
21    type: "boolean"
22    default: true
23  redundancy:
24    type: "enum"
25    values: ["lrs", "zrs", "grs", "gzrs", "ra-grs", "ra-gzrs"]
26    default: "lrs"
27  endpoint:
28    type: "enum"
29    values: ["dfs", "blob"]
30    default: "dfs"
31  accessControl:
32    type: "enum"
33    values: ["rbac", "posix-acl", "both"]
34    default: "both"

The same page as markdown, for an agent: /catalogue/components/azure-data-lake-storage.md