AWS DataSync

Managed transfer task copying file and object data between on-premises storage, AWS services and other clouds.

Provider: aws
Kind: Worker
Category: Compute
Official docs: https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html

Properties · 3 set by this component

property	type	values	required	kind default	this component
accelerator	enum	none \| gpu \| asic	no	none	inherited
agentRequired · component property	boolean	—	no	false	inherited
bandwidthThrottling · component property	boolean	—	no	false	inherited
concurrency	integer	—	no	—	inherited
encryptionInTransit · component property	boolean	—	no	true	inherited
fanoutStrategy · advanced	enum	none \| push \| pull \| hybrid	no	none	inherited
fanoutThreshold · advanced	integer	—	no	—	inherited
idempotent	boolean	—	no	false	inherited
instances	integer	—	no	—	inherited
managed	boolean	—	no	false	▸ true
rateControl · advanced	enum	none \| fixed \| aimd	no	none	▸ none
schedule	string	—	no	—	inherited
taskMode · component property	enum	basic \| enhanced	no	—	inherited
trigger	enum	queue \| stream \| schedule \| batch \| manual	yes	—	▸ manual
verifyMode · component property	enum	point-in-time-consistent \| only-files-transferred \| none	no	point-in-time-consistent	inherited

Rules that apply · 3

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Push fan-out without a celebrity thresholdwarning

push-fanout-without-celebrity-path

Pure push fan-out over a social graph is brutal at the tail: one post from an account with millions of followers becomes millions of cache writes. A follower threshold above which readers merge at read time bounds that write amplification.

— all of these hold:
— the node's `fanoutStrategy` is "push"
— the node's `fanoutThreshold` has no value
— the design contains a node of kind graph-database

Shared worker pool across queueswarning

shared-worker-pool-across-queues

One worker pool consuming several queues lets a slow or poisoned queue starve the others. That is precisely what the design's per-queue isolation requirement forbids.

— all of these hold:
— the node has more than 1 incoming edge from a queue
— the design requirement `isolation.perQueue` is true

Stateful node without zone redundancyerror

stateful-node-without-zone-redundancy

A stateful compute node pinned to exactly one availability zone disappears with that zone, and its state goes with it. Nodes inside a cell are exempt, because there the cell is deliberately single-zone and redundancy comes from the other cells.

— all of these hold:
— the node's `stateless` is false
— the node sits inside exactly 1 availability-zone group
— this does not hold:
— the node sits inside at least one cell group

Use it in a design

nodes:
  - { id: "datasync", component: "aws-datasync" }

Other Worker products · 82

SchedulerXalibaba-schedulerxalibabacloudworker Amazon AppFlowaws-appflowawsworker AWS B2B Data Interchangeaws-b2b-data-interchangeawsworker AWS Batchaws-batchawsworker AWS Glue DataBrewaws-databrewawsworker AWS Deadline Cloudaws-deadline-cloudawsworker AWS Entity Resolutionaws-entity-resolutionawsworker Amazon EventBridge Scheduleraws-eventbridge-schedulerawsworker AWS Fault Injection Serviceaws-fisawsworker AWS Glue crawleraws-glue-crawlerawsworker AWS Glue for Ray jobaws-glue-ray-jobawsworker AWS Application Migration Serviceaws-mgnawsworker Amazon MSK Connectaws-msk-connectawsworker Amazon SageMaker AI Training Jobaws-sagemaker-training-jobawsworker AWS Systems Manageraws-systems-managerawsworker AWS Thinkbox Deadline 10aws-thinkbox-deadlineawsworker AWS Glue jobglue-jobawsworker Azure Automationazure-automationazureworker Azure Batchazure-batchazureworker Azure Chaos Studioazure-chaos-studioazureworker Azure Data Factoryazure-data-factoryazureworker Azure Data Shareazure-data-shareazureworker Azure Database Migration Serviceazure-database-migration-serviceazureworker Microsoft Entra Connect Syncazure-entra-connect-syncazureworker Microsoft Graph Data Connectazure-graph-data-connectazureworker Azure Storage Moverazure-storage-moverazureworker Azure Update Managerazure-update-managerazureworker Browser Runcloudflare-browser-runcloudflareworker Cloudflare Snippetscloudflare-snippetscloudflareworker Cloudflare Zarazcloudflare-zarazcloudflareworker Airbyteairbyteworker Airflow DAGairflow-dagworker Chaos Meshchaos-meshworker Chunkrchunkrworker Container workercontainer-workerworker Cron jobcron-jobworker Daytonadaytonaworker E2Be2bworker Embulkembulkworker Fan-out daemonfanout-daemonworker

and 42 more → Worker page

Source

1# yaml-language-server: $schema=../../schema/component.schema.json
2# AWS DataSync task: managed transfer job that copies file and object data between on-premises storage,
3# AWS storage services and other clouds. "What is AWS DataSync?": transfers include automatic encryption
4# and data integrity validation; "Configuring how AWS DataSync verifies data integrity": checksum
5# verification runs during every transfer and VerifyMode selects the end-of-transfer check.
6schemaVersion: "1"
7type: "component"
8id: "aws-datasync"
9kind: "worker"
10provider: "aws"
11docs: "https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html"
12defaults:
13  trigger: "manual"
14  rateControl: "none"
15  managed: true
16properties:
17  verifyMode:
18    type: "enum"
19    values: ["point-in-time-consistent", "only-files-transferred", "none"]
20    default: "point-in-time-consistent"
21  encryptionInTransit:
22    type: "boolean"
23    default: true
24  agentRequired:
25    type: "boolean"
26    default: false
27  taskMode:
28    type: "enum"
29    values: ["basic", "enhanced"]
30  bandwidthThrottling:
31    type: "boolean"
32    default: false

The same page as markdown, for an agent: /catalogue/components/aws-datasync.md