Manufacturing Data Engine

gcp Stream processorgoogle-manufacturing-data-engine

Configuration-driven pipeline normalising factory telemetry into Bigtable and BigQuery.

Provider: gcp
Kind: Stream processor
Category: Messaging
Official docs: https://docs.cloud.google.com/manufacturing-data-engine/docs/overview

Properties · 2 set by this component

property	type	values	required	kind default	this component
archiveToCloudStorage · component property	boolean	—	no	true	inherited
checkpointIntervalSeconds	integer	—	no	—	inherited
configurationDrivenMapping · component property	boolean	—	no	true	inherited
exactlyOnce	boolean	—	no	false	inherited
instances	integer	—	no	—	inherited
lateDataPolicy	enum	drop \| shift \| sideline	no	—	inherited
managed	boolean	—	no	false	▸ true
runtime · component property	enum	dataflow	no	dataflow	inherited
stateBackend	enum	memory \| rocksdb \| external	no	memory	▸ external
timeSeriesStore · component property	enum	bigtable \| bigquery \| both	no	both	inherited
watermarkSeconds	integer	—	no	—	inherited
windowing	enum	none \| event-time \| processing-time	no	none	inherited

Rules that apply · 1

Every rule whose selector covers this component's kind or its category. Whether a rule fires depends on the design: its properties, its edges and its groups.

Event-time windows without watermarkwarning

windowed-aggregation-without-watermark

Event-time windows need a watermark to know when a window is complete. Without one the processor either waits forever or closes windows early and silently drops late events.

— all of these hold:
— the node's `windowing` is "event-time"
— the node's `watermarkSeconds` has no value

Use it in a design

nodes:
  - { id: "manufacturing-data-engine", component: "google-manufacturing-data-engine" }

Other Stream processor products · 33

Realtime Compute for Apache Flinkalibaba-flinkalibabacloudstream-processor Amazon EventBridge Pipesaws-eventbridge-pipesawsstream-processor Amazon Data Firehoseaws-firehoseawsstream-processor Amazon Managed Service for Apache Flinkaws-managed-flinkawsstream-processor Amazon OpenSearch Ingestionaws-opensearch-ingestionawsstream-processor Azure Stream Analyticsazure-stream-analyticsazurestream-processor Cloudflare Logpushcloudflare-logpushcloudflarestream-processor Cloudflare Pipelinescloudflare-pipelinescloudflarestream-processor Arroyoarroyostream-processor Apache Beambeamstream-processor Bytewaxbytewaxstream-processor Drasidrasistream-processor Apache Flinkflinkstream-processor Kafka Streamskafka-streamsstream-processor ksqlDBksqldbstream-processor Logstashlogstashstream-processor Materializematerializestream-processor Apache NiFinifistream-processor Norikranorikrastream-processor Redpanda Connectredpanda-connectstream-processor RisingWaverisingwavestream-processor Apache Samzasamzastream-processor Snowplowsnowplowstream-processor Spark Structured Streamingspark-streamingstream-processor Apache Stormstormstream-processor Generic stream processorstream-processorstream-processor Tremortremorstream-processor Google Cloud Dataflowgoogle-dataflowgcpstream-processor Activity Tracker Event Routingibm-activity-tracker-event-routingibmstream-processor IBM Cloud Logs Routingibm-logs-routingibmstream-processor IBM Cloud Metrics Routingibm-metrics-routingibmstream-processor OCI Connector Huboci-connector-hubocistream-processor OCI GoldenGate Stream Analyticsoci-goldengate-stream-analyticsocistream-processor

Source

1# yaml-language-server: $schema=../../schema/component.schema.json
2# Manufacturing Data Engine: the cloud-side pipeline that normalises factory telemetry.
3# "Manufacturing Data Engine overview": "Pub/Sub is the message broker of MDE that is used
4# to route messages between the different components", a Dataflow message mapper applies
5# configuration-driven transforms, and records land in Bigtable, BigQuery and Cloud
6# Storage; real-time analytics are computed from user configuration.
7schemaVersion: "1"
8type: "component"
9id: "google-manufacturing-data-engine"
10kind: "stream-processor"
11provider: "gcp"
12docs: "https://docs.cloud.google.com/manufacturing-data-engine/docs/overview"
13defaults:
14  stateBackend: "external"
15  managed: true
16properties:
17  runtime:
18    type: "enum"
19    values: ["dataflow"]
20    default: "dataflow"
21  timeSeriesStore:
22    type: "enum"
23    values: ["bigtable", "bigquery", "both"]
24    default: "both"
25  archiveToCloudStorage:
26    type: "boolean"
27    default: true
28  configurationDrivenMapping:
29    type: "boolean"
30    default: true

The same page as markdown, for an agent: /catalogue/components/google-manufacturing-data-engine.md