This architecture is for one data pipeline approach called ETL— or extract, transform, load — which allows you to maximize flexibility in how you transform your data while also providing dynamic scaling to handle large amounts of incoming data and processing demands.
It uses Cloud Storage for landing data, Dataflow as a managed service for processing data, BigQuery as a destination data warehouse, and Cloud Composer, as an orchestration tool that helps you author, schedule, and monitor data pipelines.
You can install this application using the Open in Google Cloud Shell button
below.
Clicking this link will take you right to the DeployStack app, running in your Cloud Shell environment. It will walk you through setting up your architecture.
To remove all billing components from the project
- Typing
deploystack uninstall
This is not an official Google product.