Implementing an Azure Data Solutions

Introduction to Azure

 Introduction to Azure Cloud

 What is difference between Azure Cloud and On-Premises
 What is Subscriptions and Resource Groups
 Different offerings of Cloud IaaS, PaaS and SaaS
 Creation of Virtual Machine

Introduction to Storage

 Azure Storage
o Azure Blob
o Table
o Message
o Queue
 Azure Data Lake Store Gen I & Gen II
o What is Data Lake
o Data Lake vs. Hadoop
o Blob Storage vs. Data Lake
o Hierarchical Namespace
o Ingestion through different tools i.e.; Azure Data Explorer, AzCopy, Azure CLI,
Powershell

Introduction to Azure SQL Database

 Introduction to Azure SQL Database

 Why choosing SQL Server in Azure
 Azure IaaS vs. PaaS database offerings
 IaaS vs. Managed Instance
 SQL Server PaaS deployment options
 Demo - Azure Single Database
 Purchasing models and Service Tier
 Azure Database vs. Azure Data Warehouse
 Elastic Database Pool
o Introduction
o Azure Elastic Database
o Demo - Azure Elastic Database
 Managed Instance Database
o Introduction
o Azure Managed Instance Database
o Difference between on-premises and managed instance
o Migration options for Managed Instance
 o Service tiers for Managed Instance
o Demo - Managed Instance

 Azure Database Security

o Introduction
o Azure Database and Managed Instance Security options
o Encrypting Data at Rest and Motion
o High Availability vs. Disaster Recovery
o RTO vs. RPO
o Azure SQL Database High Availability and Disaster Recovery options
o Azure SQL Database Scaling
 Installation of SQL Server 2016 and above in Virtual Machine
 Creation of External Table or PolyBase in On-Premise SQL Server
o Creation of Master Key
o Creation of Database Scoped Credential
o Creation of External Data Source
o Creation of External File Format
o Creation of External Table
 Creation of External Table or PolyBase in Azure SQL Data Warehouse
o Creation of Master Key
o Creation of Database Scoped Credential
o Creation of External Data Source
o Creation of External File Format
o Creation of External Table
 Different Distribution or Shredding Patterns
o ROUND ROBIN
o HASH
o REPLICATION
 Cross Query Databases in Azure SQL Database
o Creation of Master Key
o Creation of Database Scoped Credential
o Creation of External Data Source
o Creation of External Table
 Creation of Elastic Pools in Azure SQL Server between Databases

Data Warehouse Internals and Architecture

 Introduction
 Azure Synapse MPP Architecture
 Storage and Sharding patterns
 Data Distribution and Distributing Keys
 Data Types and Table Types
 Partitioning
 Data Warehouse Concepts
 Dimensions and Facts
 Types of Dimensions and Facts
 Different types of Schemas in Data Warehouse
  Relationship types in Data Warehouse
 Best Practices for Fact and Dimension tables
 Demo - Analyze Data distribution before migration to Azure Synapse

Azure Data Factory

 Introduction to Azure Data Factory

 Creation of Linked Services, Datasets, Pipelines
 Creation of Integration Runtime and different types
 Slowly Changing Dimensions
 Design and implement a Type 1 slowly changing dimension with mapping data flows
 Debug data factory pipelines
 Understand the Azure SSIS Integration Runtime
 Set-up Azure SSIS Integration Runtime
 Run SSIS Package in Azure Data Factory
 Migrate SSIS Packages to Azure Data Factory
 Integrate SQL Server Integration Services Packages within Azure Data Factory
 Activities
o Copy
o Data flow
o Stored Procedure
o Lookup
o ForEach
o Get Metadata
o Filter Activity
o Spark
o U-SQL
o Databricks Notebooks
o Web
o If Condition
o Delete
 Data Flows
o Derived Column
o Join
o filter
o exists
o conditional split
o Lookup, Exists
o Select
o Aggregate
o Rank
o Filter
o Sort
o Alter Row
 Dynamic Queries in ADF
  Sending mails through Logic Apps
 Few more Activities ......
 Dataset and Pipeline Parameterization
 Monitor -- Azure and Visually
 Setup Alerts from Azure Data Factory

Realize Integrated Analytical Solutions with Azure Synapse Analytics

 Introduction
 What is Azure Synapse Analytics
 How Azure Synapse Analytics works
 When to use Azure Synapse Analytics
 Create Azure Synapse Analytics workspace
 Exercise - Create and manage Azure Synapse Analytics workspace
 Describe Azure Synapse Analytics SQL
 Explain Apache Spark in Azure Synapse Analytics
 Exercise - Create pools in Azure Synapse Analytics
 Orchestrate data integration with Azure Synapse pipelines
 Exercise-Identifying Azure Synapse pipeline components
 Visualize your analytics with Power BI
 Understand hybrid transactional analytical processing with Azure Synapse Link
 Use Azure Synapse Studio
 Understand the Azure Synapse Analytical processes
 Explore the Data hub, Develop hub, Integrate hub
 Explore the Monitor hub, Manage hub
 Describe a modern data warehouse
 Define a modern data warehouse architecture
 Exercise - Identify modern data warehouse architecture components
 Design ingestion patterns for a modern data warehouse
 Understand data storage for a modern data warehouse
 Understand file formats and structure for a modern data warehouse
 Prepare and transform data with Azure Synapse Analytics
 Serve data for analysis with Azure Synapse Analytics

Azure Synapse Analytics

Introduction
 Why Warehouse in cloud
 Traditional vs. Modern Warehouse architecture
 What is Synapse Analytics Service
 Create Dedicated SQL Pool and Spark Pool
 Create Azure Synapse Analytics Studio Workspace
 Analyze Data using Dedicated SQL Pool and Spark Pool
 Analyze Data using Apache Spark Notebook
 Analyze Data using Serverless SQL Pool
  Azure Synapse Benefits

Azure Event Hub, IoT Hub and Azure Stream Analytics

 Introduction to Azure Event Hub, IoT Hub and Stream Analytics

 Azure Stream Analytics Job
 Azure Stream Analytics Components
 Azure Stream Analytics Job
 Batching Streaming using Azure Event Hub
 Real Time Streaming using Azure IoT Hub
 Types of Window Functions
o Tumbling Window
o Hoping Window
o Sliding Window
o Session Window

Azure Databricks

 Spark Basics
 Why Spark is difficult? Why Databricks Evolved?
 Why Databricks in Cloud? Introduction to Azure Databricks
 Demo
 Provision Databricks, Clusters and workbook
 Mount Data Lake to Databricks DBFS
 Explore, Analyze, Clean, Transform and Load Data in Databricks
 Azure Databricks Clusters
 Azure Databricks other Important Components
 Databricks - Monitoring
 How to create Cluster
 How to work with Databricks File System
 How to create notebooks and Integrate with ADF
 How to import and export the Notebooks
 How to connect to blob, SQL DB from Databricks
 How to read data files from Azure Blob and Azure Data Lake Store
 Using Scala, R, Python, Spark SQL Language
 Creating Data Frames
 Converting Data Frames into Temporary Table or Temporary View
 Incremental and Full Load with Azure SQL Data Warehouse
 Understand the architecture of Azure Databricks spark cluster
 Understand the architecture of spark job
 Read data in CSV format
 Read data in JSON format
 Read data in Parquet format
 Read data stored in tables and views
  Write data
 Describe a DataFrame
 Use common DataFrame methods
 Use the display function
 Exercise: Distinct articles
 Describe the difference between eager and lazy execution
 Describe the fundamentals of how the Catalyst Optimizer works
 Define and identify actions and transformations
 Describe the column class
 Work with column expressions
 Perform date and time manipulation
 Use aggregate functions
 Exercise: Deduplication of data
 Describe the Azure Databricks platform architecture
 Perform data protection
 Describe Azure key vault and Databricks security scopes
 Secure access with Azure IAM and authentication
 Describe security
 Exercise: Access Azure Storage with key vault-backed secrets
 Describe the open source Delta Lake
 Exercise: Work with basic Delta Lake functionality
 Describe how Azure Databricks manages Delta Lake
 Exercise: Use the Delta Lake Time Machine and perform optimization
 Describe Azure Databricks structured streaming
 Perform stream processing using structured streaming
 Work with Time Windows
 Process data from Event Hubs with structured streaming
 Describe bronze, silver, and gold architecture
 Perform batch and stream processing
 Schedule Databricks jobs in a data factory pipeline
 Pass parameters into and out of Databricks jobs in data factory
 Integrate with Azure Synapse Analytics
 Understand workspace administration best practices
 List security best practices
 Describe tools and integration best practices
 Explain Databricks runtime best practices
 Understand cluster best practices

Azure Cosmos DB

Introduction to NoSQL DB
 Introduction to NoSQL
 SQL vs. NoSQL
 Types of NoSQL
 NoSQL Offerings by Microsoft
 Introduction to Cosmos DB
 Cosmos DB Features
 Cosmos DB - Multi Model 5 APIs
 Table Storage vs. Cosmos DB
 Provision Cosmos DB Account

On-Premise Databases Migration

 DMS -- Database Migration Service
 On-Premise SQL Server to Azure Virtual Machine
 On-Premise SQL Server to Azure SQL Server