Data Engineering On Microsoft Azure DP203 | Microsoft Azure in Exeter

Data Engineering on Microsoft Azure DP203 Course Outline

Module 1: Introduction to Data Engineering on Azure

• Introduction
• What Is Data Engineering?
• Important Data Engineering Concepts
• Data Engineering in Microsoft Azure

Module 2: Introduction to Azure Data Lake Storage Gen2

• Introduction
• Understand Azure Data Lake Storage Gen2
• Enable Azure Data Lake Storage Gen2 in Azure Storage
• Compare Azure Data Lake Store to Azure Blob Storage
• Understand the Stages for Processing Big Data
• Use Azure Data Lake Storage Gen2 in Data Analytics Workloads

Module 3: Introduction to Azure Synapse Analytics

• Introduction
• What is Azure Synapse Analytics?
• How Azure Synapse Analytics Works?
• When to Use Azure Synapse Analytics?
• Exercise: Explore Azure Synapse Analytics

Module 4: Use Azure Synapse Serverless SQL Pool to Query Files in a Data Lake

• Introduction
• Understand Azure Synapse Serverless SQL Pool Capabilities and Use Cases
• Query Files Using a Serverless SQL Pool
• Create External Database Objects
• Exercise: Query Files Using a Serverless SQL Pool

Module 5: Use Azure Synapse Serverless SQL Pools to Transform Data in a Data Lake

• Introduction
• Transform Data Files with the CREATE EXTERNAL TABLE AS SELECT Statement
• Encapsulate Data Transformations in a Stored Procedure
• Include a Data Transformation Stored Procedure in a Pipeline
• Exercise: Transform Files Using a Serverless SQL Pool

Module 6: Create a Lake Database in Azure Synapse Analytics

• Introduction
• Understand Lake Database Concepts
• Explore Database Templates
• Create a Lake Database
• Use a Lake Database
• Exercise: Analyse Data in a Lake Database

Module 7: Analyse Data with Apache Spark in Azure Synapse Analytics

• Introduction
• Get to Know Apache Spark
• Use Spark in Azure Synapse Analytics
• Analyse Data with Spark
• Visualise Data with Spark
• Exercise: Analyse Data with Spark

Module 8: Transform Data with Spark in Azure Synapse Analytics

• Introduction
• Modify and Save Dataframes
• Partition Data Files
• Transform Data with SQL
• Exercise: Transform Data with Spark in Azure Synapse Analytics

Module 9: Use Delta Lake in Azure Synapse Analytics

• Introduction
• Understand Delta Lake
• Create Delta Lake Tables
• Create Catalog Tables
• Use Delta Lake with Streaming Data
• Use Delta Lake in a SQL Pool
• Exercise: Use Delta Lake in Azure Synapse Analytics

Module 10: Analyse Data in a Relational Data Warehouse

• Introduction
• Design a Data Warehouse Schema
• Create Data Warehouse Tables
• Load Data Warehouse Tables
• Query a Data Warehouse
• Exercise: Explore a Data Warehouse

Module 11: Load Data into a Relational Data Warehouse

• Introduction
• Load Staging Tables
• Load Dimension Tables
• Load Time Dimension Tables
• Load Slowly Changing Dimensions
• Load Fact Tables
• Perform Post-Load Optimisation
• Exercise: Load Data into a Relational Data Warehouse

Module 12: Build a Data Pipeline in Azure Synapse Analytics

• Introduction
• Understand Pipelines in Azure Synapse Analytics
• Create a Pipeline in Azure Synapse Studio
• Define Data Flows
• Run a Pipeline
• Exercise: Build a Data Pipeline in Azure Synapse Analytics

Module 13: Use Spark Notebooks in an Azure Synapse Pipeline

• Introduction
• Understand Synapse Notebooks and Pipelines
• Use a Synapse Notebook Activity in a Pipeline
• Use Parameters in a Notebook
• Exercise: Use an Apache Spark Notebook in a Pipeline

Module 14: Plan Hybrid Transactional and Analytical Processing Using Azure Synapse Analytics

• Introduction
• Understand Hybrid Transactional and Analytical Processing Patterns
• Describe Azure Synapse Link

Module 15: Implement Azure Synapse Link with Azure Cosmos DB

• Introduction
• Enable Cosmos DB Account to Use Azure Synapse Link
• Create an Analytical Store Enabled Container
• Create a Linked Service for Cosmos DB
• Query Cosmos DB Data with Spark
• Query Cosmos DB with Synapse SQL
• Exercise: Implement Azure Synapse Link for Cosmos DB

Module 16: Implement Azure Synapse Link for SQL

• Introduction
• What Is Azure Synapse Link for SQL?
• Configure Azure Synapse Link for Azure SQL Database
• Configure Azure Synapse Link for SQL Server 2022
• Exercise: Implement Azure Synapse Link for SQL

Module 17: Get Started with Azure Stream Analytics

• Introduction
• Understand Data Streams
• Understand Event Processing
• Understand Window Functions
• Exercise: Get Started with Azure Stream Analytics

Module 18: Ingest Streaming Data Using Azure Stream Analytics and Azure Synapse Analytics

• Introduction
• Stream Ingestion Scenarios
• Configure Inputs and Outputs
• Define a Query to Select, Filter, and Aggregate Data
• Run a Job to Ingest Data
• Exercise: Ingest Streaming Data into Azure Synapse Analytics

Module 19: Visualize Real-Time Data with Azure Stream Analytics and Power BI

• Introduction
• Use a Power BI Output in Azure Stream Analytics
• Create a Query for Real-Time Visualization
• Create Real-Time Data Visualizations in Power BI
• Exercise: Create a Real-Time Data Visualization

Module 20: Introduction to Microsoft Purview

• Introduction
• What Is Microsoft Purview?
• How Microsoft Purview Works?
• When to Use Microsoft Purview?

Module 21: Integrate Microsoft Purview and Azure Synapse Analytics

• Introduction
• Catalog Azure Synapse Analytics Data Assets in Microsoft Purview
• Connect Microsoft Purview to an Azure Synapse Analytics Workspace
• Search a Purview Catalog in Synapse Studio
• Track Data Lineage in Pipelines
• Exercise: Integrate Azure Synapse Analytics and Microsoft Purview

Module 22: Explore Azure Databricks

• Introduction
• Get Started with Azure Databricks
• Identify Azure Databricks Workloads
• Understand Key Concepts
• Exercise: Explore Azure Databricks

Module 23: Use Apache Spark in Azure Databricks

• Introduction
• Get to Know Spark
• Create a Spark Cluster
• Use Spark in Notebooks
• Use Spark to Work with Data Files
• Visualise Data
• Exercise: Use Spark in Azure Databricks

Module 24: Run Azure Databricks Notebooks with Azure Data Factory

• Introduction
• Understand Azure Databricks Notebooks and Pipelines
• Create a Linked Service for Azure Databricks
• Use a Notebook Activity in a Pipeline
• Use Parameters in a Notebook
• Exercise: Run an Azure Databricks Notebook with Azure Data Factory