Scribd
Upload
34 views2 pages

ADF Class Notes

The document outlines a real-time use case for an Azure Data Factory (ADF) pipeline that processes daily sales data from CSV files stored in Azure Blob Storage and loads it into an Azure SQL Database. The pipeline includes activities such as fetching metadata, iterating over files, copying data, and optionally calling a stored procedure for validation. It also provides design tips like using parameterized datasets, logging, and error handling strategies.

Uploaded by

lavanyayadav301998
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
34 views2 pages

ADF Class Notes

The document outlines a real-time use case for an Azure Data Factory (ADF) pipeline that processes daily sales data from CSV files stored in Azure Blob Storage and loads it into an Azure SQL Database. The pipeline includes activities such as fetching metadata, iterating over files, copying data, and optionally calling a stored procedure for validation. It also provides design tips like using parameterized datasets, logging, and error handling strategies.

Uploaded by

lavanyayadav301998
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Azure Data Factory (ADF) - Class Notes

Real-Time Use Case: Sales Data Load Pipeline

Scenario:

A retail company receives daily sales data from various branches as CSV files in an Azure Blob Storage

container.

These files need to be processed and loaded into an Azure SQL Database for reporting purposes.

ADF Pipeline Flow

1. Trigger:

- Schedule trigger runs the pipeline every night at 2 AM.

2. Activities:

- Get Metadata Activity:

Fetches file names from Blob Storage.

- ForEach Activity:

Iterates over each file returned by the metadata activity.

- Copy Activity (inside ForEach):

Source: CSV file from Blob

Sink: Azure SQL Sales table

- Stored Procedure Activity (optional):


Azure Data Factory (ADF) - Class Notes

Calls a procedure to validate and summarize data post-load.

3. Monitoring:

- Check the pipeline run status and performance in ADF Monitor.

Pipeline Design Tips

- Use parameterized datasets for dynamic file names and paths.

- Use logging tables in SQL DB to record successful/failed loads.

- Implement error handling using the Web Activity or Fail Activity.

You might also like