Learning project: Hands-on Azure Data Factory practice project by Prakash Kantumutchu (KP) implementing watermark-based incremental extraction from Azure SQL Database into Azure Data Lake Storage Gen2.
This repository stores the Git-integrated Azure Data Factory artifacts for a watermark-based incremental ingestion pipeline (pipeline/incrementalingestion.json):
- Lookup last watermark & current watermark: Reads the prior high-water mark from ADLS Gen2 (
dataset/Jsoncdc.json) and queries the current timestamp from Azure SQL Database (dataset/AzureSqlTable.json). - Incremental copy: Extracts rows added or modified since the last watermark and writes them as Parquet (
dataset/Parquet1.json) to ADLS Gen2 (linkedService/lsdatalakegen2.json). - Conditional watermark update: If the copy activity transfers one or more rows, updates the stored watermark file (
dataset/Jsonupdatedcdc.json) so subsequent runs only process new changes.
├── dataset/
│ ├── AzureSqlTable.json # Source Azure SQL table dataset
│ ├── Jsoncdc.json # Stored watermark JSON dataset
│ ├── Jsonempty.json # Empty JSON template dataset
│ ├── Jsonupdatedcdc.json # Updated watermark JSON dataset
│ └── Parquet1.json # Sink Parquet dataset on ADLS Gen2
├── factory/
│ └── azuredatafactorycsr212.json
├── linkedService/
│ ├── lsdatalakegen2.json # Azure Data Lake Storage Gen2 linked service
│ └── lssqldb.json # Azure SQL Database linked service
├── pipeline/
│ └── incrementalingestion.json # Watermark lookup + conditional copy pipeline
├── publish_config.json
└── LICENSE
MIT — see LICENSE.