Data orchestration pipeline using Apache Airflow. Implements the Bronze/Silver/Gold medallion architecture for claims data processing with data quality validation, SCD Type-2 dimension management, and operational monitoring.
┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Raw Source │───>│ Bronze │───>│ Silver │───>│ Gold │
│ (CSV/API) │ │ (as-is + │ │ (cleaned, │ │ (aggregates,│
│ │ │ metadata) │ │ dedup, SCD) │ │ features) │
└─────────────┘ └─────────────┘ └──────────────┘ └─────────────┘
│ │ │
┌────┴────┐ ┌─────┴────┐ ┌────┴─────┐
│Validate │ │Transform │ │ Quality │
│ Schema │ │ & Merge │ │ Gates │
└─────────┘ └──────────┘ └──────────┘
- Bronze: Raw data ingested as-is with added metadata (ingestion timestamp, source, batch ID). No transformations.
- Silver: Cleaned, deduplicated, and type-cast data. SCD Type-2 logic applied for dimension changes. Referential integrity enforced.
- Gold: Business-level aggregated tables and feature sets ready for analytics and ML consumption.
airflow-etl-pipeline/
├── dags/
│ ├── claims_etl_dag.py # Main daily ETL DAG
│ └── model_retraining_dag.py # Weekly model retraining DAG
├── plugins/
│ ├── quality_checks.py # Great Expectations validation
│ └── transformations.py # Data transformation utilities
├── scripts/
│ └── init_airflow.sh # Airflow initialization script
├── configs/
│ └── config.yaml # Pipeline configuration
├── data/
│ ├── raw/
│ ├── bronze/
│ ├── silver/
│ └── gold/
├── requirements.txt
└── README.md
git clone https://github.com/ln-bodicherla/airflow-etl-pipeline.git
cd airflow-etl-pipeline
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# Initialize Airflow
bash scripts/init_airflow.shRuns daily. Processes raw claims data through the full medallion architecture:
- extract_raw_claims -- Reads source data (CSV files or simulated API calls)
- validate_raw_data -- Runs Great Expectations validation suite on raw data
- bronze_ingestion -- Writes raw data to Bronze layer with metadata
- silver_transformation -- Cleans, deduplicates, casts types, applies SCD-2
- gold_aggregation -- Builds provider-level, patient-level, and temporal aggregates
- data_quality_check -- Validates Gold layer against quality gates
- notify_completion -- Sends pipeline completion notification
Runs weekly. Retrains ML models using data from the Gold layer:
- check_data_drift -- Statistical tests for distribution shift
- extract_training_data -- Pulls feature tables from Gold
- train_model -- Trains a new model version
- evaluate_model -- Evaluates on holdout set
- compare_champion -- Compares new model vs. current champion
- promote_model -- Promotes the new model if it outperforms (conditional)
- update_registry -- Updates the model registry
All pipeline parameters are centralized in configs/config.yaml:
| Section | Key | Description |
|---|---|---|
airflow.schedule_interval |
Cron or preset schedule | @daily |
paths.bronze |
Bronze layer path | ./data/bronze |
quality.max_null_rate |
Maximum null fraction per column | 0.05 |
quality.min_completeness |
Minimum row completeness | 0.95 |
notifications.on_failure |
Alert on DAG failure | true |
Quality is enforced at multiple checkpoints using Great Expectations:
- Schema validation: Column presence, data types, allowed values
- Completeness: Null rate thresholds per column
- Distribution checks: Statistical bounds on numeric columns
- Referential integrity: Foreign key consistency between tables
- Business rules: Claim amount ranges, valid date sequences
The Silver layer implements Slowly Changing Dimension Type 2 for tracking historical changes to dimension attributes (provider info, patient demographics). Each dimension row carries:
effective_start_dateeffective_end_dateis_currentflag
- Python 3.9+
- Apache Airflow 2.8+
- (Optional) Spark 3.5+ for large-scale processing
MIT License