This project was co-built by a 4-member team:
- Tirth Patel (@tirthpatel90): Led pipeline architecture, orchestration and the configuration engine.
- Hitesh (@hitesh0106): Contributed to pipeline modules.
- Prince (@prince3235): Contributed to data handling and machine learning algorithms.
- Riya (@riya119215): Contributed to data preprocessing work.
Contribution breakdown is available in the commit history.
⚠️ Note: This project is currently under active development. Some features may be incomplete or subject to change.
A modular and extensible MLOps pipeline that automatically handles different data formats and performs end-to-end data processing and model training.
- Data-Agnostic Ingestion – Supports CSV, Excel (.xlsx/.xls), and JSON files
- Auto Preprocessing – Automatically detects column types and handles:
- Missing value imputation (mean for numeric, mode for categorical)
- Label encoding for categorical columns
- Modular Architecture – Clean separation of concerns with dedicated modules
- Logging – Professional logging with timestamps for debugging
Data-agnostic-MLOps-pipeline/
├── main.py # Entry point - runs the data pipeline
├── requirements.txt # Python dependencies
├── configs/
│ └── params.yaml # Configuration parameters
├── data/
│ ├── raw/ # Input data files
│ └── processed/ # Cleaned output data
├── pipelines/
│ ├── train_pipeline.py # Training pipeline orchestration
│ └── inference_pipeline.py # Inference pipeline (WIP)
├── src/
│ ├── ingestion/
│ │ └── ingest_data.py # Data loading module
│ ├── preprocessing/
│ │ └── preprocess.py # Data cleaning & encoding
│ ├── training/
│ │ └── train.py # Model training (Logistic Regression)
│ ├── validation/
│ │ └── validate_data.py # Data validation checks
│ ├── evaluation/
│ ├── deployment/
│ ├── monitoring/
│ ├── registry/
│ └── utils/
├── docker/
│ ├── Dockerfile
│ └── docker-compose.yml
└── tests/
└── test_pipeline.py
# Clone the repository
git clone https://github.com/yourusername/Data-agnostic-MLOps-pipeline.git
cd Data-agnostic-MLOps-pipeline
# Install dependencies
pip install -r requirements.txtpython main.pyThis will:
- Load raw data from
data/raw/ - Clean and preprocess the data
- Save processed data to
data/processed/clean_data.csv
- pandas
- numpy
- scikit-learn
- openpyxl
| Module | Description |
|---|---|
ingest_data.py |
Loads data from CSV/Excel/JSON based on file extension |
preprocess.py |
Handles missing values and encodes categorical columns |
train.py |
Trains Logistic Regression model |
validate_data.py |
Checks for missing values in dataset |
This project is open source and available under the MIT License.