👷♂️ DevOps-Centric Cloud Cost Governance & Automation
A comprehensive cloud billing automation tool built for Cloud Engineers and DevOps Engineers to gain visibility, control, and automation over cloud billing across AWS, Azure, and GCP.
- 🚨 Prevent unexpected cloud bills with real-time monitoring and alerts
- 🤖 Automate cloud cost monitoring through scheduled jobs and CI/CD integration
- 🏷️ Enforce tagging & cost ownership with automated compliance checks
- 🔍 Detect cost anomalies early using machine learning-based anomaly detection
- ⚙️ Integrate billing checks into automation workflows with CLI and API access
- AWS: Cost Explorer, EC2, RDS, Lambda, S3 integration
- Azure: Billing Management, Resource Manager, Compute, Storage APIs
- GCP: Cloud Billing, Compute Engine, Cloud Storage integration
- Cost breakdown by service, region, environment, and cost center
- Resource-level analysis with efficiency scoring and recommendations
- Trend analysis with seasonal pattern detection
- Advanced forecasting using machine learning models
- Budget threshold monitoring with configurable warning and critical levels
- Anomaly detection using Z-score, IQR, and percentage deviation methods
- Multi-channel notifications (email, webhooks, Slack integration)
- Escalation policies for critical alerts
- Automated tag validation for required tags (Environment, CostCenter, etc.)
- Cost ownership tracking through mandatory tagging policies
- Compliance reporting with violation detection and remediation suggestions
- Idle resource detection for compute instances, storage, and databases
- Cost optimization recommendations with AI-powered insights
- Rightsizing suggestions with potential savings calculations
- Automated optimization workflows for common cost-saving opportunities
- Quick win identification for low-effort, high-impact savings
- Advanced ML models - Linear Regression, Random Forest, Gradient Boosting
- Time series analysis with seasonal pattern detection and trend analysis
- Feature engineering - Lag features, rolling statistics, trend indicators
- Model comparison - Automatic model selection with accuracy metrics
- Confidence intervals - Probabilistic forecasting with risk assessment
- Flexible scheduling - Daily, weekly, monthly, quarterly automated reports
- Multiple output formats - JSON, CSV, HTML with PDF/Excel support
- Background processing - Thread-based scheduler for reliable automation
- Rich report content - Cost analysis, forecasts, optimization recommendations
- Email delivery - Automated distribution to stakeholders with attachments
- CLI tool for automation and scripting
- YAML configuration for infrastructure-as-code compatibility
- CI/CD pipeline integration examples
- Cron job scheduling for automated monitoring
- Python 3.8+ - Core automation framework
- Cloud SDKs - boto3 (AWS), azure-mgmt, google-cloud
- Data Processing - pandas, numpy for analysis
- Machine Learning - scikit-learn for forecasting
- Security - cryptography, keyring for credential management
- CLI - Click, Typer, Rich for command-line interface
- Configuration - YAML for declarative config
- Python 3.8 or higher
- Cloud provider credentials (AWS, Azure, or GCP)
- Required billing permissions in your cloud accounts
# Clone the repository
git clone https://github.com/NotHarshhaa/cloud-billing-automation.git
cd cloud-billing-automation
# Install in development mode
pip install -e .
# Or install from requirements
pip install -r requirements.txtCreate a configuration file:
# config/billing-config.yaml
debug: false
log_level: "INFO"
data_retention_days: 90
providers:
aws:
enabled: true
account_id: "123456789012"
regions: ["us-east-1", "us-west-2"]
tags_required: ["Environment", "CostCenter", "Owner"]
cost_center_tag: "CostCenter"
environment_tag: "Environment"
azure:
enabled: false
subscription_id: "your-subscription-id"
regions: ["eastus", "westus2"]
tags_required: ["Environment", "CostCenter"]
gcp:
enabled: false
project_id: "your-gcp-project"
regions: ["us-central1", "us-east1"]
tags_required: ["Environment", "CostCenter"]
budget:
monthly_limit: 10000.0
warning_threshold: 0.8 # 80%
critical_threshold: 0.95 # 95%
currency: "USD"
alert_emails: ["[email protected]"]
alert_webhooks: ["https://hooks.slack.com/your-webhook"]
reports:
output_dir: "reports"
formats: ["json", "csv", "html"]
schedule: "daily"
include_charts: true
email_reports: true
email_recipients: ["[email protected]"]# AWS credentials
cba credentials setup-aws \
--access-key-id YOUR_ACCESS_KEY \
--secret-access-key YOUR_SECRET_KEY
# Azure service principal
cba credentials setup-azure \
--tenant-id YOUR_TENANT_ID \
--client-id YOUR_CLIENT_ID \
--client-secret YOUR_CLIENT_SECRET \
--subscription-id YOUR_SUBSCRIPTION_ID
# GCP service account
cba credentials setup-gcp \
--service-account-key-path /path/to/service-account.jsonThe cloud-billing-automation tool includes a comprehensive alerting system with multi-channel notifications, escalation policies, and intelligent alert management.
- Budget Alerts - Threshold-based budget monitoring (warning, critical, exceeded)
- Forecast Alerts - Predictive alerts based on spending trends
- Anomaly Alerts - Statistical detection of cost outliers
- Trend Alerts - Spending pattern changes and escalations
- Resource Alerts - Individual resource cost issues
- Email - Rich HTML emails with detailed information and recommendations
- Slack - Color-coded messages with interactive elements
- Webhook - JSON payloads for integration with monitoring systems
- Custom Channels - Extensible channel system for custom integrations
- Severity Levels - Low, Medium, High, Critical with appropriate escalation
- Cooldown Periods - Prevent alert spam with configurable cooldowns
- Alert Suppression - Temporary suppression for maintenance windows
- Template System - Jinja2-based dynamic alert templates
- Alert History - Comprehensive tracking and analytics
- Smart Grouping - Reduces noise by grouping related alerts
- Threshold-Based - Different actions for different severity levels
- Multi-Channel Routing - Critical alerts go to all channels, warnings to email only
- Time-Based Escalation - Escalate unacknowledged critical alerts
- Auto-Resolution - Automatically resolve alerts when conditions normalize
The cloud-billing-automation tool includes a comprehensive command-line interface with rich output, intuitive commands, and professional error handling.
# Install the tool
pip install cloud-billing-automation
# Initialize configuration
cba init --output-dir config
# Set up credentials
cba credentials setup-aws --access-key-id YOUR_KEY --secret-access-key YOUR_SECRET
cba credentials setup-azure --tenant-id TENANT_ID --client-id CLIENT_ID --client-secret SECRET --subscription-id SUB_ID
cba credentials setup-gcp --service-account-key-path /path/to/key.json
# Validate credentials
cba credentials validate
cba credentials status
# List stored credentials
cba credentials list
# Export credentials
cba credentials export aws --output-file aws-credentials.shcba [GLOBAL_OPTIONS] COMMAND [COMMAND_OPTIONS]
Global Options:
-c, --config FILE Configuration file path
--debug Enable debug mode
-v, --verbose Enable verbose output
--help Show help message# Analyze costs for specific period
cba analyze costs \
--start-date 2024-01-01 \
--end-date 2024-01-31 \
--providers aws,azure \
--output table
# Get cost breakdown by service
cba analyze breakdown \
--group-by service \
--period monthly
# Detect cost anomalies
cba analyze anomalies \
--methods zscore,iqr,percentage \
--threshold 2.0
# Analyze cost trends
cba analyze trends \
--metrics total_cost \
--period weekly
# Generate cost forecast with ML models
cba analyze forecast \
--days 30 \
--model random_forest# Analyze cost optimization opportunities
cba optimize analyze \
--start-date 2024-01-01 \
--end-date 2024-01-31 \
--min-savings 50 \
--effort low
# Find quick win opportunities
cba optimize quick-wins \
--config config/billing-config.yaml
# Get recommendations by type
cba optimize by-type unused_resources
cba optimize by-type rightsizing
cba optimize by-type scheduled_shutdown# Check current budget status
cba budget status \
--config config/billing-config.yaml \
--forecast
# Set monthly budget limit
cba budget set-limit 10000
# Set up budget alerts
cba budget alerts --setup
# Test alert channels
cba budget alerts --test
# View budget history
cba budget history --days 30
# Generate budget forecast
cba budget forecast --days 30# Test notification channels
cba alerts test \
--channels email,slack,webhook
# View alert history
cba alerts history \
--days 30 \
--severity critical,high \
--status active
# Suppress alerts for maintenance
cba alerts suppress \
--resource-id i-1234567890abcdef0 \
--duration-hours 12 \
--reason "Scheduled maintenance"
# Acknowledge an alert
cba alerts acknowledge alert-id-123 --by "John Doe"
# Resolve an alert
cba alerts resolve alert-id-123
# Check alert system status
cba alerts status
# Manage alert rules
cba alerts rules --list
cba alerts rules --enable budget_warning
cba alerts rules --disable anomaly_detection- Rich Terminal Output - Beautiful tables, panels, and colored output
- Progress Indicators - Progress bars for long-running operations
- Error Handling - User-friendly error messages and validation
- Auto-completion - Command auto-completion (bash/zsh)
- Help System - Comprehensive help for all commands
- Configuration Detection - Automatic config file discovery
- Multiple Output Formats - Table, JSON, CSV output options
Checking budget status...
Budget Status - Warning
┌─────────────────┬──────────────┐
│ Metric │ Value │
├─────────────────┼──────────────┤
│ Budget Limit │ $10,000.00 │
│ Current Spend │ $8,234.56 │
│ Usage │ 82.3% │
│ Remaining │ $1,765.44 │
│ Days Remaining │ 12 │
└─────────────────┴──────────────┘
Risk Assessment
─────────────────────────────────
Risk Level: HIGH
Daily Average: $274.49
Projected Daily: $147.11
Loading alert history for last 30 days...
Alert History (15 alerts)
┌──────────┬─────────────────────────────┬──────────┬──────────────┬─────────┬─────────────────────┐
│ ID │ Title │ Severity │ Status │ Source │ Time │
├──────────┼─────────────────────────────┼──────────┼──────────────┼─────────┼─────────────────────┤
│ 1234... │ Budget Warning: 82.3% used │ medium │ active │ Budget │ 2024-01-15 14:30 │
│ 5678... │ Cost Anomaly Detected │ high │ resolved │ Anomaly │ 2024-01-14 09:15 │
│ 9012... │ Budget Critical: 95.1% used │ critical │ acknowledged │ Budget │ 2024-01-13 16:45 │
└──────────┴─────────────────────────────┴──────────┴──────────────┴─────────┴─────────────────────┘
# Analyze costs for the last 30 days
cba analyze costs \
--start-date 2024-01-01 \
--end-date 2024-01-31 \
--config config/billing-config.yaml
# Get cost breakdown by service
cba analyze breakdown \
--group-by service \
--period monthly
# Detect cost anomalies
cba analyze anomalies \
--methods zscore,iqr,percentage \
--threshold 2.0# Check budget status
cba budget status \
--config config/billing-config.yaml
# Set up budget alerts with multiple channels
cba budget alerts setup \
--emails [email protected],[email protected] \
--slack-webhook https://hooks.slack.com/your-webhook \
--webhook https://monitoring.company.com/webhooks/billing
# Test alert channels
cba alerts test \
--channels email,slack,webhook
# View alert history
cba alerts history \
--days 30 \
--severity critical,high
# Suppress alerts for maintenance
cba alerts suppress \
--resource-id i-1234567890abcdef0 \
--duration-hours 12 \
--reason "Scheduled maintenance"# config/alerts.yaml
notifications:
channels:
email:
type: email
smtp_server: smtp.company.com
smtp_port: 587
username: [email protected]
password: ${EMAIL_PASSWORD}
from_email: [email protected]
to_emails:
- [email protected]
- [email protected]
use_tls: true
slack:
type: slack
webhook_url: ${SLACK_WEBHOOK_URL}
channel: "#billing-alerts"
username: "CloudBillingBot"
icon_emoji: ":moneybag:"
webhook:
type: webhook
url: https://monitoring.company.com/webhooks/billing
method: POST
headers:
Authorization: "Bearer ${WEBHOOK_TOKEN}"
Content-Type: "application/json"
timeout: 30
retry_count: 3
alert_rules:
budget_warning:
enabled: true
severity: medium
cooldown_period: 60 # minutes
channels: [email, slack]
budget_critical:
enabled: true
severity: high
cooldown_period: 30
channels: [email, slack, webhook]
anomaly_detection:
enabled: true
min_confidence: 0.7
min_deviation_percentage: 20.0
channels: [email]# Create configuration file
cba config create --template production --output-file billing-config.yaml
# Validate configuration
cba config validate --config billing-config.yaml
# Show current configuration
cba config show --section providers
# Edit configuration interactively
cba config edit --section budget
# Merge configurations
cba config merge source-config.yaml --target-file billing-config.yaml --strategy merge# Generate comprehensive cost report
cba reports generate \
--period monthly \
--formats html,pdf \
--email [email protected] \
--include-forecasts \
--include-optimizations
# Schedule automated reports
from cloud_billing_automation.reports.scheduler import ReportScheduler
scheduler = ReportScheduler(config)
scheduler.add_report(ReportConfig(
name="monthly-cost-report",
schedule=ReportSchedule.MONTHLY,
format=ReportFormat.HTML,
recipients=["[email protected]", "[email protected]"],
include_forecasts=True,
include_optimizations=True
))
scheduler.start_scheduler()
# Run report immediately
cba reports run-now \
--report-name monthly-cost-report \
--output-format htmlcloud-billing-automation/
├── cloud_billing_automation/ # Main package
│ ├── core/ # Core infrastructure
│ │ ├── config.py # Configuration management
│ │ ├── credentials.py # Secure credential handling
│ │ └── exceptions.py # Custom exceptions
│ ├── collectors/ # Data collection
│ │ ├── base.py # Base collector interface
│ │ ├── aws_collector.py # AWS billing data
│ │ ├── azure_collector.py # Azure billing data
│ │ └── gcp_collector.py # GCP billing data
│ ├── analyzers/ # Cost analysis
│ │ ├── cost.py # Cost analysis & breakdown
│ │ ├── anomaly.py # Anomaly detection
│ │ ├── trend.py # Trend analysis
│ │ ├── forecast.py # Cost forecasting
│ │ ├── optimizer.py # Cost optimization recommendations
│ │ └── ml_forecaster.py # Machine learning forecasting
│ ├── alerts/ # Alerting system
│ │ ├── base.py # Base alert management
│ │ ├── budget.py # Budget alert manager
│ │ ├── anomaly.py # Anomaly alert manager
│ │ ├── channels.py # Notification channels
│ │ └── templates.py # Alert templates
│ ├── reports/ # Report generation
│ │ ├── scheduler.py # Automated report scheduler
│ │ └── generators.py # Report format generators
│ ├── utils/ # Security and utility modules
│ ├── security.py # IAM and access control
│ ├── encryption.py # Data encryption utilities
│ ├── validation.py # Input validation and sanitization
│ └── helpers.py # Formatting and data utilities
│ ├── cli/ # Command-line interface
│ │ ├── main.py # Main CLI application
│ │ └── commands/ # CLI command modules
│ │ ├── analyze.py # Cost analysis commands
│ │ ├── budget.py # Budget monitoring commands
│ │ ├── alerts.py # Alert management commands
│ │ ├── optimize.py # Cost optimization commands
│ │ ├── credentials.py # Credential management commands
│ │ └── config.py # Configuration management commands
├── tests/ # Test suite
├── config/ # Configuration examples
├── docs/ # Documentation
├── examples/ # Usage examples
├── pyproject.toml # Project configuration
├── requirements.txt # Dependencies
├── setup.py # Setup script
└── README.md # This file
# Run all tests
pytest
# Run with coverage
pytest --cov=cloud_billing_automation
# Run specific test categories
pytest tests/unit/
pytest tests/integration/# Format code
black . tests/
# Lint code
flake8 . tests/
# Type checking
mypy .- Create a new collector class inheriting from
BaseCollector - Implement required abstract methods:
authenticate()collect_billing_data()collect_resource_data()get_cost_breakdown()
- Add provider configuration to
Configclass - Update CLI commands and documentation
- Create a new command module in
cli/commands/ - Inherit from
typer.Typerfor command structure - Implement command logic with proper error handling
- Add rich output formatting using
rich.console.Console - Register the command in the main CLI app
- Create a new channel class inheriting from
BaseChannel - Implement
send()andtest_connection()methods - Add channel type to
ChannelManager._initialize_channels() - Create alert templates for the new channel
- Update configuration schema
-
Core Infrastructure
- ✅ Configuration management (YAML + environment variables)
- ✅ Secure credential storage (encrypted keyring)
- ✅ Comprehensive error handling
- ✅ Project structure and packaging
-
Data Collection & Analysis
- ✅ Multi-cloud billing data collection (AWS, Azure, GCP)
- ✅ Cost analysis and breakdown by multiple dimensions
- ✅ Anomaly detection with statistical methods
- ✅ Trend analysis and forecasting
- ✅ Resource-level cost analysis
- ✅ Machine Learning forecasting with advanced models
- ✅ Cost optimization recommendations with AI insights
-
Budget Monitoring & Alerting
- ✅ Budget threshold monitoring and alerts
- ✅ Multi-channel notifications (Email, Slack, Webhook)
- ✅ Alert escalation policies and suppression
- ✅ Dynamic alert templates (Jinja2)
- ✅ Real-time monitoring and history tracking
-
Cost Optimization & Automation
- ✅ Intelligent optimization recommendations (rightsizing, unused resources, scheduled shutdowns)
- ✅ Quick win identification for low-effort savings
- ✅ Automated report scheduler with flexible scheduling
- ✅ Multi-format report generation (JSON, CSV, HTML)
- ✅ Background processing for reliable automation
-
CLI Interface
- ✅ Rich terminal output with tables and progress bars
- ✅ Comprehensive command structure (analyze, budget, alerts, optimize)
- ✅ Professional error handling and validation
- ✅ Interactive configuration management
- ✅ Credential management commands
- ✅ Cost optimization commands with filtering options
-
IAM & Security
- ✅ Role-based access control (RBAC)
- ✅ JWT-based session management
- ✅ Secure credential encryption (AES-256)
- ✅ Comprehensive audit logging
- ✅ Multi-cloud validation and sanitization
- ✅ Security utilities and helper functions
- Tag Compliance - Automated tag validation and enforcement (next priority)
- Advanced Anomaly Detection - Deep learning models for complex pattern detection
- Multi-Cloud Resource Discovery - Automated resource inventory across providers
- Cost Allocation Engine - Advanced cost attribution and showback/chargeback
- Integration Marketplace - Pre-built integrations with popular monitoring tools
- Mobile App - Native mobile applications for on-the-go cost monitoring
Released: February 2026
- 🤖 Machine Learning Forecasting - Advanced ML models (Linear Regression, Random Forest, Gradient Boosting) with time series analysis and confidence intervals
- 🎯 Cost Optimization Engine - AI-powered recommendations for rightsizing, unused resources, scheduled shutdowns, and storage optimization
- 📊 Automated Report Scheduler - Background report generation with flexible scheduling (daily, weekly, monthly, quarterly) and multiple output formats
cba optimize analyze- Comprehensive cost optimization analysiscba optimize quick-wins- Low-effort, high-impact savings opportunitiescba optimize by-type <type>- Filter recommendations by type
- Enhanced progress indicators for long-running operations
- Improved error handling with structured logging
- Better configuration validation and management
- Optional sklearn dependency with graceful fallbacks
- ML-based forecasting with model comparison and accuracy metrics
- Intelligent optimization recommendations with confidence scores
- Automated report scheduling with email delivery
- Rich CLI output with progress bars and status indicators
Released: February 2026
- Added comprehensive logging configuration with Rich formatting
- Enhanced error handling and validation throughout the application
- Fixed configuration bugs and improved CLI structure
- Added progress indicators for data collection operations
- Improved package structure and dependency management
Released: February 2026
- Multi-cloud billing data collection (AWS, Azure, GCP)
- Cost analysis and breakdown by multiple dimensions
- Budget monitoring and alerting system
- Anomaly detection with statistical methods
- Rich CLI interface with professional output
- Comprehensive credential management
- Multi-channel notifications (Email, Slack, Webhook)
- Role-Based Access Control (RBAC) - Granular permissions with roles (admin, operator, viewer, billing_manager, devops)
- JWT Session Management - Secure token-based authentication with configurable timeouts
- AES-256 Encryption - Military-grade encryption for all sensitive data and credentials
- Comprehensive Audit Logging - Complete audit trail for all security events and operations
- Multi-Cloud Validation - Input validation and sanitization for AWS, Azure, and GCP
- Security Utilities - Password hashing, IP validation, secure file operations
- Account Lockout - Automatic account lockout after failed login attempts
- Session Revocation - Immediate session termination capabilities
- Secure Key Storage - Encrypted key management with restrictive file permissions
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
Built with passion and purpose by Harshhaa.
Your ideas, feedback, and contributions are what make this project better.
⚡ Made with ❤️ for DevOps Engineers by Harshhaa
Connect & Collaborate:
- GitHub: @NotHarshhaa
- Links: Links - Portfolio
- Portfolio: Portfolio
- CV Portfolio: CV Portfolio
- LinkedIn: Harshhaa Vardhan Reddy
If you found this project useful:
- ⭐ Star the repository to show your support
- 📢 Share it with your friends and colleagues
- 📝 Open issues or submit pull requests to help improve it
Join the community, share your experience, and help us grow!