Skip to content

RogoLabs/CVE-Updates

BranchesTags

Repository files navigation

CVE Update Analysis

CVE Update Report Python 3.11+ License

Automated analysis of CVE update patterns in the official CVEProject/cvelist repository. This project tracks how frequently CVE records are updated, identifies the most actively maintained CVEs, and provides insights into vulnerability disclosure and maintenance patterns.

📊 Live Report

View the latest CVE Update Analysis Report →

The report is automatically updated every 4 hours with the latest data from the CVE repository.

🎯 Features

  • 🌐 Interactive Website: Single-page responsive HTML report deployed via GitHub Pages
  • 📊 Professional Visualizations: Publication-quality graphs with blue color scheme (300 DPI)
  • 📈 Statistical Analysis: Update distributions, trends, and comprehensive metrics
  • 🏆 Top Updated CVEs: Identifies most actively maintained vulnerability records
  • 📅 Historical Trends: Commit activity over time with rolling averages
  • ⏱️ Update Velocity Analysis: Time from publication to metadata update with distribution insights
  • ⚡ Fast Analysis: Process 300K+ CVEs in under 2 minutes
  • 🤖 Automated Updates: GitHub Actions workflow runs every 4 hours
  • 💾 Multiple Formats: JSON data, Markdown reports, PNG/SVG graphs, HTML website

📸 Visualizations

The analysis generates six key visualizations:

  1. Update Distribution: Histogram showing how many CVEs have N updates
  2. Cumulative Distribution (CDF): Percentage of CVEs with N or fewer updates
  3. Top Updated CVEs: Bar chart of the 50 most frequently updated CVEs
  4. Commits Over Time: Monthly commit activity with 12-month rolling average
  5. Updates Frequency (Log Scale): Logarithmic view of update patterns
  6. Time to Metadata Update: Distribution of time from publication to metadata update (cveMetadata.dateUpdated)

All graphs use a professional blue color scheme and are available in both PNG (300 DPI) and SVG formats.

🚀 Quick Start

Prerequisites

  • Python 3.11 or higher
  • Git (for analyzing CVE repository history)

Installation

  1. Clone this repository:
git clone https://github.com/RogoLabs/CVE-Updates.git
cd CVE-Updates

### Installation

1. Clone this repository:
```bash
git clone https://github.com/RogoLabs/CVE-Updates.git
cd CVE-Updates
  1. Install dependencies:
pip install -r requirements.txt
  1. Clone the CVE database (optional, if not already present):
git clone https://github.com/CVEProject/cvelistV5.git

Running the Analysis

Generate a complete CVE update report:

# 1. Analyze the CVE database (creates reports/report.json)
python analyze_cvelist.py --ci --verbose

# 2. Generate visualizations (creates PNG and SVG files)
python generate_report_assets.py --report reports/report.json --outdir reports --top 50

# 3. Create Markdown summary (creates report.md)
python generate_markdown_report.py --ci

# 4. Generate HTML website (creates web/index.html)
python generate_website.py --report reports/report.json --outdir web

Viewing Results

📊 Output Files

After running the analysis, you'll find:

CVE-Updates/
├── report.md                          # Markdown summary report
├── reports/
│   ├── report.json                    # Raw analysis data
│   ├── updates_distribution.png       # Update frequency bar chart
│   ├── updates_distribution.svg       # (SVG version)
│   ├── updates_cdf.png               # Cumulative distribution
│   ├── updates_cdf.svg               # (SVG version)
│   ├── top_updated_cves.png          # Top CVEs chart
│   ├── top_updated_cves.svg          # (SVG version)
│   ├── commits_over_time.png         # Time series
│   ├── commits_over_time.svg         # (SVG version)
│   ├── updates_frequency_log.png     # Log-scale distribution
│   ├── updates_frequency_log.svg     # (SVG version)
└── web/
    ├── index.html                     # Single-page website
    ├── .nojekyll                      # GitHub Pages config
    └── README.md                      # Web directory docs

🛠️ Command-Line Options

analyze_cvelist.py

Main analysis script that processes CVE files and generates statistics.

python analyze_cvelist.py [OPTIONS]

Options:
  --repo-path PATH       Path to cvelistV5 repository (default: ./cvelistV5)
  --ci                   CI mode: write report to reports/report.json
  --indepth              Use per-file git log (slower but authoritative)
  --tolerance SECONDS    Date mismatch tolerance in seconds (default: 3600)
  --top N                Number of top CVEs to include (default: 50)
  --verbose              Enable detailed logging
  --help                 Show help message

generate_report_assets.py

Creates publication-quality visualizations from analysis data.

python generate_report_assets.py [OPTIONS]

Options:
  --report PATH          Path to report JSON (default: reports/report.json)
  --outdir PATH          Output directory for graphs (default: reports)
  --top N                Number of top CVEs to visualize (default: 30)
  --verbose              Enable detailed logging
Generates publication-quality visualizations from analysis data.

```bash
python generate_report_assets.py [OPTIONS]

Options:
  --report PATH          Path to report.json (required)
  --outdir PATH          Output directory for graphs (default: reports)
  --top N                Number of top CVEs to show (default: 50)
  --verbose              Enable detailed logging
  --help                 Show help message

generate_markdown_report.py

Generates a formatted Markdown report with embedded visualizations.

python generate_markdown_report.py [OPTIONS]

Options:
  --report PATH          Path to JSON report (default: auto-detect latest)
  --output PATH          Output markdown path (default: derived from report)
  --top N                Number of top CVEs in table (default: 10)
  --ci                   CI mode: write to report.md in repo root
  --verify-top           Run per-file git queries for top CVEs (slow)
  --repo-path PATH       CVE repository path for --verify-top
  --verbose              Enable detailed logging
  --help                 Show help message

generate_website.py

Generates a single-page HTML website with embedded visualizations.

python generate_website.py [OPTIONS]

Options:
  --report PATH          Path to report.json (required)
  --outdir PATH          Output directory (default: web)
  --verbose              Enable detailed logging
  --help                 Show help message

🤖 Automated Reports (GitHub Actions)

The repository includes a GitHub Actions workflow that automatically generates and deploys reports every 4 hours.

Workflow Features

  • Scheduled Runs: Executes every 4 hours automatically
  • Manual Trigger: Can be run on-demand via GitHub UI
  • Artifact Upload: All reports and graphs available for 7 days
  • GitHub Pages Deployment: Website automatically deployed to gh-pages branch
  • Fast Execution: Completes in 2-3 minutes

Workflow Steps

  1. Checkout repository with full Git history
  2. Set up Python 3.11 environment
  3. Install dependencies from requirements.txt
  4. Run CVE analysis (generates report.json)
  5. Generate visualizations (PNG + SVG)
  6. Create Markdown report
  7. Build HTML website
  8. Upload artifacts (JSON, Markdown, graphs, website)
  9. Deploy website to GitHub Pages

Accessing Reports

Live Website: https://rogolabs.github.io/CVE-Updates/

Artifacts (last 7 days):

  1. Go to the Actions tab
  2. Click on the latest "CVE Update Report" workflow run
  3. Scroll to the Artifacts section
  4. Download any of:
    • cve-report-json - Raw data (JSON)
    • cve-report-markdown - Formatted summary
    • cve-report-graphs-png - PNG visualizations
    • cve-report-graphs-svg - SVG visualizations
    • cve-report-website - Complete website

Manual Workflow Trigger

To run the workflow manually:

  1. Navigate to Actions → CVE Update Report
  2. Click Run workflow button
  3. Select branch (usually main)
  4. Click Run workflow

🌐 GitHub Pages Deployment

The website is automatically deployed to GitHub Pages after each successful workflow run.

Initial Setup

To enable GitHub Pages for this repository:

  1. Go to repository Settings → Pages
  2. Under Source, select:
    • Branch: gh-pages
    • Folder: / (root)
  3. Click Save
  4. Website will be available at: https://rogolabs.github.io/CVE-Updates/

How It Works

  1. GitHub Actions workflow runs analysis and generates website
  2. Website files (HTML with embedded images) are created in web/ directory
  3. peaceiris/actions-gh-pages action deploys web/ directory to gh-pages branch
  4. GitHub Pages serves the website from gh-pages branch
  5. .nojekyll file ensures GitHub doesn't process HTML through Jekyll

Deployment Details

  • Branch: gh-pages (auto-created by workflow)
  • Deploy Frequency: Every 4 hours (or on push to main)
  • Content: Self-contained HTML with base64-encoded images
  • No External Dependencies: All assets embedded in HTML

📋 Report Contents

The generated website and reports include:

  1. Executive Summary: Key statistics at a glance
  2. Update Distribution: Visual and tabular breakdown
  3. Top Updated CVEs: Most frequently updated entries with CNA info and direct links
  4. Update Velocity Analysis: Time from publication to metadata update with distribution graphs
  5. Longest Time to Update: CVEs with the most extended maintenance cycles
  6. Analysis Methodology: How the analysis works
  7. Supporting Visualizations: Additional graphs for deeper insights
  8. Technical Details: Analysis parameters and metadata

Key Metrics

  • Total CVEs Analyzed: Number of CVE records in repository
  • CVEs with Updates: CVEs modified after initial publication
  • Total Git Commits: All commits affecting CVE files
  • Never Updated CVEs: Count and percentage of CVEs with zero updates
  • Update Statistics: Mean, median, percentiles (75th, 90th, 95th)
  • Top CVEs: Ranked by update frequency with CNA attribution
  • Time-to-Update Statistics: Median, mean, and distribution of time from publication to metadata update

🔍 Understanding the Analysis

What is an "Update"?

An update is defined as:

updates = max(0, totalGitCommits - 1)

This means:

  • 0 updates: CVE created and never modified
  • 1 update: CVE created, then modified once
  • N updates: CVE modified N times after creation

Fast vs. In-Depth Mode

Fast Mode (default):

  • Uses aggregated git log across entire CVE tree
  • Processes 300K+ CVEs in ~60 seconds
  • Recommended for regular analysis

In-Depth Mode (--indepth):

  • Runs individual git log per CVE file
  • More authoritative but slower (2-3 minutes)
  • Use when accuracy is critical

Date Mismatch Detection

The tool compares:

  • Internal Date: cveMetadata.dateUpdated field in CVE JSON
  • Git Date: Latest commit timestamp for the file

A mismatch is flagged when the internal date is more than tolerance seconds (default: 3600) later than the Git date, which may indicate:

  • CVE metadata updated without committing the file
  • Clock skew issues
  • Manual metadata edits

Time-to-Update Analysis

The tool tracks the time elapsed between CVE publication and metadata update:

  • Start: cveMetadata.datePublished (when CVE was initially published)
  • End: cveMetadata.dateUpdated (most recent metadata update timestamp)
  • Filter: Only includes CVEs with totalGitCommits > 1 (confirmed updates)

Note: This measures time to the most recent metadata update timestamp from the JSON file, not the date of the first Git commit.

Statistics include:

  • Median and mean time to update
  • Percentage updated within 24 hours, 7 days, 30 days, and 1 year
  • Distribution across time windows (< 1 day, 1-7 days, 7-30 days, etc.)
  • Identification of CVEs with longest update delays

📦 Dependencies

  • GitPython (≥3.1.0): Git repository analysis
  • pandas (≥2.2.3): Data manipulation and analysis
  • matplotlib (≥3.10.1): Graph generation
  • seaborn (≥0.13.2): Statistical visualizations
  • numpy (≥1.26.4): Numerical computations

Install all dependencies:

pip install -r requirements.txt

🎨 Customization

Changing Graph Colors

Edit generate_report_assets.py and modify the MAIN_COLORMAP variable:

# Current: Blue palette
MAIN_COLORMAP = "Blues"

# Alternatives:
MAIN_COLORMAP = "Greens"   # Green palette
MAIN_COLORMAP = "Reds"     # Red palette
MAIN_COLORMAP = "Viridis"  # Viridis colormap

Adjusting Graph Sizes

Modify the figure sizes in generate_report_assets.py:

plt.rcParams.update({
    "figure.figsize": (16, 8),  # Change to your preferred size
})

Custom Analysis Filters

To analyze specific CVE years or patterns, modify the is_cve_filename function in analyze_cvelist.py.

🐛 Troubleshooting

"No module named 'git'"

Install GitPython:

pip install GitPython

"cvelistV5 repository not found"

Clone the CVE database:

git clone https://github.com/CVEProject/cvelistV5.git

Or specify a different path:

python analyze_cvelist.py --repo-path /path/to/your/cvelistV5

Matplotlib Backend Errors

If you encounter display errors, set the backend:

export MPLBACKEND=Agg
python generate_report_assets.py ...

Out of Memory

For very large analyses, increase Python's memory or filter the dataset:

# Limit analysis to specific years (requires code modification)
# Or use a machine with more RAM

📈 Performance

Typical performance on a modern machine:

Operation CVE Count Time
Fast Analysis 314,685 ~60 seconds
In-Depth Analysis 314,685 ~2-3 minutes
Graph Generation All graphs ~10 seconds
Report Generation Full report ~1 second

Total pipeline: 2-3 minutes for complete analysis with all visualizations.

🤝 Contributing

Contributions are welcome! Please feel free to submit pull requests or open issues for:

  • Bug fixes
  • New visualizations
  • Performance improvements
  • Documentation enhancements
  • Feature requests

Development Setup

# Clone the repository
git clone https://github.com/RogoLabs/CVE-Updates.git
cd CVE-Updates

# Install dependencies
pip install -r requirements.txt

# Run tests (if available)
python -m pytest

# Make your changes and test
python analyze_cvelist.py --ci --verbose

📄 License

This project is licensed under the terms specified in the LICENSE file.

🔗 Related Projects

📧 Contact

For questions, issues, or suggestions:

🙏 Acknowledgments

  • CVE Program and the CVE Numbering Authorities (CNAs)
  • MITRE Corporation for CVE database maintenance
  • Open source community for the excellent Python data science libraries

Made with ❤️ by RogoLabs

Helping security professionals understand CVE update patterns

About

Tracks CVEs Updates

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages