GitHub Data Scraper

A Node.js/TypeScript tool to analyze language statistics across all your GitHub repositories (both public and private) to identify repos with high JavaScript percentages.

Purpose

This tool helps identify which repositories might have committed node_modules or other JavaScript files that are inflating your GitHub language statistics.

Setup

1. Install dependencies:

   pnpm install

2. Create a GitHub Personal Access Token:

   Create your token here: https://github.com/settings/tokens/new

   • Note: Give it a descriptive name (e.g., "GitHub Stats Analyzer")
   • Expiration: Choose your preferred expiration (recommended: 90 days)
   • Select scopes: Check the following scope:
     • ✅ repo - Full control of private repositories
       • This includes access to public repositories, private repositories, and archived repositories
   • Click "Generate token" at the bottom of the page
   • Important: Copy the token immediately - you won't be able to see it again!

3. Configure environment variables:

   cp .env.example .env

   Edit the .env file and add your token:

   GITHUB_TOKEN=your_actual_token_here
   

Usage

Run the analyzer:

pnpm start

What It Does

The tool will:

1. Fetch all your repositories (public and private)
2. Get language statistics for each repository
3. Scan for suspicious files (node_modules, .env, build outputs, etc.)
4. Flag repositories with committed node_modules
5. Calculate JavaScript percentages
6. Display results sorted by JavaScript percentage
7. Show repositories with >50% JavaScript (likely suspects)
8. Calculate your overall language distribution across all repos
9. Generate comprehensive reports in multiple formats

Output

Console Output

The tool displays:

• Progress updates: Real-time progress as it analyzes each repo
• Repository list: All repos with JavaScript, sorted by JS percentage
• Detailed stats: For each repo, shows all languages and their percentages
• Suspects: Highlights repos with >50% JavaScript
• node_modules warnings: Lists any repos with committed node_modules
• Overall distribution: Your actual language breakdown across all repos

Generated Reports

All reports are saved in the reports/ directory with timestamps:

1. JSON Report (suspicious-repos-YYYY-MM-DD.json)

   • Structured data grouped by issue type:

     {
       "node_modules": ["https://github.com/user/repo1", ...],
       "build_dirs": ["https://github.com/user/repo2", ...],
       "dotenv_files": ["https://github.com/user/repo3", ...],
       "extremely_large_repos": ["https://github.com/user/repo4", ...],
       "single_commit_repos": ["https://github.com/user/repo5", ...],
       "metadata": {
         "generated_at": "2025-11-12T...",
         "total_repos": 260,
         "repos_analyzed": 260
       }
     }

   • Note: .env.example files are NOT flagged - only actual .env and .env.local files

2. Markdown Report (report-YYYY-MM-DD.md)

   • Human-readable summary with:
     • Overall statistics
     • Language distribution table
     • Repositories with node_modules committed
     • High JavaScript percentage repositories
     • Repositories with build directories
     • Repositories with .env files
     • Extremely large repositories (>100MB)

Example Output

📊 REPOSITORIES WITH JAVASCRIPT (sorted by JS percentage)
================================================================================

username/suspect-repo (🔒 PRIVATE)
  URL: https://github.com/username/suspect-repo
  Repo Size: 245,832 KB | Language Stats: 234.56 MB
  JavaScript: 98.45%
  All languages:
    - JavaScript: 98.45% (230.91 MB)
    - TypeScript: 1.55% (3.65 MB)

...

⚠️  SUSPECTS (>50% JavaScript):
  - username/suspect-repo (98.45% JS)
    https://github.com/username/suspect-repo

Tips

After identifying the problem repositories:

1. Check if node_modules was committed
2. Remove it with:

   git rm -r node_modules
   git commit -m "Remove committed node_modules"
   git push

3. Ensure .gitignore includes node_modules
4. Consider using .gitattributes to exclude certain files from language statistics

License

ISC