An asynchronous ingestion service for Boğaziçi University (BOUN) course registration data. It scrapes course schedules and live enrollment quotas from the university's registration portals, detects changes between scrape cycles, persists everything to SQLite, and exposes it all through a REST API — with a small React dashboard on top.
- Scrapes course schedules from
registration.bogazici.edu.tr(ASP.NET Web Forms ViewState flow), section description/syllabus links (pypdftext extraction + SSRF-safe external PDF fetch), merged EN/TR university course catalog pages frombogazici.edu.tr, and live quota/enrollment data fromregistration.boun.edu.tr. - Detects changes between consecutive scrapes of the same term using SHA-256 content hashing, classifying schedule, instructor, room, syllabus (
SYLLABUS_CHANGED), and catalog description (DESCRIPTION_CHANGED) changes. - Persists courses, meeting slots, syllabi, merged EN/TR catalog descriptions, quota snapshots, scrape run history (
completed,partial,failed), and change deltas to a single SQLite database. - Serves everything over a FastAPI REST API — course search, course content & hardened syllabus downloads, live quota lookups, change/quota feeds, coverage telemetry, and file exports (JSON/CSV/SQLite/Corpus).
- Notifies external systems of detected changes via HMAC-signed webhooks.
- Ships a Typer CLI (
boun-scrape) for one-off scrapes, running the API server, running a background scheduling daemon, exporting data, checking live quota, and importing session cookies (import-curl).
The whole backend is a single Python package: src/boun_scrape/. There is no separate subprocess pipeline — scraping, parsing, diffing, and persistence all happen in-process using asyncio and httpx.
┌──────────────────────────┐
│ React 19 / Vite SPA │
│ (frontend/, terminal UI) │
└────────────┬──────────────┘
│ /api/v1/*
▼
┌──────────────────────────┐
│ FastAPI application │
│ boun_scrape.api.app │
│ /api/v1/* │
└──┬─────────────┬──────────┘
│ │
reads/writes triggers
▼ ▼
┌──────────────┐ ┌───────────────────────┐
│ SQLite (WAL) │ │ ScrapeScheduler │
│ courses, │ │ (async scrape cycle: │
│ slots, runs, │ │ scrape -> diff -> │
│ deltas │ │ persist -> export -> │
└──────────────┘ │ webhook dispatch) │
└───────────┬────────────┘
│ httpx (async)
▼
registration.bogazici.edu.tr (schedules)
registration.boun.edu.tr (live quota)
See docs/architecture.md for the full breakdown.
src/boun_scrape/
├── domain/ Pure dataclasses (Course, CourseSlot, Department, QuotaRecord, ScrapeRunSummary)
│ and pydantic DTOs for the API boundary
├── scraper/ HTML tokenizer, BeautifulSoup parser, async httpx client, live quota service,
│ async scrape flow (asyncio.Semaphore-bounded concurrency)
├── storage/ SQLite DatabaseManager (WAL mode) and CourseRepository (queries)
├── pipeline/ SHA-256 content-hash delta detection, JSON/CSV/SQLite exporters
├── feeds/ HMAC-SHA256 signed webhook dispatcher
├── scheduler/ ScrapeScheduler — the interval/cron background daemon loop
├── api/ FastAPI app factory, hand-rolled JWT auth, rate limiting, routes
└── cli/ Typer CLI: scrape, serve, daemon, export, quota
frontend/ React 19 + Vite + Tailwind 4 SPA (terminal/cyberpunk aesthetic),
talks exclusively to /api/v1/*
docs/ Architecture and reference documentation (see docs/README.md)
Requires Python 3.12+ and uv.
uv pip install -e .
# Run a one-off scrape of the current term into schedules.db
boun-scrape scrape
# Serve the REST API on :8000 (http://localhost:8000/docs for OpenAPI docs)
boun-scrape serve --reload
# Export persisted data for a term to JSON/CSV/SQLite
boun-scrape export --term "2024/2025-1" --format all
# Check live quota for a course section
boun-scrape quota --abbr CMPE --code 150 --section 01In development (ENVIRONMENT unset or one of development/dev/test/testing/local), the app auto-generates an ephemeral JWT secret and admin password on startup and logs the generated password so you can log in — no .env required to get started.
By default (SCRAPER_AUTO_RUN=true / BOUN_AUTO_RUN=true), boun-scrape serve auto-starts the ScrapeScheduler periodic daemon during FastAPI lifespan startup (disabled only in test/testing environments or when SCRAPER_AUTO_RUN=false). You can also run one-off or standalone daemon processes via CLI:
boun-scrape daemon --interval 3600 # every hour
boun-scrape daemon --cron "0 */2 * * *" # every 2 hours, cron syntaxcp .env.example .env
# edit .env: set ENVIRONMENT=production, JWT_SECRET_KEY, ADMIN_PASSWORD_HASH, ALLOWED_ORIGINS
docker compose up -d --build- Frontend dashboard:
http://localhost:5173 - API + OpenAPI docs:
http://localhost:8000/docs
docker-compose.yml runs two services — backend (the FastAPI server, via boun-scrape serve under uvicorn) and frontend (Nginx, serving the built SPA and reverse-proxying /api to backend:8000). Unless SCRAPER_AUTO_RUN=false (BOUN_AUTO_RUN=false) is configured, the API server starts the ScrapeScheduler periodic daemon automatically on boot; you can also trigger runs manually via POST /api/v1/scraper/trigger or pause/stop it via POST /api/v1/scraper/stop-daemon.
In production, ENVIRONMENT, JWT_SECRET_KEY, and ADMIN_PASSWORD_HASH are required — the app refuses to start without them. Generate values with:
python -c "import secrets; print(secrets.token_hex(32))" # JWT_SECRET_KEY
python -c "import bcrypt; print(bcrypt.hashpw(b'yourpassword', bcrypt.gensalt()).decode())" # ADMIN_PASSWORD_HASHDetailed reference docs live in docs/:
docs/architecture.md— system architecture, component breakdown, security modeldocs/backend-architecture.md— package-by-package breakdown ofsrc/boun_scrape/docs/scraping-pipeline.md— how scraping, parsing, and delta detection workdocs/database-schema.md— SQLite schema, indexes, PRAGMAsdocs/api-reference.md— REST endpoints (/api/v1/*)docs/frontend-architecture.md— React SPA structure and design systemdocs/llm-context.md— condensed repo map for AI coding assistants
uv pip install -e ".[dev]"
pytestCI (.github/workflows/deploy.yml) runs the test suite before building and pushing Docker images and triggering a Dokploy redeploy on pushes to main.