ThreatLens is an explainable, context-aware intrusion detection and response prototype. It collects structured application and Nginx access logs, detects suspicious behavior in real time, scores alerts using user context, and presents the results in a live analyst dashboard.
The repository includes a complete local demonstration: a monitored web portal, Nginx reverse proxy, durable log collector, FastAPI detection backend, PostgreSQL, React dashboard, and an optional Ollama/Mistral explanation layer.
- Brute-force login attempts and a successful login after repeated failures
- Port scans in directly ingested network events
- Logins from unusual IP addresses after a user baseline is established
- Excessive HTTP request rates
- Probing of many distinct missing paths
- Repeated HTTP 5xx responses
Detector windows and thresholds can be changed at runtime from the Settings page. Overrides are validated, persisted in PostgreSQL, and applied without restarting the stack.
- Docker Desktop with Docker Compose
- Python 3.11+ only if you want to run the traffic scripts or backend tests
- Node.js 20.19+ only for standalone frontend development
Copy the environment template and start the stack:
Copy-Item .env.example .env
docker compose up --build -dOn macOS or Linux, use cp .env.example .env instead of Copy-Item.
Open:
Open the portal's Security Scenarios tab to generate bounded attack and normal traffic. The account name can be changed inside the portal; passwords of at least six characters succeed in its demo authentication form.
All published ports bind to 127.0.0.1. The default credentials and secrets are
for a local demonstration only. Replace every value in .env before using the
stack on a shared machine.
Check service health and follow logs with:
docker compose ps
docker compose logs -f collector backend monitored-nginxSee RUN_COMMANDS.md for the short operations reference.
The Sentinel portal sends its authentication, background-traffic, and
security-scenario events directly to POST /api/v1/log. Page and asset
requests still pass through monitored Nginx and are collected from its access log.
For command-line scenarios, use the structured-event generator:
python scripts/generate_logs.py --scenario mixed --speed 10 --target-url http://localhost:8002Only run the supplied scenarios against systems you own or are authorized to test. Their default targets are local services.
Detection, scoring, reporting, and the dashboard work without a language model. To enable alert explanations and the assistant with the Compose-managed Ollama service:
docker compose --profile ai up -d ollama
docker compose exec ollama ollama pull mistralRecent alerts can temporarily show no explanation while the background request is running. If Ollama is unavailable or times out, ingestion continues and the assistant returns an availability message.
Open http://localhost:5173/settings, enter the ADMIN_API_KEY from the root
.env, and unlock Detection Limits. The key is held in browser session
storage, so it must be entered again in a new browser session.
Saving settings clears active in-memory detector windows to avoid interpreting old
events under new limits. Restore defaults removes the database override and
reloads the environment defaults from backend/.env.example.
Browser -> monitored Nginx -> React demo portal
| |
v | direct structured events
JSON access log |
| |
v v
collector -> authenticated batch API
|
v
validate and persist event
|
profile -> detect -> score
|
PostgreSQL alerts and metrics
| |
WebSocket updates background AI
|
v
React dashboard
The collector checkpoints only records accepted by the backend, retries temporary failures, handles log rotation, removes query strings, and replaces anonymous visitor IPs with keyed hashes. Nginx access logs cannot reveal request bodies, application failures hidden behind HTTP 200 responses, or network traffic that never reaches the proxy.
Detector windows and chat history are process-local. The current Compose topology runs one backend instance; scaling it horizontally would require shared state such as Redis.
For component responsibilities and the event lifecycle, see Architecture. A more detailed implementation narrative is available in description.md.
The React application includes:
- Overview metrics, severity breakdowns, activity charts, source health, and live alerts
- Filterable Threat Feed and Alerts pages
- Per-user behavioral analytics and risk rankings
- CSV report generation with time, severity, type, and resolution filters
- Runtime detection settings protected by the admin API key
- A shared AI assistant conversation and light/dark themes
REST endpoints are under /api/v1; the WebSocket and health routes are not.
Interactive request and response schemas are available at /docs.
| Method | Endpoint | Purpose |
|---|---|---|
POST |
/api/v1/log |
Ingest one structured event |
POST |
/api/v1/events/batch |
Idempotent collector ingestion (X-ThreatLens-Key) |
POST |
/api/v1/sources/{source_id}/heartbeat |
Update collector health and counters |
GET |
/api/v1/sources |
List monitoring-source status |
GET |
/api/v1/alerts |
Filter and list alerts |
PATCH |
/api/v1/alerts/{id}/resolve |
Resolve an alert |
PATCH |
/api/v1/alerts/{id}/unresolve |
Reopen an alert |
GET |
/api/v1/users/high-risk |
Rank users by rolling risk |
GET |
/api/v1/users/{user_id}/profile |
Read a behavioral profile |
GET |
/api/v1/metrics/summary |
Read dashboard totals |
GET |
/api/v1/metrics/activity |
Read alert activity buckets |
GET |
/api/v1/metrics/threats |
Read filtered threat analytics |
GET |
/api/v1/reports/alerts.csv |
Download a filtered alert report |
POST |
/api/v1/chat |
Ask the alert-grounded assistant |
GET/PUT/DELETE |
/api/v1/admin/detection-settings |
Manage detector limits (X-ThreatLens-Admin-Key) |
POST |
/api/v1/admin/decay-now |
Run risk decay (X-ThreatLens-Admin-Key) |
GET |
/health |
Backend liveness |
WS |
/ws/alerts |
Live alert stream |
The fastest validation path does not require a running PostgreSQL instance because backend tests use SQLite databases created by the test fixtures.
cd backend
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m pytest
python -m ruff check .On macOS or Linux, activate with source .venv/bin/activate.
Frontend validation:
cd frontend
npm install
npm test
npm run lint
npm run buildSentinel portal validation and collector tests:
cd demo_site
python -m pytest -q
cd ..
python -m pytest collector/test_collector.pyFor local development workflows and contribution conventions, see Contributing.
backend/ FastAPI API, detectors, profiling, scoring, AI, and migrations
collector/ Durable Nginx JSON-log tailer and authenticated batch client
demo_site/ Sentinel portal and direct security-scenario generator
frontend/ React 18 and Vite dashboard
nginx/ Reverse-proxy and structured access-log configuration
scripts/ Direct structured-event generators and seed utilities
files/ Architecture, roadmap, changelog, and contribution docs
compose.yaml Complete local demonstration topology
docker compose downThis stops services and preserves data. The following command also permanently deletes the PostgreSQL, Nginx log, collector checkpoint, and Ollama volumes:
docker compose down -vFinal-year major project, Department of CSE, Bangalore Institute of Technology (2025-26).
- Abhinav Kumar Singh - 1BI23CS011
- Anurag Patil - 1BI23CS034
- Harshitha M P - 1BI23CS091
- Shivansh Bhageria - 1BI23CS194
Guide: Dr. Hemavathi P, Professor, Department of CSE.
Academic and educational use only. No separate open-source license is currently included in the repository.