PolicyIQ

PolicyIQ is a distributed, microservice-based ingestion platform for compliance analytics.

Architecture

1. Ingestion Service: Authenticates and accepts events.
2. Queue Service: Durable SQLite-based FIFO queue.
3. Validation Service: Polls queue, deduplicates, and validates.
4. Enrichment Service: Adds sensitivity scores.
5. Compliance Service: Evaluates policy rules.
6. Anomaly Service: Detects behavioral anomalies (Z-score).
7. Salesforce Sink: Pushes governed data to Data Cloud.

Quick Start

1. Build and Run:

   docker-compose up --build -d

2. Send Event:

   curl -X POST http://localhost:8000/events \
     -H "Content-Type: application/json" \
     -H "X-API-Key: secret" \
     -d @sample_event.json

3. Check Queue Size:

   curl -X GET http://localhost:8001/size

4. View Logs:

   docker-compose logs -f

Development

• Shared Library: Located in shared/, copied to all services.
• Configuration: Environment variables (see docker-compose.yml).
• Ports:
  • Ingestion: 8000 (Internal target 8000)
  • Queue: 8001 (Internal target 8000)

Failure Demonstration

1. Stop Downstream Service:

   docker-compose stop compliance-service

2. Expect Queue Growth (Validation Service will retry, then backoff):

   curl -X GET http://localhost:8001/size

3. Trigger Permanent Failure (DLQ): Wait for retries to exhaust. Check DLQ:

   docker-compose exec queue-service sqlite3 /data/queue.db "SELECT COUNT(*) FROM dead_letter_events"

4. Restart Service:

   docker-compose start compliance-service

5. Trigger Replay (Optional):

   curl -X POST http://localhost:8001/replay

Automated Verification

I have provided a script to verify compatibility end-to-end.

1. Ensure Stack is Running:

   docker-compose up -d

2. Run Test Script:

   # Requires python installed locally
   pip install httpx
   python tests/e2e_verify.py

   This script will:

   • Check Ingestion and Queue health.
   • Post a test event.
   • Verify valid Trace ID return.
   • Verify Queue processing drainage.

3. Advanced: Verify Internal Services: To confirm all internal microservices are running and reachable:

   docker-compose exec ingestion-service python -c "import httpx; s=['http://queue-service:8000','http://validation-service:8000','http://enrichment-service:8000','http://compliance-service:8000','http://anomaly-service:8000','http://salesforce-sink:8000']; [print(u, '->', httpx.get(u+'/health').status_code) for u in s]"

   Expected Output: All services return -> 200.