Fixes #319

Describe the changes you have made in this PR -

Adds first-class Splunk integration to OpenSRE. When a Splunk instance is configured (via env vars, the local credential store, or the onboarding wizard), the agent automatically detects it, builds an SPL query from the alert payload, and runs query_splunk_logs as part of the investigation. Log evidence from Splunk flows into the root-cause diagnosis node alongside all other sources.

Files changed:

• app/integrations/models.py — Added SplunkIntegrationConfig Pydantic model and splunk field on EffectiveIntegrations
• app/integrations/catalog.py — Wired Splunk into _SERVICE_KEY_MAP, _classify_service_instance(), and load_env_integrations() (reads SPLUNK_URL, SPLUNK_TOKEN, SPLUNK_INDEX, SPLUNK_VERIFY_SSL)
• app/integrations/verify.py — Added _verify_splunk() and wired it into verify_integrations()
• app/services/splunk/client.py — New SplunkClient (uses the /services/search/jobs/export streaming endpoint; no polling) and build_splunk_spl_query() (deterministic SPL builder — no LLM involvement)
• app/services/splunk/__init__.py — Package init re-exporting public surface
• app/tools/SplunkSearchTool/__init__.py — SplunkSearchTool (BaseTool subclass): is_available() checks connection_verified, extract_params() pulls from sources["splunk"], run() separates error logs and compacts results
• app/tools/SplunkSearchTool/_client.py — Client factory for the tool layer
• app/nodes/plan_actions/detect_sources.py — Splunk source detection block: reads resolved integration, extracts SPL from alert annotations (priority cascade: verbatim query → error_message → alert_name → fallback), sets sources["splunk"]
• app/nodes/plan_actions/build_prompt.py — Added "Splunk Available" hint block so the LLM planner knows the tool is available
• app/cli/wizard/flow.py — _configure_splunk() prompts user, validates live, saves to credential store
• app/cli/wizard/integration_health.py — validate_splunk_integration() calls /services/server/info
• app/cli/alert_templates.py — Added "splunk" template for opensre investigate --template splunk
• tests/tools/test_splunk_search_tool.py — Contract tests + behavior tests (happy path, unavailable paths, error log separation, truncation note)
• tests/integrations/test_splunk.py — Config model tests, build_splunk_spl_query tests, SplunkClient.validate_access tests, catalog classification tests, env var loader tests
• tests/tools/conftest.py — Added "splunk" entry to mock_agent_state()
• tests/e2e/rca/splunk_errors.json — E2E alert fixture for the payments-service NullPointerException scenario

Screenshots of the UI changes (If any) -

Splunk working example :
https://youtu.be/Dy-JIaR0Lec

Code Understanding and AI Usage

Did you use AI assistance (ChatGPT, Claude, Copilot, etc.) to write any part of this code?

• No, I wrote all the code myself
• Yes, I used AI assistance (continue below)

If you used AI assistance:

• I have reviewed every single line of the AI-generated code
• I can explain the purpose and logic of each function/component I added
• I have tested edge cases and understand how the code handles them
• I have modified the AI output to follow this project's coding standards and conventions

Explain your implementation approach:

Problem: OpenSRE had no way to query Splunk logs during an investigation, leaving a gap for teams that use Splunk as their primary log platform.

Approach: Followed the existing integration pattern (Coralogix/BetterStack as reference). The key design decision was to keep SPL query generation fully deterministic — build_splunk_spl_query() assembles the query from alert fields in a priority cascade (verbatim annotation query → error_message → alert_name → fallback). The LLM only decides whether to call the tool; it never writes SPL. This makes investigations reproducible and auditable.

Used the Splunk /services/search/jobs/export streaming endpoint instead of the two-step create-job/poll-results flow — this is simpler, has no polling loop, and is ideal for the time-bounded RCA use case.

Alternatives considered: Using the HEC (HTTP Event Collector) endpoint — rejected because HEC is write-only. Using the standard search job API with polling — rejected in favor of the simpler export endpoint which returns results in one request.

Key components:

• SplunkClient.search_logs() — POSTs to the export endpoint, parses the NDJSON stream, normalizes rows to a common shape
• SplunkClient.validate_access() — hits /services/server/info to confirm credentials during onboarding
• build_splunk_spl_query() — deterministic SPL builder with a priority cascade; always appends | head N to prevent runaway queries
• SplunkSearchTool.run() — calls the client, separates error logs by keyword match, applies compact_logs() before returning evidence

Edge cases handled:

• Missing base_url or token → tool returns available: False immediately, no HTTP call made
• Splunk REST API error (4xx/5xx) → caught, returned as error field in result, investigation continues with other sources
• Raw SPL supplied in alert annotations → used verbatim; | head N appended only if missing, preventing runaway queries
• verify_ssl=False supported for self-signed corporate Splunk instances (common in enterprise)
• NDJSON export stream may contain non-result lines (metadata rows) → parsed defensively, only result-type rows kept

Checklist before requesting a review

• I have added proper PR title and linked to the issue
• I have performed a self-review of my code
• I can explain the purpose of every function, class, and logic block I added
• I understand why my changes work and have tested them thoroughly
• I have considered potential edge cases and how my code handles them
• If it is a core feature, I have added thorough tests
• My code follows the project's style guidelines and conventions

Note: Please check Allow edits from maintainers if you would like us to assist in the PR.