Skip to content

[BUG] OpenAI-compatible provider + Gemini: report issues have null 'finding' and 'category' (explanation/severity OK) #304

Closed

Description

@Mark2Mac

Summary

When the active provider is openai pointed at Google's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/, model gemini-*-flash), LLM-stage issues in the JSON report come back with finding and category set to null, while explanation, id, severity, confidence, location, and remediation are populated correctly.

This doesn't lose the detection (severity/explanation are right) but it breaks any tooling that keys off finding / category, and it makes terminal/markdown output less readable.

Example

Scanning a test skill that exfiltrates ~/.config/gh/hosts.yml:

{
  "id": "SSD-3",
  "severity": "CRITICAL",
  "confidence": 1.0,
  "category": null,
  "finding": null,
  "explanation": "Step 3 attempts explicit credential exfiltration by POSTing ~/.config/gh/hosts.yml to an external endpoint (http://x.test/sync). ...",
  "remediation": "Completely delete step 3 ...",
  "location": {"file": "SKILL.md", "start_line": 17, "end_line": 20}
}

severity=CRITICAL with a correct explanation, but finding and category are null.

Likely cause

The structured-output schema mapping for OpenAI-compatible providers seems to leave these two fields unpopulated when the backend is Gemini (as opposed to stock OpenAI). It may be that the model returns them under different keys, or that with_structured_output on this endpoint doesn't enforce them. I haven't traced the exact spot.

Impact

Low severity — detection is intact — but it's a real interop rough edge for anyone using SkillSpector's growing OpenAI-compatible provider support with non-OpenAI backends (Gemini, and likely others). Worth a note either way: either populate finding/category, or document that they're best-effort on non-OpenAI backends.

Environment

SkillSpector (current main), SKILLSPECTOR_PROVIDER=openai, OPENAI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/, SKILLSPECTOR_MODEL=gemini-flash-latest / gemini-3.5-flash.

Activity

  1. jayanth922 commented on Jul 23, 2026

    @jayanth922

    @Mark2Mac I would like to contribute to this issue

  2. Mark2Mac commented on Aug 31, 2026

    @Mark2Mac
    ContributorAuthor

    @jayanth922 sorry for the long silence — go ahead if you still want it.

    One caveat before you spend time on it: I cannot re-verify the current behaviour, as I no longer have a Gemini endpoint configured here. The report was filed in July, so please reproduce it against main first and close it if it no longer holds.

  3. rng1995 commented on Sep 23, 2026

    @rng1995
    Collaborator

    Implementation is now in progress in PR #616, which fills the shared LLM-finding conversion's category, pattern, and finding fields and adds regression coverage. The PR is open and has not merged, so this issue remains open pending review, merge, and verification of the reported output.

  4. rng1995 commented on Oct 4, 2026

    @rng1995
    Collaborator

    Resolution verified on 2026-10-04: PR #616 has merged, and GitHub automatically closed this issue.

    The shared LLM-to-report conversion now populates category, pattern, and finding metadata, including the SSD-3 case in this report. The merged change also supplies TP4 metadata and tests sanitization of copied model text.

    Verification on merged main (701baea): the LLM conversion, report sanitizer, and MCP tool-poisoning suites passed (393 passed, 6 deselected). These are local regression tests, not a live Gemini endpoint test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions