Failing LLM Anthropic Calls

We are evaluating this for AIpkgs and we noticed that the scan, even with LLM integration to claude-opus-4-6 was presenting a large amount of false positives.

For example we were evaluating this skill https://github.com/antfu/skills/tree/main/skills/vue and it had multiple issues raised that seemed like it could be filtered out via the LLM integration or statically.

This was the report that was generated:
report.json

It looks like the "Hidden Instructions" don't consider the file extension or what type of file when classifying comments.
We saw a pattern where the mere mention of "Always use" or "return rules" would trigger the "Behavior Manipulation" or "Direct Prompt Extraction" respectively.
There was also an issue with bundled and minified JS always being flagged as"Context Window Stuffing"

Bellow are some screenshots of findings in their code_snippets highlighted in the context that were flagged in the above skill. These were all markdown files.

I think the idea and work SkillSpector is doing is great, btw!

While looking through the report and the code I also noticed that intent doesn't look to ever get populated. I was checking if it was possible this wasn't analyzed by the LLM to filter the false positive.

SkillSpector/src/skillspector/nodes/meta_analyzer.py

Line 216 in 1a7bf02

intent=None,

Also some warnings I am seeing

WARNING [skillspector.nodes.analyzers.semantic_security_discovery] semantic_security_discovery failed: Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': "For 'integer' type, property 'minimum' is not supported", 'type': 'invalid_request_error', 'param': None}}
WARNING [skillspector.nodes.analyzers.semantic_quality_policy] semantic_quality_policy failed: Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': "For 'integer' type, property 'minimum' is not supported", 'type': 'invalid_request_error', 'param': None}}
WARNING [skillspector.nodes.analyzers.semantic_developer_intent] semantic_developer_intent failed: Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': "For 'integer' type, property 'minimum' is not supported", 'type': 'invalid_request_error', 'param': None}}
WARNING [skillspector.nodes.meta_analyzer] LLM call failed, using fallback: Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': "For 'number' type, properties maximum, minimum are not supported", 'type': 'invalid_request_error', 'param': None}}

I'm using Anthropic claude-opus-4-6

I think it has to do with the Anthropic's OpenAI-compatibility endpoint via ChatOpenAI which has some restrictions

This looks like it worked, removing the ge and le.

SkillSpector/src/skillspector/llm_analyzer_base.py

Line 66 in 1a7bf02

    
           confidence: float = Field(ge=0.0, le=1.0, default=0.5, description="Confidence score")

-    confidence: float = Field(ge=0.0, le=1.0, default=0.5, description="Confidence score")
+    confidence: float = Field(default=0.5, description="Confidence score between 0.0 and 1.0")

SkillSpector/src/skillspector/llm_analyzer_base.py

Line 64 in 1a7bf02

start_line: int = Field(ge=1, description="Starting line number")

-    start_line: int = Field(ge=1, description="Starting line number")
+    start_line: int = Field(description="Starting line number")

Sync OSS release snapshot - Merge pull request #85 from NVIDIA/keshav…

Closing as resolved by #85, which adds native Anthropic provider support and resolves the structured-output tool schema failures caused by unsupported minimum/maximum parameters.

Sync OSS release snapshot - Merge pull request NVIDIA#85 from NVIDIA/…

Navigation Menu

Repository navigation

Failing LLM Anthropic Calls #34

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Failing LLM Anthropic Calls #34

Description

Activity

builder-org443 commented on Jun 13, 2026

builder-org443 commented on Jun 13, 2026

keshprad commented on Jun 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions