Skip to content

[BUG]: PII scan reports Chrome User-Agent versions as public IP addresses #167

Closed

Description

@JayOfTheKeyboard

What happened?

The ip_addresses PII check reports a Chromium User-Agent version as a non-RFC1918 IP address. Since Chrome 101, Chromium's reduced User-Agent writes every version as <major>.0.0.0, so any skill that documents or sets a browser User-Agent gets a [PII-MEDIUM] Non-RFC1918 IP address finding for Chrome/140.0.0.0, HeadlessChrome/151.0.0.0, Edg/140.0.0.0 or OPR/122.0.0.0.

Expected: a product token followed by a version, as in an HTTP User-Agent header (product "/" product-version), is not reported. _is_version_literal already exempts version labels (version = "4.4.0.1") and package artifacts (pkg-4.5.0.0.whl), but it has no rule for this form.

I found it by running skillevaluator tier1 --no-llm over my own skills. A browser-automation skill got one finding from a docstring, HeadlessChrome/151.0.0.0 -> Chrome/151.0.0.0.

Reproduction steps

ua-skill/SKILL.md:

---
name: ua-skill
description: Fetch a page with a desktop Chrome user agent. Use when a site rejects the default client.
license: Apache-2.0
metadata:
  author: Jane Doe <[email protected]>
---

# UA skill

Send this header:

```
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36
```

Edge reports `Edg/140.0.0.0` too.
$ skillevaluator pii-scan ua-skill
...
  1. [PII-MEDIUM] Non-RFC1918 IP address: 140.0.0.0 — 2 occurrences (SKILL.md lines 14, 17)
     File:    SKILL.md:14
     Check:   ip_addresses
     Content: User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHT...
     Fix:     Use private IP (10.x.x.x, 192.168.x.x) or placeholder like 0.0.0.0

The finding is non-blocking, but it adds noise to every report for a skill that works with browsers, and its suggested fix (replace the value with a private IP) would make a real User-Agent string wrong.

SkillEvaluator version or commit

main at a2636b2

Environment

Fedora 44, Python 3.12.14 through uv sync --all-extras from a clone. No provider configured.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions