Skip to content

Commit 1feb2c8

Browse files
authored
Fix Pipenv dependency extraction and locked version checks (#765)
* fix(supply-chain): parse Pipenv dependency manifests Parse Pipfile TOML and Pipfile.lock JSON dependencies, reuse exact lock versions, and report incomplete dependency parsing. Add regression and CLI coverage. Signed-off-by: YMuskrat <[email protected]> * fix(supply-chain): retain findings after Pipenv parser errors Catch ValueError from Pipenv JSON/TOML parsing, including oversized integer conversion failures, and record a nonfatal dependency_parse_error for the offending file. Add four regressions covering 5,000-digit metadata integers in both formats and CLI scans that retain unrelated SC2 and SC4 findings. Restore the interpreter digit limit after each case. Signed-off-by: YMuskrat <[email protected]> * fix(supply-chain): keep locked versions with their project Scope Python lock lookups to the manifest directory and keep Pipenv locks separate from other manifest types. Leave conflicting locked versions unresolved while still checking each locked entry. Addresses the project association review on #765. Signed-off-by: YMuskrat <[email protected]> * test(supply-chain): isolate dependency versions across projects Add 23 cases covering the exact docs Pipenv/root requirements layout, independent Pipenv projects, both component orders, unrelated same-directory manifests, conflicting categories, uv/Poetry scope and aggregate limits. Validation: 662 focused tests, 10,027 unit tests and 124 integration tests passed, plus lint and formatting. All 23 new cases passed against a freshly built wheel using deterministic OSV responses. Signed-off-by: YMuskrat <[email protected]> * fix(supply-chain): continue after file-local lockfile limitations Keep collecting independent project lockfiles after Pipenv parse errors, unsupported sources and file-local limits. Stop only for the aggregate file, runtime or package budget. Signed-off-by: YMuskrat <[email protected]> * test(supply-chain): retain resolution after partial Pipenv locks Cover unsupported Git dependencies, malformed lockfiles, oversized entries and file-local entry limits in both file orders. Verify unrelated manifests still receive their locked versions and existing parse limitations remain reported. Validation: 672 focused tests passed; all ten new cases passed against source and a freshly installed wheel. Unit tests passed after retrying 40 environment failures with the missing SOCKS dependency installed; 124 integration tests passed. Signed-off-by: YMuskrat <[email protected]> --------- Signed-off-by: YMuskrat <[email protected]>
1 parent 2b85712 commit 1feb2c8

4 files changed

Lines changed: 943 additions & 16 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -979,6 +979,10 @@ SC4 uses the [OSV.dev](https://osv.dev) API to check dependencies against the fu
979979
- **Batch queries** — all dependencies are checked in a single HTTP call.
980980
- **Automatic fallback** — if OSV.dev is unreachable (air-gapped/offline), a small built-in fallback list is used.
981981
- **Caching** — results are cached in-memory for 1 hour to avoid redundant API calls during a session.
982+
- **Pipenv** — reads `Pipfile` package tables and the direct/transitive packages in
983+
`Pipfile.lock`, including development and custom categories. Exact lock versions
984+
can resolve unpinned manifest dependencies. Malformed entries and unsupported
985+
VCS/local sources are reported as incomplete dependency checks.
982986

983987
The tool requires outbound HTTPS access to `api.osv.dev` for live vulnerability data. When that is not available, findings are limited to the static fallback list.
984988

‎src/skillspector/inspection_ledger.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ class LedgerReason(StrEnum):
102102
REFERENCE_MISSING = "reference_missing"
103103
REFERENCE_UNRESOLVED = "reference_unresolved"
104104
MANIFEST_PARSE_ERROR = "manifest_parse_error"
105+
DEPENDENCY_PARSE_ERROR = "dependency_parse_error"
105106
MANIFEST_PARSE_LIMIT = "manifest_parse_limit"
106107
ARTIFACT_COUNT_LIMIT = "artifact_count_limit"
107108
TRAVERSAL_DEPTH_LIMIT = "traversal_depth_limit"
@@ -233,6 +234,9 @@ def only_missing_reference_exceptions(exceptions: object) -> bool:
233234
LedgerReason.MANIFEST_PARSE_ERROR: (
234235
"Manifest frontmatter is malformed or uses an unsupported value shape."
235236
),
237+
LedgerReason.DEPENDENCY_PARSE_ERROR: (
238+
"Dependency metadata could not be parsed into supported registry coordinates."
239+
),
236240
LedgerReason.MANIFEST_PARSE_LIMIT: (
237241
"Manifest frontmatter could not be completely examined within its resource limits."
238242
),

‎src/skillspector/nodes/analyzers/static_patterns_supply_chain.py‎

Lines changed: 252 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1619,6 +1619,207 @@ def extend_specs(values: object) -> None:
16191619
r"(?ms)^\s*\[\[package\]\]\s*$.*?(?=^\s*\[\[package\]\]\s*$|\Z)"
16201620
)
16211621

1622+
_PIPFILE_METADATA_SECTIONS = frozenset({"source", "requires", "scripts", "pipenv"})
1623+
_PIPFILE_KEY_RE = re.compile(r"""^[ \t]*("(?:\\.|[^"\\])*"|'[^']*'|[A-Za-z0-9_-]+)[ \t]*=""")
1624+
_JSON_TOKEN_RE = re.compile(r'"(?:\\.|[^"\\])*"|[{}\[\]]')
1625+
_JSON_KEY_SUFFIX_RE = re.compile(r"\s*:")
1626+
_TOML_STRING_TOKEN_RE = re.compile(r'''"""|\x27{3}|"(?:\\.|[^"\\])*"|'[^']*'|\#.*''')
1627+
_TOML_MULTILINE_END_RE = {
1628+
'"""': re.compile(r'(?<!\\)(?:\\\\)*"""'),
1629+
"'''": re.compile("'''"),
1630+
}
1631+
1632+
1633+
def _pipenv_package_lines(
1634+
content: str, requested: set[tuple[str, str]], *, is_lockfile: bool
1635+
) -> dict[tuple[str, str], int]:
1636+
"""Index package keys once, retaining separate locations for each category."""
1637+
locations: dict[tuple[str, str], int] = {}
1638+
if not requested:
1639+
return locations
1640+
if is_lockfile:
1641+
depth = 0
1642+
section = ""
1643+
line = 1
1644+
position = 0
1645+
for token in _JSON_TOKEN_RE.finditer(content):
1646+
line += content.count("\n", position, token.start())
1647+
position = token.start()
1648+
value = token.group(0)
1649+
if value in {"{", "["}:
1650+
depth += 1
1651+
elif value in {"}", "]"}:
1652+
depth -= 1
1653+
elif depth in {1, 2} and _JSON_KEY_SUFFIX_RE.match(content, token.end()):
1654+
key = json.loads(value)
1655+
if depth == 1:
1656+
section = key
1657+
elif (section, key) in requested:
1658+
locations[(section, key)] = line
1659+
return locations
1660+
1661+
section = ""
1662+
multiline_quote: str | None = None
1663+
for line_number, line in enumerate(io.StringIO(content), 1):
1664+
was_multiline = multiline_quote is not None
1665+
position = 0
1666+
if multiline_quote is not None:
1667+
closing = _TOML_MULTILINE_END_RE[multiline_quote].search(line)
1668+
if closing is None:
1669+
continue
1670+
multiline_quote = None
1671+
position = closing.end()
1672+
while token := _TOML_STRING_TOKEN_RE.search(line, position):
1673+
quote = token.group(0)
1674+
if quote.startswith("#"):
1675+
break
1676+
position = token.end()
1677+
if quote in _TOML_MULTILINE_END_RE:
1678+
closing = _TOML_MULTILINE_END_RE[quote].search(line, position)
1679+
if closing is None:
1680+
multiline_quote = quote
1681+
break
1682+
position = closing.end()
1683+
if was_multiline:
1684+
continue
1685+
stripped = line.strip()
1686+
if stripped.startswith("["):
1687+
try:
1688+
header = tomllib.loads(stripped)
1689+
except tomllib.TOMLDecodeError:
1690+
section = ""
1691+
continue
1692+
section = next(iter(header), "")
1693+
children = header.get(section)
1694+
if isinstance(children, dict):
1695+
for name in children:
1696+
if (section, name) in requested:
1697+
locations[(section, name)] = line_number
1698+
continue
1699+
key_match = _PIPFILE_KEY_RE.match(line)
1700+
if key_match is None:
1701+
continue
1702+
try:
1703+
key = next(iter(tomllib.loads(f"{key_match.group(1)} = 0")))
1704+
except tomllib.TOMLDecodeError:
1705+
continue
1706+
if (section, key) in requested:
1707+
locations[(section, key)] = line_number
1708+
return locations
1709+
1710+
1711+
def _extract_packages_from_pipenv(
1712+
content: str,
1713+
*,
1714+
is_lockfile: bool,
1715+
limit: int | None = None,
1716+
) -> tuple[list[tuple[str, str | None, int]], list[OsvQueryLimitation]]:
1717+
"""Extract bounded Pipenv registry dependencies without executing project code.
1718+
1719+
Pipfile uses TOML package tables; Pipfile.lock uses JSON categories containing
1720+
all resolved direct and transitive packages. Metadata is never a dependency.
1721+
Non-registry sources and malformed entries remain explicit coverage gaps.
1722+
"""
1723+
entry_limit = MAX_DEPENDENCY_PACKAGES_PER_FILE if limit is None else max(0, limit)
1724+
if entry_limit == 0:
1725+
return [], []
1726+
try:
1727+
data = json.loads(content) if is_lockfile else tomllib.loads(content)
1728+
except (ValueError, RecursionError) as error:
1729+
return [], [
1730+
OsvQueryLimitation(
1731+
reason=LedgerReason.DEPENDENCY_PARSE_ERROR,
1732+
error_class=type(error).__name__,
1733+
)
1734+
]
1735+
if not isinstance(data, dict):
1736+
return [], [OsvQueryLimitation(reason=LedgerReason.DEPENDENCY_PARSE_ERROR)]
1737+
1738+
results: list[tuple[str, str | None, int]] = []
1739+
location_keys: list[tuple[str, str]] = []
1740+
limitations: list[OsvQueryLimitation] = []
1741+
limitation_keys: set[tuple[LedgerReason, str | None]] = set()
1742+
1743+
def note(limitation: OsvQueryLimitation) -> None:
1744+
key = (limitation.reason, limitation.error_class)
1745+
if key not in limitation_keys:
1746+
limitations.append(limitation)
1747+
limitation_keys.add(key)
1748+
1749+
seen = 0
1750+
for section, entries in data.items():
1751+
if section in ({"_meta"} if is_lockfile else _PIPFILE_METADATA_SECTIONS):
1752+
continue
1753+
if not isinstance(entries, dict):
1754+
note(OsvQueryLimitation(reason=LedgerReason.DEPENDENCY_PARSE_ERROR))
1755+
continue
1756+
for name, entry in entries.items():
1757+
seen += 1
1758+
if seen > entry_limit:
1759+
note(
1760+
OsvQueryLimitation(
1761+
reason=LedgerReason.OUTPUT_LIMIT,
1762+
observed_records=seen,
1763+
limit_records=entry_limit,
1764+
)
1765+
)
1766+
break
1767+
if len(name) > MAX_DEPENDENCY_NAME_CHARS:
1768+
note(
1769+
OsvQueryLimitation(
1770+
reason=LedgerReason.SIZE_LIMIT,
1771+
observed_characters=len(name),
1772+
limit_characters=MAX_DEPENDENCY_NAME_CHARS,
1773+
)
1774+
)
1775+
continue
1776+
if isinstance(entry, dict):
1777+
if any(source in entry for source in ("git", "path", "file", "hg", "svn", "bzr")):
1778+
note(
1779+
OsvQueryLimitation(
1780+
reason=LedgerReason.DEPENDENCY_PARSE_ERROR,
1781+
error_class="UnsupportedDependencySource",
1782+
)
1783+
)
1784+
continue
1785+
spec = entry.get("version", "" if is_lockfile else "*")
1786+
else:
1787+
spec = entry if not is_lockfile else None
1788+
if not isinstance(spec, str):
1789+
requirement = None
1790+
elif len(spec) > MAX_DEPENDENCY_SPEC_CHARS:
1791+
note(
1792+
OsvQueryLimitation(
1793+
reason=LedgerReason.SIZE_LIMIT,
1794+
observed_characters=len(spec),
1795+
limit_characters=MAX_DEPENDENCY_SPEC_CHARS,
1796+
)
1797+
)
1798+
continue
1799+
else:
1800+
requirement = _extract_python_requirement(
1801+
name if spec.strip() == "*" else f"{name}{spec}"
1802+
)
1803+
if requirement is None or requirement[0] != name:
1804+
note(OsvQueryLimitation(reason=LedgerReason.DEPENDENCY_PARSE_ERROR))
1805+
continue
1806+
if is_lockfile and requirement[1] is None:
1807+
note(OsvQueryLimitation(reason=LedgerReason.DEPENDENCY_PARSE_ERROR))
1808+
continue
1809+
results.append((*requirement, 1))
1810+
location_keys.append((section, name))
1811+
if seen > entry_limit:
1812+
break
1813+
locations = _pipenv_package_lines(content, set(location_keys), is_lockfile=is_lockfile)
1814+
return [
1815+
(name, version, locations.get(key, 1))
1816+
for (name, version, _line), key in zip(results, location_keys, strict=True)
1817+
], limitations
1818+
1819+
1820+
def _is_pipenv_lockfile(file_path: str) -> bool:
1821+
return Path(file_path).name.lower() == "pipfile.lock"
1822+
16221823

16231824
def _normalize_package_name(name: str) -> str:
16241825
"""Normalize package names the same way OSV/fallback coverage does."""
@@ -1627,7 +1828,7 @@ def _normalize_package_name(name: str) -> str:
16271828

16281829
def _is_python_lockfile(file_path: str) -> bool:
16291830
lower_path = file_path.lower()
1630-
return "uv.lock" in lower_path or "poetry.lock" in lower_path
1831+
return "uv.lock" in lower_path or "poetry.lock" in lower_path or _is_pipenv_lockfile(file_path)
16311832

16321833

16331834
def _normalize_npm_package_name(name: str) -> str:
@@ -1789,19 +1990,26 @@ def _apply_locked_versions(
17891990
return resolved
17901991

17911992

1993+
def _python_lock_scope(file_path: str) -> tuple[str, bool]:
1994+
"""Associate Pipenv locks only with sibling Pipfiles, not other manifests."""
1995+
path = Path(file_path)
1996+
return str(path.parent), path.name.lower() in {"pipfile", "pipfile.lock"}
1997+
1998+
17921999
def _collect_locked_versions(
17932000
file_cache: dict[str, str],
17942001
components: list[str],
17952002
*,
2003+
file_path: str = "requirements.txt",
17962004
limit: int = MAX_DEPENDENCY_PACKAGES_PER_SCAN,
17972005
) -> dict[str, str]:
1798-
"""Build package -> exact version map from Python lockfiles in the project."""
2006+
"""Build the exact version map associated with a Python manifest."""
17992007
locked_versions, _limitations = _collect_locked_versions_detailed(
18002008
file_cache,
18012009
components,
18022010
limit=limit,
18032011
)
1804-
return locked_versions
2012+
return locked_versions.get(_python_lock_scope(file_path), {})
18052013

18062014

18072015
def _collect_locked_versions_detailed(
@@ -1811,9 +2019,10 @@ def _collect_locked_versions_detailed(
18112019
limit: int = MAX_DEPENDENCY_PACKAGES_PER_SCAN,
18122020
max_files: int | None = None,
18132021
timeout_seconds: float | None = None,
1814-
) -> tuple[dict[str, str], list[tuple[str, OsvQueryLimitation]]]:
2022+
) -> tuple[dict[tuple[str, bool], dict[str, str]], list[tuple[str, OsvQueryLimitation]]]:
18152023
"""Build a bounded lock map and identify any manifest whose tail was omitted."""
1816-
locked_versions: dict[str, str] = {}
2024+
locked_versions: dict[tuple[str, bool], dict[str, str]] = {}
2025+
ambiguous_versions: set[tuple[tuple[str, bool], str]] = set()
18172026
limitations: list[tuple[str, OsvQueryLimitation]] = []
18182027
packages_seen = 0
18192028
lockfiles_seen = 0
@@ -1855,7 +2064,7 @@ def _collect_locked_versions_detailed(
18552064
)
18562065
break
18572066
content = file_cache.get(path)
1858-
if not content:
2067+
if content is None or (not content and not _is_pipenv_lockfile(path)):
18592068
continue
18602069
remaining = max(0, limit - packages_seen)
18612070
if remaining <= 0:
@@ -1870,11 +2079,17 @@ def _collect_locked_versions_detailed(
18702079
)
18712080
)
18722081
break
1873-
packages = _extract_packages_from_toml_lock(
1874-
content,
1875-
limit=remaining + 1,
1876-
)
1877-
if len(packages) > remaining:
2082+
if _is_pipenv_lockfile(path):
2083+
packages, parse_limitations = _extract_packages_from_pipenv(
2084+
content,
2085+
is_lockfile=True,
2086+
limit=remaining + 1,
2087+
)
2088+
limitations.extend((path, limitation) for limitation in parse_limitations)
2089+
else:
2090+
packages = _extract_packages_from_toml_lock(content, limit=remaining + 1)
2091+
package_limit_exceeded = len(packages) > remaining
2092+
if package_limit_exceeded:
18782093
limitations.append(
18792094
(
18802095
path,
@@ -1887,10 +2102,24 @@ def _collect_locked_versions_detailed(
18872102
)
18882103
packages = packages[:remaining]
18892104
packages_seen += len(packages)
2105+
scope = _python_lock_scope(path)
18902106
for name, version, _line_num in packages:
18912107
if version:
1892-
locked_versions[_normalize_package_name(name)] = version
1893-
if limitations:
2108+
project_versions = locked_versions.setdefault(scope, {})
2109+
normalized_name = _normalize_package_name(name)
2110+
key = scope, normalized_name
2111+
if key in ambiguous_versions:
2112+
continue
2113+
previous_version = project_versions.get(normalized_name)
2114+
if previous_version is not None and previous_version != version:
2115+
# Different categories or lockfiles can pin different versions.
2116+
# Keep scanning each pin, but do not guess for an unpinned manifest.
2117+
project_versions.pop(normalized_name)
2118+
ambiguous_versions.add(key)
2119+
else:
2120+
project_versions[normalized_name] = version
2121+
# File-local parse gaps must not prevent resolution in other projects.
2122+
if package_limit_exceeded:
18942123
break
18952124
return locked_versions, limitations
18962125

@@ -2931,7 +3160,14 @@ def _analyze_dependencies_detailed(
29313160
extraction_limit = package_limit + 1
29323161

29333162
if is_python_dep:
2934-
if "pyproject.toml" in lower_path:
3163+
if Path(file_path).name.lower() in {"pipfile", "pipfile.lock"}:
3164+
packages, parse_limitations = _extract_packages_from_pipenv(
3165+
content,
3166+
is_lockfile=_is_pipenv_lockfile(file_path),
3167+
limit=extraction_limit,
3168+
)
3169+
limitations.extend(parse_limitations)
3170+
elif "pyproject.toml" in lower_path:
29353171
packages = _extract_packages_from_pyproject(content, limit=extraction_limit)
29363172
elif is_lockfile:
29373173
packages = _extract_packages_from_toml_lock(content, limit=extraction_limit)
@@ -3960,12 +4196,12 @@ def dependency_remaining_seconds() -> float:
39604196
)
39614197
break
39624198
content = file_cache.get(path)
3963-
if not content:
4199+
if content is None:
39644200
continue
39654201
dep_findings, dependency_limitations, packages_seen = _analyze_dependencies_detailed(
39664202
content,
39674203
path,
3968-
locked_versions,
4204+
locked_versions.get(_python_lock_scope(path)),
39694205
npm_locked_versions,
39704206
max_packages=min(MAX_DEPENDENCY_PACKAGES_PER_FILE, remaining_packages),
39714207
max_findings=min(MAX_DEPENDENCY_FINDINGS_PER_FILE, remaining_dependency_findings),

0 commit comments

Comments
 (0)