Full disclosure

The issue in question, the fix and the test was to 100% analzyzed and written by Claude Code and Sonnet 4.6. I have verified locally that this change fixes the issue reported in #5721. That said, I have no idea if the fix makes sense. It took a few iterations, but the analysis sounds plausible. OTOH, it might equally well be hallucination. 🙈

Describe the PR

Root cause

TypeOps.projectUpwards() / projectDownwards() use UPWARDS_PROJECTOR /
DOWNWARDS_PROJECTOR, which are JTypeVisitor implementations that recursively
walk a type and substitute captured type variables with their bounds.

For a captured var α whose upper bound is a parameterized type that contains
α again as a wildcard bound (e.g. Container<T extends Container<? extends T>>
produces α with upper bound Container<? extends α>), the projection entered
infinite mutual recursion:

visitTypeVar(α) → follows bound Container<? extends α>
→ visitClass → recurseIfNotDone(? extends α) [wildcard: no cycle guard]
→ visitWildcard(? extends α) → follows bound α
→ visitTypeVar(α) → follows bound Container<? extends α> [cycle!]

RecursionStop.recurseIfNotDone only guarded against JTypeVar type args
(direct type-arg position in visitClass). Wildcards bypassed that guard,
so the wildcard → captured-var → bound → wildcard cycle was undetected.

Two distinct cycle paths require two distinct guards

visitTypeVar for a captured var α can be reached via two paths:

1. Via recurseIfNotDone body (direct type-arg in visitClass):
   recurseIfNotDone adds α to its set and calls body = visitTypeVar(α).
   Detected by contains(α) == true. The cycle guard for DIRECT type args
   is already handled by recurseIfNotDone itself (second encounter of α
   as a direct type arg returns α unchanged → isCvar → replaced with ?).
   However, α's bound may still lead back to α via a WILDCARD (not a
   direct type arg), escaping recurseIfNotDone's guard. This path needs
   its own cycle detection: containsCallStack.

2. Via a fresh call (top level) or via visitWildcard (wildcard-bound path):
   α is NOT yet in recurseIfNotDone's set. This is the original SOE path.
   Detected by contains(α) == false. This path needs freshCallStack.

The two stacks cannot be merged: the Scratch<T extends Scratch> case
requires the recurseIfNotDone body call to proceed (even though an outer
fresh call may have already entered the same var in freshCallStack), because
the legitimate first-level unrolling (Scratch → Scratch>)
depends on the body following the bound before recurseIfNotDone catches the
second direct occurrence. Using a single shared stack would confuse the two
paths and prevent the correct projection.

RecursionStop now carries:

• contains(tvar): read-only check of the recurseIfNotDone set.
• enterFreshVar / leaveFreshVar: for fresh/wildcard-path cycle detection.
• enterContainsVar / leaveContainsVar: for contains-path cycle detection.

Related issues

• Fixes [java] StackOverflowError in 7.17.0 with nested wildcard generics #5721

Ready?

• Added unit tests for fixed bug/feature
• Passing all unit tests
• Complete build ./mvnw clean verify passes (checked automatically by github actions)
• Added (in-code) documentation (if needed)