[Task][AI Proxy] Add tenant/model-scoped concurrency admission and bounded queueing #8254

Goal and reviewed baseline

Protect matched LLM traffic with opt-in deterministic per-gateway concurrency admission and finite waiting, reusing existing runtime and configuration mechanisms. Parent: #8055. Source reviewed: 8046953ca.

AI transport pool properties bound connections and pending connection acquisition, but a connection pool is not trusted-tenant/model request admission or tenant fairness. This is a resource-control proposal, not a reproduced production overload or a cluster-wide quota claim.

First delivery

• Derive tenant/subject only from authenticated request context. Use authorized logical-model/target references ([Feature][AI] Add capability-aware virtual models and weighted upstream pools #8046), never caller-controlled tenant headers, credential values or arbitrary endpoint strings.
• Provide explicit per-gateway tenant and model-pool in-flight caps and finite queue size/wait limits. Document deterministic scheduling, tenant fairness and overload outcomes. Keep disabled behavior compatible; validate limits at publication/runtime.
• Define separate ownership for a logical invocation permit and actual target-attempt permits. Failed attempts release their own capacity; backoff holds no failed target permit; successful streaming attempts retain capacity until body completion/error/cancellation, not merely response headers. Avoid partial acquisition/deadlock or compensate deterministically.
• Reuse [Feature] Share an end-to-end request deadline across plugins and upstream attempts #7931/[Task][AI Proxy] Apply a shared request deadline across AI attempts, backoff and fallback #8121 remaining deadline, including time already queued. Expired/cancelled waiters start no work. Configuration replacement/removal and shutdown cannot reset outstanding capacity or strand queues; retired generations drain with bounded state.
• Provide bounded observability and audit reasons through [Task] Add unified AI and MCP invocation auditing #7950/[Task] Define shared shadow and enforce governance decisions #7956. Do not add tenant/request IDs as unrestricted metrics labels or replace Netty's transport pool.

Acceptance

• Concurrent fixtures prove each supported cap, deterministic queue discipline and non-starvation between eligible tenants, including unequal streaming durations.
• Cancellation, timeout, queue removal, overload, config reload/removal and shutdown release permits/references exactly once.
• Retry/fallback retain logical ownership while transferring target ownership correctly. Authentication and target authorization cannot be bypassed by admission configuration.
• A real HTTP/SSE fixture holds bodies open and proves bounded waiting/inflight work plus actual upstream cancellation and unaffected independent targets.
• The 3,000 ms / 2,800 ms primary scenario from [Task][AI Proxy] Apply a shared request deadline across AI attempts, backoff and fallback #8121 cannot grant a fresh 5,000 ms fallback allowance.
• Measure queue/inflight state and ordinary-path overhead with stated scope; document aggregate exposure across replicas rather than claiming a distributed hard limit.

Boundaries

#7951/#8095 own local/remote MCP resource admission, #7957 distributed MCP quotas, #7702/#7848 reservations, #7958 token/cost policy, and #7930 latency-adaptive concurrency. Reuse compatible primitives after their contracts are accepted; do not copy an unmerged implementation or expand this task to automatic adaptive control, a new queue broker, monetary accounting or tool replay. Gateway runtime does not query the Admin database.

Queue and rejection metrics (2026-10-30)

Implement the #8055 performance/reliability dictionary for the actual admission boundary: queue residence, inflight/queued state, wait deadline, overload and enforced rejection. Separate application queue time from connection-pool acquisition, upstream latency and retry backoff. Define the evaluated/eligible admission population and distinguish local auth failure, overload, quota rejection, cancellation-before-launch, shadow would-reject and provider 429.

Use bounded reasons and controlled dimensions, without raw tenant/request identifiers as general metric labels. Preserve actual-start counting and streaming permit lifetime; queued, suppressed or cache-served work is not an upstream attempt. Test cancellation/deadline removal and queue saturation without inflating provider-unavailability or tool/model failure rates. Adaptive/breaker metrics remain with their owning integrations; this static admission task does not fabricate them.