feat(lambda): initialize provisioned concurrency environments - #6320

• Initialize provisioned Docker environments without invoking handlers, and report actual readiness
 and capacity.
• Route matching qualifiers to ready environments, with on-demand spillover when capacity is busy.
• Retire environments on configuration changes and shutdown; document limitations and test lifecycle
 behavior.

graph TD
  API["Configuration API"] --> Pool["Provisioned pool"] --> Launcher["Docker launcher"] --> Runtime["Runtime readiness"]
  Runtime -->|ready or failed| Pool
  Invoke["Lambda invocation"] --> Route{"Matching idle capacity?"}
  Route -->|yes| Pool
  Route -->|no| Warm["On-demand pool"]

Loading

The following are alternative approaches to this PR:

1. Unify provisioned and on-demand pools

• ➕ Could reuse a single container inventory and lease mechanism.
• ➖ Would couple provisioned capacity to idle eviction and complicate qualifier isolation, quota accounting, and teardown.

Recommendation: Keep provisioned environments in a separate owning pool, with the existing warm pool providing spillover and a shared shutdown deadline. Their distinct readiness and lifecycle requirements outweigh the bookkeeping a unified pool might save.

Enhancement (7) +649 / -61

LambdaExecutorService.javaRoute qualified invocations through provisioned capacity +7/-4

Route qualified invocations through provisioned capacity

• Passes the invoked qualifier through synchronous and asynchronous execution. Attempts to acquire a matching provisioned environment before falling back to on-demand execution.

src/main/java/io/github/hectorvent/floci/services/lambda/LambdaExecutorService.java

ProvisionedConcurrencyPool.javaOwn provisioned allocations and their full lifecycle +529/-48

Own provisioned allocations and their full lifecycle

• Turns persisted capacity requests into asynchronously initialized environments with actual status and availability. Manages qualifier-specific leases, capacity ceilings, failure reporting, reconciliation, replacement, and bounded cleanup across deletion, reset, and shutdown.

src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java

WarmPool.javaCoordinate provisioned leases and shared shutdown +24/-3

Coordinate provisioned leases and shared shutdown

• Delegates provisioned handle release and destruction to their owning pool. Drains both pools concurrently against one deadline and closes provisioned workers during shutdown.

src/main/java/io/github/hectorvent/floci/services/lambda/WarmPool.java

ContainerLauncher.javaLaunch Docker environments in provisioned mode +37/-5

Launch Docker environments in provisioned mode

• Adds a provisioned launch path that sets the initialization-type environment variable and requires configured layers and extensions to be available. Leaves ordinary launch behavior unchanged while deferring provisioned readiness to the runtime barrier.

src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java

LambdaRuntimeLauncher.javaExpose optional provisioned-launch capability +8/-0

Expose optional provisioned-launch capability

• Adds capability detection and a dedicated launch method, defaulting to unsupported for other executors.

src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncher.java

RegisteredExtension.javaExpose extension initialization progress +4/-0

Expose extension initialization progress

• Makes receipt of an extension’s first Next request observable to the runtime readiness barrier.

src/main/java/io/github/hectorvent/floci/services/lambda/model/RegisteredExtension.java

RuntimeApiServer.javaTrack initialization without a handler invocation +40/-1

Track initialization without a handler invocation

• Completes readiness only after the runtime and expected extensions issue their first Next requests. Makes initialization waitable and invalidates readiness on shutdown or initialization faults.

src/main/java/io/github/hectorvent/floci/services/lambda/runtime/RuntimeApiServer.java

Tests (5) +1285 / -141

SamTransformIntegrationTest.javaCheck generated SAM alias configuration lookup +7/-1

Check generated SAM alias configuration lookup

• Verifies that the generated published-version alias can be addressed by the provisioned concurrency endpoint and returns the expected not-found response before allocation.

src/test/java/io/github/hectorvent/floci/services/cloudformation/SamTransformIntegrationTest.java

ProvisionedConcurrencyPoolTest.javaExercise allocation, routing, failure, and teardown +939/-130

Exercise allocation, routing, failure, and teardown

• Expands pool tests around initialization, qualifier isolation, persistence, alias changes, quota accounting, failed stops, and concurrent shutdown. Uses a controllable launcher to test late launches, retries, and bounded cleanup.

src/test/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPoolTest.java

WarmPoolTest.javaVerify both pools share a shutdown deadline +73/-9

Verify both pools share a shutdown deadline

• Adds a concurrent-drain test covering the common deadline, canceled removal, and reset retry behavior.

src/test/java/io/github/hectorvent/floci/services/lambda/WarmPoolTest.java

ContainerLauncherTest.javaVerify provisioned Docker launch requirements +126/-1

Verify provisioned Docker launch requirements

• Tests version and initialization-type environment variables, extension discovery, account-owned layer resolution, and rejection of missing layers. Confirms ordinary launch retains its existing behavior.

src/test/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncherTest.java

RuntimeInitializationTest.javaTest the invocation-free readiness barrier +140/-0

Test the invocation-free readiness barrier

• Adds HTTP-level tests showing that runtime and extension Next requests establish readiness without creating a handler event. Covers initialization errors, process exit, and shutdown invalidation.

src/test/java/io/github/hectorvent/floci/services/lambda/runtime/RuntimeInitializationTest.java

Documentation (3) +60 / -41

lambda.mdDocument provisioned runtime behavior and limitations +58/-39

Document provisioned runtime behavior and limitations

• Replaces the control-plane-only description with initialization, routing, failure, persistence, cleanup, and alias-retargeting behavior. Clarifies eligibility, local capacity limits, and the shared shutdown budget.

docs/services/lambda.md

EmulatorConfig.javaClarify the Lambda drain-budget scope +1/-1

Clarify the Lambda drain-budget scope

• Updates the configuration comment to state that provisioned and on-demand containers share one shutdown budget.

src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java

application.ymlClarify the configured shared drain budget +1/-1

Clarify the configured shared drain budget

• Updates the setting’s comment to reflect that its shutdown deadline covers both Lambda pools.

src/main/resources/application.yml

Description

requireCleanupComplete throws a 500 ServiceException after put has already persisted and
swapped in the new allocation, and after deleteMatching has already erased the configuration. It
also fires whenever another thread is already stopping the same handle: stop(handle, false, …)
returns at once in the alreadyStopping branch, owned still holds the handle, and the check
fails. That other thread can be the 1 s reconcileSafely retry, an acquire discard or
initialize's finally block. When this happens, DeleteFunction, DeleteAlias and DeleteVersion stop
partway through under the function lock while the provisioned config is already gone. A retried Put
then gets 409 "still being removed" for a change that has already taken effect.

Code

src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[R352-358]

+    private void requireCleanupComplete(List<ContainerHandle> retired) {
+        synchronized (lock) {
+            if (retired.stream().anyMatch(owned::containsKey)) {
+                throw new AwsException("ServiceException",
+                        "Provisioned environment cleanup is incomplete; retry the operation", 500);
+            }
        }

Relevance

●●● Strong

Recent Lambda lifecycle race fixes are accepted; this exposes persisted-state inconsistency and
partial deletion.

PR-#4733

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Put writes and replaces the allocation (lines 214-220), then calls stop and requireCleanupComplete
(222-223). deleteMatching erases and removes entries before it runs the same check (335-349). stop()
returns immediately for a handle another thread is already stopping when shutdownDrain is false
(621-631), so owned still contains it. LambdaService calls deleteProvisionedFunction/Qualifier
before deleting the function, version or alias, so the exception skips those deletes.

src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[209-225]
src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[328-359]
src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[607-632]
src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java[1104-1125]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`requireCleanupComplete` throws a 500 after Put or Delete has already committed its change. It also throws when a concurrent stopper owns the handle, which makes DeleteFunction, DeleteAlias and DeleteVersion fail partway through.

## Fix Focus Areas
- src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[209-225]
- src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[328-359]
- src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[607-632]

## Recommended Fix
Leave handles whose stop fails in `pendingTeardown` and let `reconcileSafely` retry them, as WarmPool does, instead of throwing after the commit. If a synchronous guarantee is still wanted, have the non-shutdown `alreadyStopping` path wait on the existing latch within the deadline, and run the check before persisting or erasing anything.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

release() retires a handle that is no longer current and calls stop(List.of(handle)), which uses
shutdownDrain=false. Once stopped is set, stop(handles, false, …) returns before any Docker
stop, and the per-handle stop also returns early under stopped && !shutdownDrain. If the shutdown
drain's stop of a leased provisioned container fails or runs out of time, the invocation's later
release never retries the removal, and reconcileSafely also returns while stopped. The container
is orphaned, unlike the on-demand pool, which the docs say starts a separate bounded cleanup.

Code

src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[R599-601]

+            if (stopped && !shutdownDrain) {
+                return;
+            }

Relevance

●●● Strong

WarmPool had the same late-release cleanup bug, with explicit retry-after-shutdown precedent.

PR-#6299

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

release() calls stop(List.of(handle)), which becomes stop(handles, false, now+drain). That method
marks the handle STOPPED and returns when stopped && !shutdownDrain. reconcileSafely also returns
early when stopped. The docs promise a separate bounded cleanup for invocations that finish after
the shutdown deadline, and WarmPool needed the same fix in a past PR.

src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[284-304]
src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[582-613]
src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[366-376]
docs/services/lambda.md[506-519]
PR-#6299

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
After shutdown, ProvisionedConcurrencyPool.release() calls stop() with shutdownDrain=false. That call returns without stopping the container, so a provisioned container whose shutdown removal failed is never removed.

## Fix Focus Areas
- src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[284-304]
- src/main/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPool.java[582-613]

## Recommended Fix
In release(), when `stopped` is true, call `stop(List.of(handle), true, deadline)`. Use `shutdownDeadlineNanos` if it is still in the future, otherwise `now + drainTimeoutNanos`, as initialize() already does in its finally block.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

ProvisionedConcurrencyPoolTest.ready() waits for readiness with Awaitility's five-second
observation window instead of driving reconciliation through a deterministic completion hook.
Scheduler timing or a slow test environment can make these newly added readiness assertions fail
even when initialization eventually succeeds.

Code

src/test/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPoolTest.java[R230-232]

+    private void ready() {
+        await().atMost(Duration.ofSeconds(5)).until(() -> "READY".equals(get().status()));
+    }

Relevance

● Weak

The author explicitly dismissed a matching Awaitility timing concern as intentional.

PR-#6299

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The checklist prohibits fixed observation windows when the system under test provides deterministic
completion mechanisms. The added ready() helper waits up to five seconds for the status to change,
while the pool exposes explicit reconciliation and the tests already invoke it directly elsewhere.

Rule 4156674: In tests, prefer deterministic hooks (e.g., request wait parameters or reconcileAll) over fixed sleeps/polling windows
src/test/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPoolTest.java[230-232]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The provisioned-concurrency tests use a fixed Awaitility timeout to observe asynchronous readiness, making the tests dependent on scheduler timing and arbitrary observation windows.

## Fix Focus Areas
- src/test/java/io/github/hectorvent/floci/services/lambda/ProvisionedConcurrencyPoolTest.java[230-232]

## Recommended Fix
Replace the Awaitility-based `ready()` helper with a deterministic test hook that explicitly completes or reconciles initialization, then assert the final state directly without polling against a fixed timeout.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗