memory-lancedb: add configurable timeout/retry for embedding calls#56517
memory-lancedb: add configurable timeout/retry for embedding calls#56517amittell wants to merge 2 commits into
Conversation
…and make it configurable
…ng calls The OpenAI embedding client used default settings (60s timeout, 2 retries), which caused cascading failures during API rate-limit storms — the gateway event loop would stall on long-running embed() calls, triggering "Connection error" from the OpenAI SDK. Add `embedding.timeoutMs` (default 10s) and `embedding.maxRetries` (default 1) to the plugin config so users can tune for their setup (local vs remote embedding servers have very different latency profiles). Also bumps the LaunchAgent ThrottleInterval from 1s to 10s to prevent launchd respawn storms when the gateway crashes repeatedly. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
|
Closing this PR because the author has more than 10 active PRs in this repo. Please reduce the active PR queue and reopen or resubmit once it is back under the limit. You can close your own PRs to get back under the limit. |
Greptile SummaryThis PR addresses cascading "Connection error" failures during API rate-limit storms by making the OpenAI embedding client's timeout and retry behavior configurable, and separately raises the overload failover backoff ceiling from 1.5s to 30s (with a config override) and the launchd
Confidence Score: 5/5Safe to merge; all findings are minor style and hardening suggestions with no blocking defects. All three comments are P2: a redundant (but harmless) dual-timeout setting, missing lower-bound validation on config numerics, and a test coupling weakness. None represent present defects or broken behavior. The core logic — wiring configurable timeout/retry into the embedding client and raising the overload backoff ceiling — is correct and well-tested. No files require special attention; optional improvements noted in extensions/memory-lancedb/index.ts (redundant timeout) and extensions/memory-lancedb/config.ts (bounds validation).
|
| Filename | Overview |
|---|---|
| extensions/memory-lancedb/index.ts | Adds configurable timeout/maxRetries to the Embeddings class; timeout is set redundantly at both constructor and per-request level. |
| extensions/memory-lancedb/config.ts | Extends config schema and parser with timeoutMs and maxRetries fields; missing lower-bound validation for both numeric fields. |
| src/agents/pi-embedded-runner/run.ts | Moves OVERLOAD_FAILOVER_BACKOFF_POLICY inside the function and raises maxMs ceiling from 1.5s to 30s with a config-driven override. |
| src/agents/pi-embedded-runner/run.overload-backoff.test.ts | New test file verifies computeBackoff with hardcoded policy values rather than importing production constants from run.ts. |
| src/daemon/launchd-plist.ts | ThrottleInterval bumped from 1s to 10s to prevent launchd respawn storms on repeated gateway crashes. |
| extensions/memory-lancedb/index.test.ts | Test updated to assert the per-request timeout option is passed to embeddings.create. |
| src/config/types.agent-defaults.ts | Adds overloadBackoffMaxMs optional field to embeddedPi agent defaults config type. |
| src/config/schema.base.generated.ts | Generated schema updated with agents.defaults.embeddedPi.overloadBackoffMaxMs entry. |
| src/config/schema.help.ts | Help text added for the new overloadBackoffMaxMs config field. |
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/index.ts
Line: 179-195
Comment:
**Redundant per-request timeout overrides client-level default**
`timeout: this.timeoutMs` is set in both the `OpenAI` constructor (line 182) and the `embeddings.create` per-request options (line 195). The OpenAI SDK applies the constructor-level timeout as the default for every request; the per-request option then overrides it with the exact same value. The per-request call is redundant.
One or the other is sufficient. Keeping the constructor-level setting is idiomatic and covers any future methods added to the class automatically:
```suggestion
const response = await this.client.embeddings.create(params);
```
If you'd prefer the intent to be explicit at the call site instead, remove `timeout: this.timeoutMs` from the constructor options and keep only the per-request option.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: extensions/memory-lancedb/config.ts
Line: 132-133
Comment:
**No bounds validation for `timeoutMs` or `maxRetries`**
The parser accepts any numeric value for both fields without range checks. A `timeoutMs` of `0` would cause every embedding request to time out immediately (or behave unpredictably depending on the OpenAI SDK version), and a negative `maxRetries` is also not meaningful.
Consider adding guards, for example:
```ts
timeoutMs:
typeof embedding.timeoutMs === "number" && embedding.timeoutMs > 0
? embedding.timeoutMs
: undefined,
maxRetries:
typeof embedding.maxRetries === "number" && embedding.maxRetries >= 0
? embedding.maxRetries
: undefined,
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run.overload-backoff.test.ts
Line: 1-15
Comment:
**Test doesn't import production policy values from `run.ts`**
The test constructs its own `BackoffPolicy` with hardcoded values (`initialMs: 250`, `maxMs: 30_000`, `factor: 2`) that mirror the production defaults in `run.ts`. If a future change updates the production defaults (e.g. `initialMs` or `factor`), these tests will continue to pass without catching the regression.
Consider exporting the default policy constants from `run.ts` (or a separate constants file) and importing them in the test, so the test stays coupled to the production values:
```ts
// In run.ts — export the defaults so tests can reference them
export const OVERLOAD_FAILOVER_BACKOFF_DEFAULTS = {
initialMs: 250,
factor: 2,
jitter: 0.2,
defaultMaxMs: 30_000,
} as const;
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "memory-lancedb: add configurable timeout..." | Re-trigger Greptile
| @@ -180,7 +192,7 @@ class Embeddings { | |||
| if (this.dimensions) { | |||
| params.dimensions = this.dimensions; | |||
| } | |||
| const response = await this.client.embeddings.create(params); | |||
| const response = await this.client.embeddings.create(params, { timeout: this.timeoutMs }); | |||
There was a problem hiding this comment.
Redundant per-request timeout overrides client-level default
timeout: this.timeoutMs is set in both the OpenAI constructor (line 182) and the embeddings.create per-request options (line 195). The OpenAI SDK applies the constructor-level timeout as the default for every request; the per-request option then overrides it with the exact same value. The per-request call is redundant.
One or the other is sufficient. Keeping the constructor-level setting is idiomatic and covers any future methods added to the class automatically:
| const response = await this.client.embeddings.create(params); |
If you'd prefer the intent to be explicit at the call site instead, remove timeout: this.timeoutMs from the constructor options and keep only the per-request option.
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/index.ts
Line: 179-195
Comment:
**Redundant per-request timeout overrides client-level default**
`timeout: this.timeoutMs` is set in both the `OpenAI` constructor (line 182) and the `embeddings.create` per-request options (line 195). The OpenAI SDK applies the constructor-level timeout as the default for every request; the per-request option then overrides it with the exact same value. The per-request call is redundant.
One or the other is sufficient. Keeping the constructor-level setting is idiomatic and covers any future methods added to the class automatically:
```suggestion
const response = await this.client.embeddings.create(params);
```
If you'd prefer the intent to be explicit at the call site instead, remove `timeout: this.timeoutMs` from the constructor options and keep only the per-request option.
How can I resolve this? If you propose a fix, please make it concise.| baseUrl: | ||
| typeof embedding.baseUrl === "string" ? resolveEnvVars(embedding.baseUrl) : undefined, |
There was a problem hiding this comment.
No bounds validation for
timeoutMs or maxRetries
The parser accepts any numeric value for both fields without range checks. A timeoutMs of 0 would cause every embedding request to time out immediately (or behave unpredictably depending on the OpenAI SDK version), and a negative maxRetries is also not meaningful.
Consider adding guards, for example:
timeoutMs:
typeof embedding.timeoutMs === "number" && embedding.timeoutMs > 0
? embedding.timeoutMs
: undefined,
maxRetries:
typeof embedding.maxRetries === "number" && embedding.maxRetries >= 0
? embedding.maxRetries
: undefined,Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/config.ts
Line: 132-133
Comment:
**No bounds validation for `timeoutMs` or `maxRetries`**
The parser accepts any numeric value for both fields without range checks. A `timeoutMs` of `0` would cause every embedding request to time out immediately (or behave unpredictably depending on the OpenAI SDK version), and a negative `maxRetries` is also not meaningful.
Consider adding guards, for example:
```ts
timeoutMs:
typeof embedding.timeoutMs === "number" && embedding.timeoutMs > 0
? embedding.timeoutMs
: undefined,
maxRetries:
typeof embedding.maxRetries === "number" && embedding.maxRetries >= 0
? embedding.maxRetries
: undefined,
```
How can I resolve this? If you propose a fix, please make it concise.| import { describe, expect, it } from "vitest"; | ||
| import { computeBackoff, type BackoffPolicy } from "../../infra/backoff.js"; | ||
|
|
||
| /** | ||
| * Tests for the OVERLOAD_FAILOVER_BACKOFF_POLICY ceiling and config-driven override. | ||
| * | ||
| * The policy is built inside `runEmbeddedPiAgent` from | ||
| * `params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000`. | ||
| * These tests verify the policy shape and that `computeBackoff` respects `maxMs`. | ||
| */ | ||
| describe("overload failover backoff policy", () => { | ||
| it("default ceiling is 30s (not 1.5s)", () => { | ||
| const defaultMaxMs = 30_000; | ||
| const policy: BackoffPolicy = { | ||
| initialMs: 250, |
There was a problem hiding this comment.
Test doesn't import production policy values from
run.ts
The test constructs its own BackoffPolicy with hardcoded values (initialMs: 250, maxMs: 30_000, factor: 2) that mirror the production defaults in run.ts. If a future change updates the production defaults (e.g. initialMs or factor), these tests will continue to pass without catching the regression.
Consider exporting the default policy constants from run.ts (or a separate constants file) and importing them in the test, so the test stays coupled to the production values:
// In run.ts — export the defaults so tests can reference them
export const OVERLOAD_FAILOVER_BACKOFF_DEFAULTS = {
initialMs: 250,
factor: 2,
jitter: 0.2,
defaultMaxMs: 30_000,
} as const;Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run.overload-backoff.test.ts
Line: 1-15
Comment:
**Test doesn't import production policy values from `run.ts`**
The test constructs its own `BackoffPolicy` with hardcoded values (`initialMs: 250`, `maxMs: 30_000`, `factor: 2`) that mirror the production defaults in `run.ts`. If a future change updates the production defaults (e.g. `initialMs` or `factor`), these tests will continue to pass without catching the regression.
Consider exporting the default policy constants from `run.ts` (or a separate constants file) and importing them in the test, so the test stays coupled to the production values:
```ts
// In run.ts — export the defaults so tests can reference them
export const OVERLOAD_FAILOVER_BACKOFF_DEFAULTS = {
initialMs: 250,
factor: 2,
jitter: 0.2,
defaultMaxMs: 30_000,
} as const;
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fa5407017
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const overloadBackoffMaxMs = | ||
| params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000; |
There was a problem hiding this comment.
Allow overloadBackoffMaxMs through config validation
This new knob is read at runtime, but it is not actually accepted by the config schema, so users cannot set it without hitting a validation error. runEmbeddedPiAgent now reads agents.defaults.embeddedPi.overloadBackoffMaxMs, yet embeddedPi remains a strict object with only projectSettingsPolicy in the validation schemas (src/config/zod-schema.agent-defaults.ts and the generated base schema), so the override is rejected before this code runs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds reliability controls to reduce cascading failures during API overload by (a) tightening embedding request timeouts/retries in the memory-lancedb plugin and (b) slowing launchd respawn behavior; additionally introduces a new embedded Pi auth failover backoff ceiling.
Changes:
- Add
embedding.timeoutMs/embedding.maxRetriestomemory-lancedbconfig and wire them into the OpenAI embeddings client. - Increase LaunchAgent
ThrottleIntervalto reduce respawn storms on repeated crashes. - Add
agents.defaults.embeddedPi.overloadBackoffMaxMsand a unit test for the backoff ceiling behavior.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/daemon/launchd-plist.ts | Bumps launchd throttle interval constant to slow relaunch storms. |
| src/config/types.agent-defaults.ts | Adds a new agent defaults config field for embedded Pi overload backoff ceiling. |
| src/config/schema.help.ts | Documents the new embedded Pi overload backoff config help text. |
| src/config/schema.base.generated.ts | Adds generated help metadata entry for the new embedded Pi config key. |
| src/agents/pi-embedded-runner/run.ts | Reads overloadBackoffMaxMs from config to build the overload failover backoff policy. |
| src/agents/pi-embedded-runner/run.overload-backoff.test.ts | New tests asserting backoff max ceiling behavior. |
| extensions/memory-lancedb/index.ts | Wires configurable timeout/retries into OpenAI embedding calls. |
| extensions/memory-lancedb/index.test.ts | Updates expectations to include per-request timeout option. |
| extensions/memory-lancedb/config.ts | Extends plugin config parsing/UI hints to accept timeout/retry fields. |
| "agents.defaults.embeddedPi.overloadBackoffMaxMs": { | ||
| help: "Maximum backoff delay in milliseconds before retrying after an API overloaded_error during auth profile failover (default: 30000). Lower values retry faster but risk exhausting the retry budget under sustained load.", | ||
| tags: ["reliability", "performance", "storage"], | ||
| }, |
There was a problem hiding this comment.
This adds help metadata for agents.defaults.embeddedPi.overloadBackoffMaxMs, but the generated JSON schema for agents.defaults.embeddedPi (earlier in this file) still does not define overloadBackoffMaxMs as an allowed property. That leaves docs/UI/schema consumers inconsistent and may cause the setting to be rejected/stripped. Regenerate/update the schema so the key exists under embeddedPi.properties (and aligns with the Zod validator).
| "agents.defaults.embeddedPi.overloadBackoffMaxMs": { | |
| help: "Maximum backoff delay in milliseconds before retrying after an API overloaded_error during auth profile failover (default: 30000). Lower values retry faster but risk exhausting the retry budget under sustained load.", | |
| tags: ["reliability", "performance", "storage"], | |
| }, |
| baseUrl: | ||
| typeof embedding.baseUrl === "string" ? resolveEnvVars(embedding.baseUrl) : undefined, | ||
| dimensions: typeof embedding.dimensions === "number" ? embedding.dimensions : undefined, | ||
| timeoutMs: typeof embedding.timeoutMs === "number" ? embedding.timeoutMs : undefined, | ||
| maxRetries: typeof embedding.maxRetries === "number" ? embedding.maxRetries : undefined, |
There was a problem hiding this comment.
embedding.timeoutMs / embedding.maxRetries are accepted as numbers but aren’t range-checked. This allows negative / non-integer values, which can disable timeouts (or cause immediate timeouts) and can break OpenAI client retry logic. Validate these in parse() (e.g., timeoutMs int > 0, maxRetries int >= 0) and throw a clear error when out of range.
| * after an API overloaded_error. Defaults to 30000 (30 seconds). | ||
| * Higher values preserve the retry budget under sustained load; | ||
| * lower values rotate faster but risk exhausting retries sooner. | ||
| */ | ||
| overloadBackoffMaxMs?: number; |
There was a problem hiding this comment.
The PR description focuses on memory-lancedb embedding timeout/retries and launchd throttling, but this change also introduces a new agent config option (agents.defaults.embeddedPi.overloadBackoffMaxMs). Either update the PR description to cover this behavior change or split it into a separate PR so reviewers/operators don’t miss the new knob.
| const overloadBackoffMaxMs = | ||
| params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000; |
There was a problem hiding this comment.
overloadBackoffMaxMs is read from params.config, but the config validator currently uses a strict AgentDefaultsSchema that only allows projectSettingsPolicy under agents.defaults.embeddedPi. As a result, configs that set agents.defaults.embeddedPi.overloadBackoffMaxMs will be rejected and this override will never take effect. Add this field to src/config/zod-schema.agent-defaults.ts (and any related schema/label generation) so it’s accepted and validated (e.g., int + nonnegative/positive).
| const overloadBackoffMaxMs = | |
| params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000; | |
| const rawOverloadBackoffMaxMs = | |
| params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs; | |
| const overloadBackoffMaxMs = | |
| typeof rawOverloadBackoffMaxMs === "number" && | |
| Number.isFinite(rawOverloadBackoffMaxMs) && | |
| rawOverloadBackoffMaxMs > 0 | |
| ? Math.floor(rawOverloadBackoffMaxMs) | |
| : 30_000; |
Summary
embedding.timeoutMs(default 10s) andembedding.maxRetries(default 1) to the memory-lancedb plugin configThrottleIntervalfrom 1s to 10s to prevent launchd respawn storms on repeated gateway crashesTest plan
pnpm test -- extensions/memory-lancedb/index.test.ts— 12 passedpnpm test -- src/daemon/launchd.test.ts— 25 passedpnpm formatcleanextensions/telegram/(unrelated, already onmain)🤖 Generated with Claude Code