Skip to content

memory-lancedb: add configurable timeout/retry for embedding calls#56517

Closed
amittell wants to merge 2 commits into
openclaw:mainfrom
amittell:fix/lancedb-embedding-timeout
Closed

memory-lancedb: add configurable timeout/retry for embedding calls#56517
amittell wants to merge 2 commits into
openclaw:mainfrom
amittell:fix/lancedb-embedding-timeout

Conversation

@amittell

Copy link
Copy Markdown
Contributor

Summary

  • Add embedding.timeoutMs (default 10s) and embedding.maxRetries (default 1) to the memory-lancedb plugin config
  • The OpenAI embedding client previously used defaults (60s timeout, 2 retries), causing cascading "Connection error" failures during API rate-limit storms as the event loop stalled on long-running embed() calls
  • Also bumps LaunchAgent ThrottleInterval from 1s to 10s to prevent launchd respawn storms on repeated gateway crashes

Test plan

  • pnpm test -- extensions/memory-lancedb/index.test.ts — 12 passed
  • pnpm test -- src/daemon/launchd.test.ts — 25 passed
  • pnpm format clean
  • Pre-existing TS errors in extensions/telegram/ (unrelated, already on main)

🤖 Generated with Claude Code

amittell and others added 2 commits March 23, 2026 00:31
…ng calls

The OpenAI embedding client used default settings (60s timeout, 2 retries),
which caused cascading failures during API rate-limit storms — the gateway
event loop would stall on long-running embed() calls, triggering
"Connection error" from the OpenAI SDK.

Add `embedding.timeoutMs` (default 10s) and `embedding.maxRetries` (default 1)
to the plugin config so users can tune for their setup (local vs remote
embedding servers have very different latency profiles).

Also bumps the LaunchAgent ThrottleInterval from 1s to 10s to prevent
launchd respawn storms when the gateway crashes repeatedly.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Copilot AI review requested due to automatic review settings March 28, 2026 16:51
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime extensions: memory-lancedb Extension: memory-lancedb agents Agent runtime and tooling size: S r: too-many-prs Auto-close: author has more than twenty active PRs. labels Mar 28, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing this PR because the author has more than 10 active PRs in this repo. Please reduce the active PR queue and reopen or resubmit once it is back under the limit. You can close your own PRs to get back under the limit.

@greptile-apps

greptile-apps Bot commented Mar 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR addresses cascading "Connection error" failures during API rate-limit storms by making the OpenAI embedding client's timeout and retry behavior configurable, and separately raises the overload failover backoff ceiling from 1.5s to 30s (with a config override) and the launchd ThrottleInterval from 1s to 10s to dampen respawn storms.

  • extensions/memory-lancedb/config.ts / index.ts: Adds embedding.timeoutMs (default 10s) and embedding.maxRetries (default 1) to the memory-lancedb plugin config; both are wired through to the OpenAI constructor. The same timeoutMs is also passed redundantly as a per-request option (see inline comment).
  • src/agents/pi-embedded-runner/run.ts: Moves OVERLOAD_FAILOVER_BACKOFF_POLICY inside runEmbeddedPiAgent and raises maxMs from 1 500 ms to 30 000 ms; the ceiling is configurable via agents.defaults.embeddedPi.overloadBackoffMaxMs.
  • src/daemon/launchd-plist.ts: LAUNCH_AGENT_THROTTLE_INTERVAL_SECONDS bumped from 1 → 10 to prevent rapid-restart loops on repeated gateway crashes.
  • Config schema and type files are kept in sync with the new fields.

Confidence Score: 5/5

Safe to merge; all findings are minor style and hardening suggestions with no blocking defects.

All three comments are P2: a redundant (but harmless) dual-timeout setting, missing lower-bound validation on config numerics, and a test coupling weakness. None represent present defects or broken behavior. The core logic — wiring configurable timeout/retry into the embedding client and raising the overload backoff ceiling — is correct and well-tested.

No files require special attention; optional improvements noted in extensions/memory-lancedb/index.ts (redundant timeout) and extensions/memory-lancedb/config.ts (bounds validation).

Important Files Changed

Filename Overview
extensions/memory-lancedb/index.ts Adds configurable timeout/maxRetries to the Embeddings class; timeout is set redundantly at both constructor and per-request level.
extensions/memory-lancedb/config.ts Extends config schema and parser with timeoutMs and maxRetries fields; missing lower-bound validation for both numeric fields.
src/agents/pi-embedded-runner/run.ts Moves OVERLOAD_FAILOVER_BACKOFF_POLICY inside the function and raises maxMs ceiling from 1.5s to 30s with a config-driven override.
src/agents/pi-embedded-runner/run.overload-backoff.test.ts New test file verifies computeBackoff with hardcoded policy values rather than importing production constants from run.ts.
src/daemon/launchd-plist.ts ThrottleInterval bumped from 1s to 10s to prevent launchd respawn storms on repeated gateway crashes.
extensions/memory-lancedb/index.test.ts Test updated to assert the per-request timeout option is passed to embeddings.create.
src/config/types.agent-defaults.ts Adds overloadBackoffMaxMs optional field to embeddedPi agent defaults config type.
src/config/schema.base.generated.ts Generated schema updated with agents.defaults.embeddedPi.overloadBackoffMaxMs entry.
src/config/schema.help.ts Help text added for the new overloadBackoffMaxMs config field.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/index.ts
Line: 179-195

Comment:
**Redundant per-request timeout overrides client-level default**

`timeout: this.timeoutMs` is set in both the `OpenAI` constructor (line 182) and the `embeddings.create` per-request options (line 195). The OpenAI SDK applies the constructor-level timeout as the default for every request; the per-request option then overrides it with the exact same value. The per-request call is redundant.

One or the other is sufficient. Keeping the constructor-level setting is idiomatic and covers any future methods added to the class automatically:

```suggestion
    const response = await this.client.embeddings.create(params);
```

If you'd prefer the intent to be explicit at the call site instead, remove `timeout: this.timeoutMs` from the constructor options and keep only the per-request option.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: extensions/memory-lancedb/config.ts
Line: 132-133

Comment:
**No bounds validation for `timeoutMs` or `maxRetries`**

The parser accepts any numeric value for both fields without range checks. A `timeoutMs` of `0` would cause every embedding request to time out immediately (or behave unpredictably depending on the OpenAI SDK version), and a negative `maxRetries` is also not meaningful.

Consider adding guards, for example:

```ts
timeoutMs:
  typeof embedding.timeoutMs === "number" && embedding.timeoutMs > 0
    ? embedding.timeoutMs
    : undefined,
maxRetries:
  typeof embedding.maxRetries === "number" && embedding.maxRetries >= 0
    ? embedding.maxRetries
    : undefined,
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run.overload-backoff.test.ts
Line: 1-15

Comment:
**Test doesn't import production policy values from `run.ts`**

The test constructs its own `BackoffPolicy` with hardcoded values (`initialMs: 250`, `maxMs: 30_000`, `factor: 2`) that mirror the production defaults in `run.ts`. If a future change updates the production defaults (e.g. `initialMs` or `factor`), these tests will continue to pass without catching the regression.

Consider exporting the default policy constants from `run.ts` (or a separate constants file) and importing them in the test, so the test stays coupled to the production values:

```ts
// In run.ts — export the defaults so tests can reference them
export const OVERLOAD_FAILOVER_BACKOFF_DEFAULTS = {
  initialMs: 250,
  factor: 2,
  jitter: 0.2,
  defaultMaxMs: 30_000,
} as const;
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "memory-lancedb: add configurable timeout..." | Re-trigger Greptile

Comment on lines 179 to +195
@@ -180,7 +192,7 @@ class Embeddings {
if (this.dimensions) {
params.dimensions = this.dimensions;
}
const response = await this.client.embeddings.create(params);
const response = await this.client.embeddings.create(params, { timeout: this.timeoutMs });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Redundant per-request timeout overrides client-level default

timeout: this.timeoutMs is set in both the OpenAI constructor (line 182) and the embeddings.create per-request options (line 195). The OpenAI SDK applies the constructor-level timeout as the default for every request; the per-request option then overrides it with the exact same value. The per-request call is redundant.

One or the other is sufficient. Keeping the constructor-level setting is idiomatic and covers any future methods added to the class automatically:

Suggested change
const response = await this.client.embeddings.create(params);

If you'd prefer the intent to be explicit at the call site instead, remove timeout: this.timeoutMs from the constructor options and keep only the per-request option.

Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/index.ts
Line: 179-195

Comment:
**Redundant per-request timeout overrides client-level default**

`timeout: this.timeoutMs` is set in both the `OpenAI` constructor (line 182) and the `embeddings.create` per-request options (line 195). The OpenAI SDK applies the constructor-level timeout as the default for every request; the per-request option then overrides it with the exact same value. The per-request call is redundant.

One or the other is sufficient. Keeping the constructor-level setting is idiomatic and covers any future methods added to the class automatically:

```suggestion
    const response = await this.client.embeddings.create(params);
```

If you'd prefer the intent to be explicit at the call site instead, remove `timeout: this.timeoutMs` from the constructor options and keep only the per-request option.

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines 132 to 133
baseUrl:
typeof embedding.baseUrl === "string" ? resolveEnvVars(embedding.baseUrl) : undefined,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No bounds validation for timeoutMs or maxRetries

The parser accepts any numeric value for both fields without range checks. A timeoutMs of 0 would cause every embedding request to time out immediately (or behave unpredictably depending on the OpenAI SDK version), and a negative maxRetries is also not meaningful.

Consider adding guards, for example:

timeoutMs:
  typeof embedding.timeoutMs === "number" && embedding.timeoutMs > 0
    ? embedding.timeoutMs
    : undefined,
maxRetries:
  typeof embedding.maxRetries === "number" && embedding.maxRetries >= 0
    ? embedding.maxRetries
    : undefined,
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/config.ts
Line: 132-133

Comment:
**No bounds validation for `timeoutMs` or `maxRetries`**

The parser accepts any numeric value for both fields without range checks. A `timeoutMs` of `0` would cause every embedding request to time out immediately (or behave unpredictably depending on the OpenAI SDK version), and a negative `maxRetries` is also not meaningful.

Consider adding guards, for example:

```ts
timeoutMs:
  typeof embedding.timeoutMs === "number" && embedding.timeoutMs > 0
    ? embedding.timeoutMs
    : undefined,
maxRetries:
  typeof embedding.maxRetries === "number" && embedding.maxRetries >= 0
    ? embedding.maxRetries
    : undefined,
```

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines +1 to +15
import { describe, expect, it } from "vitest";
import { computeBackoff, type BackoffPolicy } from "../../infra/backoff.js";

/**
* Tests for the OVERLOAD_FAILOVER_BACKOFF_POLICY ceiling and config-driven override.
*
* The policy is built inside `runEmbeddedPiAgent` from
* `params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000`.
* These tests verify the policy shape and that `computeBackoff` respects `maxMs`.
*/
describe("overload failover backoff policy", () => {
it("default ceiling is 30s (not 1.5s)", () => {
const defaultMaxMs = 30_000;
const policy: BackoffPolicy = {
initialMs: 250,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test doesn't import production policy values from run.ts

The test constructs its own BackoffPolicy with hardcoded values (initialMs: 250, maxMs: 30_000, factor: 2) that mirror the production defaults in run.ts. If a future change updates the production defaults (e.g. initialMs or factor), these tests will continue to pass without catching the regression.

Consider exporting the default policy constants from run.ts (or a separate constants file) and importing them in the test, so the test stays coupled to the production values:

// In run.ts — export the defaults so tests can reference them
export const OVERLOAD_FAILOVER_BACKOFF_DEFAULTS = {
  initialMs: 250,
  factor: 2,
  jitter: 0.2,
  defaultMaxMs: 30_000,
} as const;
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run.overload-backoff.test.ts
Line: 1-15

Comment:
**Test doesn't import production policy values from `run.ts`**

The test constructs its own `BackoffPolicy` with hardcoded values (`initialMs: 250`, `maxMs: 30_000`, `factor: 2`) that mirror the production defaults in `run.ts`. If a future change updates the production defaults (e.g. `initialMs` or `factor`), these tests will continue to pass without catching the regression.

Consider exporting the default policy constants from `run.ts` (or a separate constants file) and importing them in the test, so the test stays coupled to the production values:

```ts
// In run.ts — export the defaults so tests can reference them
export const OVERLOAD_FAILOVER_BACKOFF_DEFAULTS = {
  initialMs: 250,
  factor: 2,
  jitter: 0.2,
  defaultMaxMs: 30_000,
} as const;
```

How can I resolve this? If you propose a fix, please make it concise.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fa5407017

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +826 to +827
const overloadBackoffMaxMs =
params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow overloadBackoffMaxMs through config validation

This new knob is read at runtime, but it is not actually accepted by the config schema, so users cannot set it without hitting a validation error. runEmbeddedPiAgent now reads agents.defaults.embeddedPi.overloadBackoffMaxMs, yet embeddedPi remains a strict object with only projectSettingsPolicy in the validation schemas (src/config/zod-schema.agent-defaults.ts and the generated base schema), so the override is rejected before this code runs.

Useful? React with 👍 / 👎.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds reliability controls to reduce cascading failures during API overload by (a) tightening embedding request timeouts/retries in the memory-lancedb plugin and (b) slowing launchd respawn behavior; additionally introduces a new embedded Pi auth failover backoff ceiling.

Changes:

  • Add embedding.timeoutMs / embedding.maxRetries to memory-lancedb config and wire them into the OpenAI embeddings client.
  • Increase LaunchAgent ThrottleInterval to reduce respawn storms on repeated crashes.
  • Add agents.defaults.embeddedPi.overloadBackoffMaxMs and a unit test for the backoff ceiling behavior.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/daemon/launchd-plist.ts Bumps launchd throttle interval constant to slow relaunch storms.
src/config/types.agent-defaults.ts Adds a new agent defaults config field for embedded Pi overload backoff ceiling.
src/config/schema.help.ts Documents the new embedded Pi overload backoff config help text.
src/config/schema.base.generated.ts Adds generated help metadata entry for the new embedded Pi config key.
src/agents/pi-embedded-runner/run.ts Reads overloadBackoffMaxMs from config to build the overload failover backoff policy.
src/agents/pi-embedded-runner/run.overload-backoff.test.ts New tests asserting backoff max ceiling behavior.
extensions/memory-lancedb/index.ts Wires configurable timeout/retries into OpenAI embedding calls.
extensions/memory-lancedb/index.test.ts Updates expectations to include per-request timeout option.
extensions/memory-lancedb/config.ts Extends plugin config parsing/UI hints to accept timeout/retry fields.

Comment on lines +16120 to +16123
"agents.defaults.embeddedPi.overloadBackoffMaxMs": {
help: "Maximum backoff delay in milliseconds before retrying after an API overloaded_error during auth profile failover (default: 30000). Lower values retry faster but risk exhausting the retry budget under sustained load.",
tags: ["reliability", "performance", "storage"],
},

Copilot AI Mar 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds help metadata for agents.defaults.embeddedPi.overloadBackoffMaxMs, but the generated JSON schema for agents.defaults.embeddedPi (earlier in this file) still does not define overloadBackoffMaxMs as an allowed property. That leaves docs/UI/schema consumers inconsistent and may cause the setting to be rejected/stripped. Regenerate/update the schema so the key exists under embeddedPi.properties (and aligns with the Zod validator).

Suggested change
"agents.defaults.embeddedPi.overloadBackoffMaxMs": {
help: "Maximum backoff delay in milliseconds before retrying after an API overloaded_error during auth profile failover (default: 30000). Lower values retry faster but risk exhausting the retry budget under sustained load.",
tags: ["reliability", "performance", "storage"],
},

Copilot uses AI. Check for mistakes.
Comment on lines 132 to +136
baseUrl:
typeof embedding.baseUrl === "string" ? resolveEnvVars(embedding.baseUrl) : undefined,
dimensions: typeof embedding.dimensions === "number" ? embedding.dimensions : undefined,
timeoutMs: typeof embedding.timeoutMs === "number" ? embedding.timeoutMs : undefined,
maxRetries: typeof embedding.maxRetries === "number" ? embedding.maxRetries : undefined,

Copilot AI Mar 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

embedding.timeoutMs / embedding.maxRetries are accepted as numbers but aren’t range-checked. This allows negative / non-integer values, which can disable timeouts (or cause immediate timeouts) and can break OpenAI client retry logic. Validate these in parse() (e.g., timeoutMs int > 0, maxRetries int >= 0) and throw a clear error when out of range.

Copilot uses AI. Check for mistakes.
Comment on lines +187 to +191
* after an API overloaded_error. Defaults to 30000 (30 seconds).
* Higher values preserve the retry budget under sustained load;
* lower values rotate faster but risk exhausting retries sooner.
*/
overloadBackoffMaxMs?: number;

Copilot AI Mar 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description focuses on memory-lancedb embedding timeout/retries and launchd throttling, but this change also introduces a new agent config option (agents.defaults.embeddedPi.overloadBackoffMaxMs). Either update the PR description to cover this behavior change or split it into a separate PR so reviewers/operators don’t miss the new knob.

Copilot uses AI. Check for mistakes.
Comment on lines +826 to +827
const overloadBackoffMaxMs =
params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000;

Copilot AI Mar 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overloadBackoffMaxMs is read from params.config, but the config validator currently uses a strict AgentDefaultsSchema that only allows projectSettingsPolicy under agents.defaults.embeddedPi. As a result, configs that set agents.defaults.embeddedPi.overloadBackoffMaxMs will be rejected and this override will never take effect. Add this field to src/config/zod-schema.agent-defaults.ts (and any related schema/label generation) so it’s accepted and validated (e.g., int + nonnegative/positive).

Suggested change
const overloadBackoffMaxMs =
params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs ?? 30_000;
const rawOverloadBackoffMaxMs =
params.config?.agents?.defaults?.embeddedPi?.overloadBackoffMaxMs;
const overloadBackoffMaxMs =
typeof rawOverloadBackoffMaxMs === "number" &&
Number.isFinite(rawOverloadBackoffMaxMs) &&
rawOverloadBackoffMaxMs > 0
? Math.floor(rawOverloadBackoffMaxMs)
: 30_000;

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling extensions: memory-lancedb Extension: memory-lancedb gateway Gateway runtime r: too-many-prs Auto-close: author has more than twenty active PRs. size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants