Skip to content

Bulk memory import can hit OpenAI 431; chunked indexing avoids it #92465

Description

@BrettHamlin

Environment:

  • OpenClaw image: ghcr.io/openclaw/openclaw:2026.6.1
  • Runtime: Docker container, Linux
  • Memory backend: builtin memory with OpenAI hybrid/vector provider
  • Corpus: generated synthetic markdown memory files at 1,000 and 10,000 entry
    scales

Summary:

When indexing a large generated memory corpus as one bulk import/indexing pass,
OpenClaw hit an OpenAI 431 request_headers_too_large failure before producing
usable search results at 1,000 and 10,000 entries. Importing the same style of
corpus in 100-entry chunks and running openclaw memory index --agent main
after each chunk avoided the 431 failure and successfully indexed through
10,000 entries.

Repro shape:

  1. Start an initialized OpenClaw 2026.6.1 container with builtin memory using
    the OpenAI provider.

  2. Generate a memory corpus with many contact-like notes, dated events,
    preferences, and conversation summaries.

  3. Place the full 1,000-entry or 10,000-entry generated memory corpus into the
    workspace memory path.

  4. Run:

    openclaw memory index --agent main

Observed in the bulk path:

  • 100-entry scale indexed successfully.
  • 1,000 and 10,000 entry scales failed before usable search with OpenAI
    431 request_headers_too_large.

Observed in the chunked path:

  • Add 100 generated entries at a time.
  • Run openclaw memory index --agent main after each chunk.
  • 100 chunks completed through 10,000 entries with no 431 error.
  • Representative chunk duration was roughly 4.5s per 100 entries in the
    follow-up run; later full agent-path runs measured about 13 entries/s in
    this local environment.

Expected:

  • Large memory imports should either batch provider calls internally or produce
    an explicit actionable error that tells operators to batch/chunk the import.
  • A single large import should not form provider requests that exceed upstream
    provider header limits.

Impact:

  • Tenant onboarding/history import can fail at realistic memory sizes unless
    the operator manually chunks imports.
  • Managed services need predictable import throughput and retry behavior for
    customer history loads.

Notes:

  • This report is based on generated synthetic data only.
  • No personal state, customer data, or raw provider keys are required for the
    repro.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions