LanceDB version

No response

What happened?

LanceDBClientError exceptions are unpickleable — silently break any multiprocessing-based client

For: LanceDB python client maintainers
Severity: Medium (correctness/observability — masks all server-side errors when raised across a process boundary)
Affected: lancedb python client, observed on lancedb==0.30.1 (paired with pylance==3.0.1)

Summary

LanceDBClientError.__init__ requires request_id as a positional argument. When Python's pickle module reconstructs an instance, it calls __init__(*args) with the args saved by BaseException.__reduce__, which contains only the message string — not the request_id. Result: any LanceDBClientError that crosses a process boundary (e.g., via multiprocessing.Queue, concurrent.futures.ProcessPoolExecutor) crashes with a TypeError on the receiver side, masking the original exception entirely.

In practice this means: any remote-side failure (OOMKilled pod, restart, transient 5xx) propagates to client-side multiprocessing benchmarks as BrokenProcessPool with no information about the original cause.

Minimal reproducer (single process, no infra needed)

# repro.py — runs in ~1s with no LanceDB server required.
import pickle
from lancedb.exceptions import LanceDBClientError  # adjust import path if needed

# Step 1 — construct a normal LanceDBClientError as the client would.
e = LanceDBClientError("connection reset by peer", request_id="abc-123")
print("original:", repr(e))

# Step 2 — pickle the exception (this is exactly what
# multiprocessing.Queue does to send exceptions between worker
# and main process).
buf = pickle.dumps(e)
print("pickled OK; size =", len(buf), "bytes")

# Step 3 — unpickle it. THIS CRASHES.
e2 = pickle.loads(buf)
print("unpickled:", repr(e2))   # never reached

Expected output:

original: LanceDBClientError('connection reset by peer', request_id='abc-123')
pickled OK; size = ... bytes
unpickled: LanceDBClientError('connection reset by peer', request_id='abc-123')

Actual output:

original: LanceDBClientError('connection reset by peer', request_id='abc-123')
pickled OK; size = ... bytes
Traceback (most recent call last):
  File "repro.py", line N, in <module>
    e2 = pickle.loads(buf)
TypeError: LanceDBClientError.__init__() missing 1 required positional argument: 'request_id'

Full reproducer (matches the production failure mode)

This is the actual shape that hits any multi-worker benchmark or app using concurrent.futures.ProcessPoolExecutor against a flaky lancedb endpoint:

# repro_pool.py
from concurrent.futures import ProcessPoolExecutor
import lancedb

REMOTE = "http://your-remote-endpoint:10024"
DB = "your-db"
TABLE = "your-table"
API_KEY = "your-key"

def worker(query_vec):
    db = lancedb.connect(f"db://{DB}", api_key=API_KEY, host_override=REMOTE)
    table = db.open_table(TABLE)
    return table.search(query_vec).limit(10).to_list()

if __name__ == "__main__":
    queries = [[0.0] * 1024 for _ in range(1000)]
    # While this is running, restart any remote pod or trigger any 5xx.
    with ProcessPoolExecutor(max_workers=4) as ex:
        for r in ex.map(worker, queries):
            pass

When a server-side fault occurs mid-run:

concurrent.futures.process.BrokenProcessPool: A process in the process pool
was terminated abruptly while the future was running or pending.

…with no indication that the underlying issue was a server fault.

Observed in production

File "/usr/lib/python3.10/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: LanceDBClientError.__init__() missing 1 required positional argument: 'request_id'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/vector_search_benchmark.py", line 624, in <module>
    main()
  …
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

The root cause was a remote pod OOMKill at 22:15:35, which we only identified by separately inspecting kubectl describe pod. The Python traceback contained zero information about that.

Root cause analysis

Python's default exception pickling uses BaseException.__reduce__(), which serializes the exception as (cls, self.args):

>>> e = LanceDBClientError("oops", request_id="abc-123")
>>> e.__reduce__()
(<class 'lancedb.exceptions.LanceDBClientError'>, ('oops',))
#                                                       ^^^^^^
#                                                       request_id is NOT in args

On unpickle, Python calls cls(*args) — i.e. LanceDBClientError("oops"), which fails because request_id is a required positional/keyword-only argument.

The issue is that LanceDBClientError.__init__ adds attributes beyond what self.args captures. BaseException.__reduce__ only knows about args, so any custom required arg is lost.

Suggested fixes (any one of these works)

Option A — accept **kwargs to absorb the missing argument

class LanceDBClientError(Exception):
    def __init__(self, message, request_id=None, **kwargs):
        super().__init__(message)
        self.request_id = request_id

Simplest possible fix. request_id becomes optional, defaulting to None after unpickle. Minor backward compatibility consideration if callers depend on it being required.

Option B — implement __reduce__ to serialize all state

class LanceDBClientError(Exception):
    def __init__(self, message, request_id):
        super().__init__(message)
        self.request_id = request_id

    def __reduce__(self):
        return (self.__class__, (str(self), self.request_id))

Preserves the strict-construction signature and roundtrips cleanly. Probably the most idiomatic fix.

Option C — __getstate__ / __setstate__

class LanceDBClientError(Exception):
    def __init__(self, message, request_id):
        super().__init__(message)
        self.request_id = request_id

    def __getstate__(self):
        return {"args": self.args, "request_id": self.request_id}

    def __setstate__(self, state):
        self.args = state["args"]
        self.request_id = state["request_id"]

Verbose but explicit. Useful if you anticipate adding more fields.

Option D — factory + permissive init

class LanceDBClientError(Exception):
    def __init__(self, message, request_id=None):
        super().__init__(message)
        self.request_id = request_id

    @classmethod
    def from_response(cls, response):
        return cls(response.text, request_id=response.headers.get("x-request-id"))

Make request_id optional, route all construction sites through a factory that supplies it. Most explicit, but biggest refactor.

Verification

After any fix, the minimal reproducer above should print all three lines successfully, ending with:

unpickled: LanceDBClientError('connection reset by peer', request_id='abc-123')

And the production case should propagate server faults as actual LanceDBClientError instances with usable tracebacks.

Impact / why this matters

• Observability: every server-side failure becomes invisible. Operators see "BrokenProcessPool" when the real cause was OOM, restart, timeout, etc.
• Productivity: we lost ~17 minutes of benchmark work to a single remote OOM. With visible error reporting, we'd have diagnosed and recovered in ~30 seconds.
• Cluster trust: this makes the LanceDB-on-Kubernetes setup feel less reliable than it is — most "client crashes" are actually upstream pod-level events, but you can't tell from the client side.
• Affects any python ecosystem tool built around concurrent.futures.ProcessPoolExecutor, multiprocessing.Pool, dask.distributed, ray, etc. — anywhere exceptions cross process boundaries.

Environment

• lancedb==0.30.1
• pylance==3.0.1
• Python 3.10.12
• Ubuntu 22.04 container

Are there known steps to reproduce?

No response