Skip to content

Commit e5d7598

Browse files
authored
fix: keep vector fields in from_existing on RESP2 unified FT.INFO replies (#742)
Fixes #712 ## Problem With redis-py 8 on a RESP2 connection created with `legacy_responses=False`, `FT.INFO` puts vector params into the field's `flags` list: ```python {'identifier': 'h', 'attribute': 'h', 'type': 'VECTOR', 'flags': ['algorithm', 'HNSW', 'data_type', 'FLOAT32', 'dim', 8, 'distance_metric', 'L2', 'M', 24, 'ef_construction', 300, 'INDEXMISSING']} ``` The dict branch of `parse_vector_attrs` skipped `flags` entirely. It found no `dims` and returned `None`, and `convert_index_info_to_schema` then dropped the field with a bare `continue`. As a result, `SearchIndex.from_existing()` + `create()` silently rebuilt the index without its vector fields. ## Changes 1. The dict branch of `parse_vector_attrs` now unpacks key/value pairs from `flags`. It skips valueless modifiers such as `INDEXMISSING`; I checked on Redis 8.4 that `INDEXMISSING` is appended as a bare token after the pairs. Top-level keys still take precedence, so RESP3 replies (params at top level, empty `flags`) behave as before. 2. When a vector field is skipped, a warning now names the field and the index, so the skip is no longer silent. ## Testing - Reproduced against `redis:8.4` + redis-py 8.1.0 with a FLAT and an HNSW (`m=24`, `ef_construction=300`) vector field. Before this change, `from_existing` returned `['cat', 'text']` for `protocol=2, legacy_responses=False`. After it, all three client configurations (`protocol=2`, `protocol=2, legacy_responses=False`, `protocol=3`) recover the identical schema, including the HNSW params. - New unit tests in `tests/unit/test_convert_index_info.py`: `flags` unpacking with a trailing `INDEXMISSING`, and the skip warning. Both fail on `main` and pass here. - `make format` and `make check-types` pass. Running `tests/unit` (excluding `test_mcp`) plus `tests/integration/test_connection.py` gives the same result as on `main`: the pre-existing `test_repr` failures and `google_genai` optional-dependency errors, which are unrelated to this change, plus the 2 new tests passing. AI disclosure: this change was prepared with an AI coding assistant (Claude Code). I reviewed the diff and ran the tests above. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes index schema recovery used by `from_existing()` and recreate flows; incorrect parsing could still drop or mis-specify vector fields, though behavior is narrowed with tests and explicit warnings. > > **Overview** > Fixes **silent loss of vector fields** when introspecting indexes with **redis-py 8** on **RESP2** and `legacy_responses=False`, where `FT.INFO` puts vector settings in the field's `flags` list instead of top-level keys. > > `parse_vector_attrs` now walks `flags` as known key/value pairs (via `_VECTOR_PARAM_KEYS`), skips valueless modifiers like `INDEXMISSING`, and tolerates unknown bare tokens without mis-pairing later params. **Dim** parsing no longer raises on bad values; unparseable vector fields are **skipped with a warning** naming the field and index. Integration coverage adds `protocol=2, legacy_responses=False`; unit tests cover HNSW recovery, unknown tokens, and skip/warn paths. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit d4d7a2e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 6cb34e2 commit e5d7598

4 files changed

Lines changed: 196 additions & 4 deletions

File tree

‎redisvl/redis/connection.py‎

Lines changed: 60 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,34 @@ def normalize_index_definition(index_info: dict[str, Any]) -> dict[str, Any]:
318318
return _normalize_index_info_mapping(index_info.get("index_definition"))
319319

320320

321+
# Vector parameter names that parse_vector_attrs reads from FT.INFO. In a
322+
# RESP2 unified ``flags`` list, only these are paired with the next token, so
323+
# an unknown bare token costs one position instead of shifting every pair.
324+
_VECTOR_PARAM_KEYS = frozenset(
325+
{
326+
"algorithm",
327+
"data_type",
328+
"datatype",
329+
"type",
330+
"dim",
331+
"dims",
332+
"distance_metric",
333+
"initial_cap",
334+
"block_size",
335+
"m",
336+
"ef_construction",
337+
"ef_runtime",
338+
"epsilon",
339+
"compression",
340+
"reduce",
341+
"graph_max_degree",
342+
"construction_window_size",
343+
"search_window_size",
344+
"training_threshold",
345+
}
346+
)
347+
348+
321349
def normalize_index_fields(index_info: dict[str, Any]) -> list[dict[str, Any]]:
322350
"""Return FT.INFO field entries in mapping form."""
323351
return [
@@ -373,6 +401,18 @@ def parse_vector_attrs(attrs):
373401
for key, value in attrs.items()
374402
if key not in {"identifier", "attribute", "type", "flags"}
375403
}
404+
# redis-py 8 on RESP2 with legacy_responses=False puts the vector
405+
# params in "flags" as key/value pairs, possibly followed by
406+
# valueless modifiers: ["algorithm", "FLAT", "dim", 4, "INDEXMISSING"]
407+
flags = attrs.get("flags") or []
408+
i = 0
409+
while i < len(flags):
410+
key = str(flags[i]).lower()
411+
if key not in _VECTOR_PARAM_KEYS or i + 1 >= len(flags):
412+
i += 1
413+
continue
414+
vector_attrs.setdefault(key, flags[i + 1])
415+
i += 2
376416
else:
377417
# Check if we have any attributes beyond the type declaration
378418
if len(attrs) <= 6:
@@ -408,10 +448,18 @@ def parse_vector_attrs(attrs):
408448
normalized = {}
409449

410450
# Handle dims/dim field - REQUIRED for vector fields
411-
if "dim" in vector_attrs:
412-
normalized["dims"] = int(vector_attrs.pop("dim"))
413-
elif "dims" in vector_attrs:
414-
normalized["dims"] = int(vector_attrs["dims"])
451+
if "dim" in vector_attrs or "dims" in vector_attrs:
452+
raw_dims = (
453+
vector_attrs.pop("dim")
454+
if "dim" in vector_attrs
455+
else vector_attrs["dims"]
456+
)
457+
try:
458+
normalized["dims"] = int(raw_dims)
459+
except (ValueError, TypeError):
460+
# Unparseable dims: fall through to the required-dims check
461+
# below, which skips the field with a warning.
462+
pass
415463
else:
416464
# If dims is missing from normal parsing, try scanning the raw attrs
417465
# This handles edge cases where the format is unexpected
@@ -604,6 +652,14 @@ def parse_attrs(attrs, field_type=None):
604652
if attrs is None:
605653
# Vector field attributes cannot be parsed on this Redis version
606654
# Skip this field - it cannot be properly reconstructed
655+
logger.warning(
656+
"Skipping vector field %r of index %r: could not parse its "
657+
"attributes from FT.INFO. The recovered schema is incomplete, "
658+
"and recreating the index from it (e.g. create(overwrite=True)) "
659+
"drops this field.",
660+
name,
661+
index_name,
662+
)
607663
continue
608664
field["attrs"] = attrs
609665
else:

‎tests/integration/test_async_search_index.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -547,6 +547,14 @@ async def test_batch_search(async_index):
547547
reason="legacy_responses requires redis-py 8",
548548
),
549549
),
550+
pytest.param(
551+
{"protocol": 2, "legacy_responses": False},
552+
id="resp2-new-response-format",
553+
marks=pytest.mark.skipif(
554+
int(redis.__version__.split(".")[0]) < 8,
555+
reason="legacy_responses requires redis-py 8",
556+
),
557+
),
550558
],
551559
)
552560
async def test_client_from_existing_and_batch_search(

‎tests/integration/test_search_index.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -610,6 +610,14 @@ def test_batch_search(index):
610610
reason="legacy_responses requires redis-py 8",
611611
),
612612
),
613+
pytest.param(
614+
{"protocol": 2, "legacy_responses": False},
615+
id="resp2-new-response-format",
616+
marks=pytest.mark.skipif(
617+
int(redis.__version__.split(".")[0]) < 8,
618+
reason="legacy_responses requires redis-py 8",
619+
),
620+
),
613621
],
614622
)
615623
def test_client_from_existing_and_batch_search(

‎tests/unit/test_convert_index_info.py‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,3 +241,123 @@ def test_convert_index_info_default_stopwords():
241241

242242
assert result["index"]["name"] == "test_default_stopwords"
243243
assert "stopwords" not in result["index"] # Should not be present
244+
245+
246+
def test_convert_index_info_resp2_unified_vector_flags():
247+
"""Vector params packed into ``flags`` (redis-py 8, RESP2, legacy_responses=False)."""
248+
index_info = {
249+
"index_name": "test_index",
250+
"index_definition": {"key_type": "HASH", "prefixes": ["doc"]},
251+
"attributes": [
252+
{
253+
"identifier": "emb",
254+
"attribute": "emb",
255+
"type": "VECTOR",
256+
"flags": [
257+
"algorithm",
258+
"HNSW",
259+
"data_type",
260+
"FLOAT32",
261+
"dim",
262+
8,
263+
"distance_metric",
264+
"L2",
265+
"M",
266+
24,
267+
"ef_construction",
268+
300,
269+
"INDEXMISSING",
270+
],
271+
}
272+
],
273+
}
274+
275+
result = convert_index_info_to_schema(index_info)
276+
277+
# INDEXMISSING is deliberately not recovered as ``index_missing``: a vector
278+
# field can't be recreated with it yet, so restoring it here would turn a
279+
# quiet loss into a failure on ``create()``.
280+
assert result["fields"] == [
281+
{
282+
"name": "emb",
283+
"type": "vector",
284+
"attrs": {
285+
"dims": 8,
286+
"distance_metric": "l2",
287+
"algorithm": "hnsw",
288+
"datatype": "float32",
289+
"m": 24,
290+
"ef_construction": 300,
291+
},
292+
}
293+
]
294+
295+
296+
def test_convert_index_info_warns_on_unparseable_vector_field(caplog):
297+
"""A vector field without params is skipped with a warning, not silently."""
298+
index_info = {
299+
"index_name": "test_index",
300+
"index_definition": {"key_type": "HASH", "prefixes": ["doc"]},
301+
"attributes": [
302+
{"identifier": "emb", "attribute": "emb", "type": "VECTOR", "flags": []}
303+
],
304+
}
305+
306+
with caplog.at_level("WARNING"):
307+
result = convert_index_info_to_schema(index_info)
308+
309+
assert result["fields"] == []
310+
assert "'emb'" in caplog.text and "'test_index'" in caplog.text
311+
312+
313+
def test_convert_index_info_resp2_unified_flags_skip_unknown_tokens():
314+
"""An unknown bare token in ``flags`` does not shift the key/value pairs."""
315+
index_info = {
316+
"index_name": "test_index",
317+
"index_definition": {"key_type": "HASH", "prefixes": ["doc"]},
318+
"attributes": [
319+
{
320+
"identifier": "emb",
321+
"attribute": "emb",
322+
"type": "VECTOR",
323+
"flags": [
324+
"algorithm",
325+
"FLAT",
326+
"data_type",
327+
"FLOAT32",
328+
"dim",
329+
4,
330+
"NEWMOD",
331+
"distance_metric",
332+
"IP",
333+
],
334+
}
335+
],
336+
}
337+
338+
result = convert_index_info_to_schema(index_info)
339+
340+
assert result["fields"][0]["attrs"]["distance_metric"] == "ip"
341+
assert result["fields"][0]["attrs"]["dims"] == 4
342+
343+
344+
def test_convert_index_info_skips_vector_field_with_unparseable_dims(caplog):
345+
"""A non-numeric ``dim`` skips the field with a warning instead of raising."""
346+
index_info = {
347+
"index_name": "test_index",
348+
"index_definition": {"key_type": "HASH", "prefixes": ["doc"]},
349+
"attributes": [
350+
{
351+
"identifier": "emb",
352+
"attribute": "emb",
353+
"type": "VECTOR",
354+
"flags": ["algorithm", "FLAT", "dim", "FUTUREMOD"],
355+
}
356+
],
357+
}
358+
359+
with caplog.at_level("WARNING"):
360+
result = convert_index_info_to_schema(index_info)
361+
362+
assert result["fields"] == []
363+
assert "'emb'" in caplog.text

0 commit comments

Comments
 (0)