Skip to content

fix(core): make legacy images kwarg migration idempotent and round-trip safe - #23463

Open
BlueX888 wants to merge 3 commits into
run-llama:mainfrom
BlueX888:fix/prep-chatmessage-images-kwargs-roundtrip-crash
Open

BlueX888 wants to merge 3 commits into
run-llama:mainfrom
BlueX888:fix/prep-chatmessage-images-kwargs-roundtrip-crash

Conversation

@BlueX888

Copy link
Copy Markdown

Description

The backward-compat validator legacy_additional_kwargs_image (llama-index-core/llama_index/core/base/llms/types.py:1182) migrates a legacy additional_kwargs["images"] list into ImageBlocks, but it left the images key in place and unconditionally called doc.resolve_image(...) on every entry. That breaks the ChatMessage in two ways:

  1. model_dump() -> model_validate() (the canonical pydantic round-trip) crashes: the dump serializes the leftover ImageDocument values into plain dicts, and re-validation calls .resolve_image() on a dict -> AttributeError: 'dict' object has no attribute 'resolve_image'. The same crash hits SimpleChatStore.from_persist_path() when reloading a store that was persisted with such a message, since the persisted JSON also carries the serialized documents.
  2. Re-validating an already-converted message appends the image blocks again, so ImageBlocks are duplicated on every re-validation.

This PR makes the migration idempotent and safe for serialized payloads:

  • Only ImageDocument entries are converted. Entries that are already serialized (plain dicts, as found in chat stores persisted before this change) are skipped; the ImageBlocks they were migrated into round-trip inside blocks, so no data is lost.
  • The images key is removed from additional_kwargs once consumed, so the migration runs exactly once.

No code in the repo reads additional_kwargs["images"] after construction (the validator is the only consumer), so dropping the key has no other behavioral impact.

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

  • Yes
  • No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

  • Yes
  • No

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

  • I added new unit tests to cover this change
  • I believe this change is already covered by existing unit tests

Added tests (all fail on main, pass with this change):

  • tests/base/llms/test_types.py::test_legacy_image_additional_kwargs_roundtrip - model_dump() -> model_validate() round-trip keeps the blocks and does not crash
  • tests/base/llms/test_types.py::test_legacy_image_additional_kwargs_migration_is_idempotent - the images key is consumed once and re-validating the message does not duplicate image blocks
  • tests/base/llms/test_types.py::test_legacy_image_additional_kwargs_skips_serialized_documents - a message validated from its serialized form (documents already migrated into blocks, serialized documents still in additional_kwargs, i.e. the pre-fix persisted format) restores without crashing
  • tests/storage/chat_store/test_simple_chat_store.py::test_persist_legacy_images_message - a message built with the legacy kwarg survives SimpleChatStore.persist() -> from_persist_path()

Commands and results (from the repo root, with llama-index-core installed editable):

$ pytest "llama-index-core/tests/base/llms/test_types.py::test_legacy_image_additional_kwargs_roundtrip" \
         "llama-index-core/tests/base/llms/test_types.py::test_legacy_image_additional_kwargs_migration_is_idempotent" \
         "llama-index-core/tests/base/llms/test_types.py::test_legacy_image_additional_kwargs_skips_serialized_documents" \
         "llama-index-core/tests/storage/chat_store/test_simple_chat_store.py::test_persist_legacy_images_message"
4 passed in 0.05s

$ pytest llama-index-core/tests/base/llms/ llama-index-core/tests/storage/chat_store/ -q
138 passed in 28.68s

$ ruff check llama-index-core/llama_index/core/base/llms/types.py llama-index-core/tests/base/llms/test_types.py llama-index-core/tests/storage/chat_store/test_simple_chat_store.py
All checks passed!

$ ruff format --check llama-index-core/llama_index/core/base/llms/types.py llama-index-core/tests/base/llms/test_types.py llama-index-core/tests/storage/chat_store/test_simple_chat_store.py
3 files already formatted

Type-checking the touched module (repo-pinned mypy 1.11.0 with the package config):

$ cd llama-index-core && mypy llama_index/core/base/llms/types.py
Success: no issues found in 1 source file

The make lint recipe (git ls-files | xargs pre-commit run --files) was also run across every tracked file in the repo: all hooks (ruff, ruff-format, mypy, codespell, prettier, black-docs, ...) passed.

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods

The backward-compat validator converting additional_kwargs['images']
into image blocks left the key behind, holding live ImageDocument
objects. Because of that:

- the canonical pydantic round-trip
  ChatMessage.model_validate(msg.model_dump()) crashed with
  AttributeError: 'dict' object has no attribute 'resolve_image',
  since serialized entries are plain dicts, so a SimpleChatStore could
  be persisted but never reloaded;
- re-validating an already-converted message appended the image block
  again.

Remove the key once converted so the migration is one-time and the
round-trip reconstructs purely from blocks.

Copy link
Copy Markdown

There is a lossy serialized-input inverse distinct from the two existing round-trip cases.

At this head, ChatMessage.legacy_additional_kwargs_image only converts isinstance(doc, ImageDocument) entries and then always deletes additional_kwargs["images"]. That protects the PR's existing fixture, where blocks already contain the converted image. But if an incoming serialized message holds its legacy image document in additional_kwargs without a pre-existing ImageBlock (e.g. a plain JSON/dict import), the validator now silently discards the image and reports a successful ChatMessage containing no image.

Suggested independent negative/control:

def test_serialized_legacy_image_without_existing_block_is_not_silently_lost(png_1px_b64):
    doc = ImageDocument(image=png_1px_b64)
    incoming = {
        "role": "user",
        "blocks": [],
        "additional_kwargs": {"images": [doc.model_dump()]},
    }
    msg = ChatMessage.model_validate(incoming)
    assert any(
        isinstance(block, ImageBlock) and block.image == png_1px_b64
        for block in msg.blocks
    )

Current-source prediction: the new isinstance guard skips the dictionary, deletes images, then returns a message with no image. A positive control using a real ImageDocument continues to convert, and your already-migrated blocks=[ImageBlock(...)] + serialized duplicate control continues to avoid duplication.

Whether to support a dictionary-only input is a compatibility choice; if it isn't supported, it may be safer to reject that input explicitly rather than silently lose image bytes during a successful parse. Could you either accept/reconstruct valid serialized ImageDocument dictionaries only when corresponding blocks aren't present, or document/guard that unsupported boundary? Please preserve idempotence and avoid trusting arbitrary unknown image dicts.

This is a source-predicted regression probe, not a claim that I ran the installed LlamaIndex chat-store suite or observed real user data loss.

…g them

Serialized documents that reconstruct into a resolvable image (path/url,
or data kept by an explicit include) are now converted instead of being
skipped; anything that cannot be reconstructed into a non-empty image is
kept in additional_kwargs unless it is the already-migrated leftover of
a persisted message (blocks already hold its image block).
@BlueX888

Copy link
Copy Markdown
Author

Good catch on the silent-discard boundary, and thank you for spelling out the controls. Pushed 3d91ed6c4 with the dict path reworked along your option (a), with one correction to the negative control itself:

MediaResource.data is declared Field(..., exclude=True) (llama-index-core/llama_index/core/schema.py), so ImageDocument.model_dump() never carries the inline image bytes — not the base64 data, only path/url survive a dump. The proposed assert block.image == png_1px_b64 from a model_dump()ed document therefore can't be satisfied by any implementation: the bytes never reach the validator. The lossy case that is reachable is a serialized document holding path/url, plus the data-less husk a plain dump produces.

The validator now, for dict entries:

  • reconstructs via ImageDocument.model_validate and converts it when it resolves to a non-empty image (path/url, or data when a caller forced it into the dump via include) — deduplicated against an already-present identical ImageBlock;
  • keeps the entry in additional_kwargs when it reconstructs but yields no image (the husk) and no image block exists to have been its migration — so a dict-only import is preserved verbatim instead of silently dropped;
  • consumes such a husk when blocks already hold image blocks, which is the persisted-leftover case this PR targets (data is excluded from dumps, so a husk next to its migrated image block can only be that leftover);
  • keeps anything that fails validation as-is — unknown dicts are never converted, only real serialized ImageDocument payloads are.

New tests: a path-based serialized document is reconstructed into a block; a husk plus an unrelated dict are both retained and survive a model_dump()/model_validate round trip unchanged. The existing persisted-store and round-trip tests still pass, and the double validation from ChatMessage's __init__ re-running the model validator stays idempotent under all of these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants