Skip to content

fix(core): stop prompt constructors from mutating the caller's metadata dict - #23244

Open
Lesereingrape wants to merge 2 commits into
run-llama:mainfrom
Lesereingrape:fix/prompt-template-metadata-copy
Open

Lesereingrape wants to merge 2 commits into
run-llama:mainfrom
Lesereingrape:fix/prompt-template-metadata-copy

Conversation

@Lesereingrape

Copy link
Copy Markdown

Description

PromptTemplate, ChatPromptTemplate and LangchainPromptTemplate record which kind of
prompt they are by writing into the metadata dict the caller handed them:

# llama-index-core/llama_index/core/prompts/base.py  (main @ cf8311c4, lines 148, 229, 411)
if metadata is None:
    metadata = {}
metadata["prompt_type"] = prompt_type

So a constructor mutates its argument. On main:

shared = {"team": "rag"}
first = PromptTemplate("one {x}", metadata=shared)
second = PromptTemplate("two {x}", metadata=shared, prompt_type="custom_two")

shared                       # {'team': 'rag', 'prompt_type': 'custom_two'}  <- key the caller never set
first.metadata               # {'team': 'rag', 'prompt_type': <PromptType.CUSTOM: 'custom'>}

Three things fall out of that, all measured locally:

  • the caller's own dict comes back carrying a key it never set — which is a problem for the
    common pattern of keeping one metadata mapping and passing it to prompts and to
    TextNode(metadata=...) / an observability span / a log line;
  • a dict reused by several prompts silently accumulates the last-built template's
    prompt_type, so the caller's mapping no longer describes any of them;
  • a caller who legitimately puts prompt_type in their own metadata loses that value, and
    has no way to notice because nothing raises.

The fix is to build the mapping instead of writing into the argument, one line per site:

metadata = {**(metadata or {}), "prompt_type": prompt_type}

The prompt's own metadata content is identical for every input (pydantic already
stores a copy of the dict — prompt.metadata is metadata is False on main too), so this
only removes the side effect on the caller's object. Nothing that works today changes.

Fixes # (no existing issue for this; filing the fix directly)

New Package?

  • Yes
  • No

Version Bump?

  • Yes
  • No — llama-index-core is exempt per the note in this checkbox.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • I added new unit tests to cover this change

llama-index-core/tests/prompts/test_metadata_ownership.py (new) asserts, for each of the
three template classes, that the caller's dict is unchanged after construction, that
prompt.metadata["prompt_type"] is still recorded, that one dict reused by two prompts
stays clean, and that a caller-supplied prompt_type value survives in the caller's dict.
The LangchainPromptTemplate case is importorskip("langchain_core") (not installed in my
local env, so it is skipped here and runs where langchain is present).

Python 3.12.11, llama-index-core 0.14.25, offline (no network in any of these runs).

Red on main (cf8311c4, this test file against the pristine prompts/base.py):

tests/prompts/test_metadata_ownership.py                                4 failed, 1 passed, 1 skipped in 0.72s
test_prompt_template_leaves_the_callers_dict_alone
  E   AssertionError: assert {'team': 'rag', 'prompt_type': <PromptType.CUSTOM: 'custom'>} == {'team': 'rag'}
test_chat_prompt_template_leaves_the_callers_dict_alone                 (same shape)
test_one_dict_reused_by_several_prompts_stays_clean                     (same shape)
test_metadata_supplied_by_the_caller_survives_untouched
  E   AssertionError: assert {'prompt_type': <PromptType.CUSTOM: 'custom'>, 'extra': 1}
                          == {'prompt_type': 'caller_value', 'extra': 1}

The one that passes on main is the control (test_default_metadata_is_still_created) — the
no-argument path, which is exactly the behaviour that must not move.

Green on this branch:

tests/prompts/test_metadata_ownership.py                    5 passed, 1 skipped in 0.04s
tests/prompts/                                             34 passed, 1 skipped in 6.10s
tests/prompts tests/selectors tests/llms tests/program
tests/question_gen tests/output_parsers                    127 passed, 1 skipped in 7.28s

Lint / types:

$ [email protected] check  llama-index-core/llama_index/core/prompts/base.py llama-index-core/tests/prompts/test_metadata_ownership.py
All checks passed!
$ [email protected] format --check <same two files>
2 files already formatted
$ mypy --ignore-missing-imports --follow-imports=silent <same two files>
Success: no issues found in 1 source file   (run per file; 2/2 clean)

ruff pinned to 0.11.8, the rev .pre-commit-config.yaml uses. I did not run the full
llama-dev test selector (its runner is Linux-only), so the numbers above are per-package
pytest runs of the prompt-adjacent suites rather than the complete llama-index-core suite.

Suggested Checklist

The remaining checklist items are left unticked on purpose: this patch was written and
measured by an automated agent session working for the account owner, and the owner has not
independently re-run it, so the self-review / "checked this doesn't break the public
interface" attestations belong to a human, not to me. Happy to answer anything a maintainer
wants verified, and to re-run a wider suite on request.

@chrikrah chrikrah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lesereingrape I would merge the production change. The Langchain hunk works, but its test cannot pass in any environment, so nothing pins it.

test_langchain_prompt_template_leaves_the_callers_dict_alone guards on langchain_core, and LangchainPromptTemplate needs langchain plus langchain_classic through core/bridge/langchain.py. With both installed, the test then passes a langchain PromptTemplate as selector=, and the constructor calls selector.default_prompt on it. Passing it as template= is what the constructor expects.

Evidence at 116ccd9, llama-index-core from the branch on PYTHONPATH:

$ pytest tests/prompts -q
34 passed, 1 skipped            # the skip is the Langchain test
# prompts/base.py from origin/main, tests kept
4 failed, 30 passed, 1 skipped

# + langchain 1.4.3, langchain-classic 1.0.8, langchain-core 1.6.6
$ pytest tests/prompts/test_metadata_ownership.py -k langchain -q
AttributeError: 'PromptTemplate' object has no attribute 'default_prompt'

$ python probe.py   # LangchainPromptTemplate(template=..., metadata={"team": "rag"})
branch:       caller dict: {'team': 'rag'}
origin/main:  caller dict: {'team': 'rag', 'prompt_type': 'langchain'}

non-blocking: guard on pytest.importorskip("llama_index.core.bridge.langchain") and pass template=. langchain-classic does not pull in langchain, so a langchain_classic guard alone fails where only that is installed. Core's pyproject has no langchain, so this test only ever skips in CI.

@Lesereingrape, want to make that switch? I'll re-run it with langchain installed.

@Lesereingrape

Copy link
Copy Markdown
Author

Both points taken — addressed in 3d51f4b.

  • The guard is now pytest.importorskip("llama_index.core.bridge.langchain"), i.e. the import the constructor actually performs, so the case is skipped when the bridge cannot be built rather than passing vacuously with only langchain_core present.
  • The langchain template is passed as template=, which is what LangchainPromptTemplate.__init__ expects; selector= is only for a ConditionalPromptSelector.

Measured with llama-index-core from this branch on PYTHONPATH (both langchain and langchain_classic installed):

$ pytest tests/prompts -q
35 passed, 0 skipped

The old form reproduces your report. Kept as selector= with this environment, the case fails instead of skipping:

$ pytest tests/prompts/test_metadata_ownership.py -q
FAILED ...::test_langchain_prompt_template_leaves_the_callers_dict_alone
AttributeError: 'PromptTemplate' object has no attribute 'default_prompt'

So the Langchain hunk is now pinned by a test that runs wherever the bridge is importable.

@chrikrah chrikrah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lesereingrape both points land in 3d51f4b and the Langchain case runs here. That is my approval.

Measured with llama-index-core from your branch on PYTHONPATH, pytest 9.1.1. langchain 1.4.3, langchain-classic 1.0.8 and langchain-core 1.6.6 went into a separate target directory:

$ pytest tests/prompts -q                                 # 3d51f4b
35 passed in 0.76s
$ pytest tests/prompts -q                                 # 116ccd9, the previous head
AttributeError: 'PromptTemplate' object has no attribute 'default_prompt'
1 failed, 34 passed in 0.75s
$ pytest tests/prompts -q                                 # cf8311c, the merge base
29 passed in 0.47s
$ pytest tests/prompts -q      # prompts/base.py from cf8311c, your tests kept
5 failed, 30 passed in 0.67s
$ pytest tests/prompts/test_metadata_ownership.py -q -rs  # 3d51f4b, langchain uninstalled
5 passed, 1 skipped in 0.02s
SKIPPED [1] tests/prompts/test_metadata_ownership.py:70: could not import 'llama_index.core.bridge.langchain': No module named 'langchain'
$ ruff check llama_index/core/prompts/base.py tests/prompts/test_metadata_ownership.py
All checks passed!                                        # ruff 0.11.11, the pin in the dev group

The fifth failure in that revert run is the Langchain case. Your third production hunk has a test behind it now.

non-blocking: llama-index-core/uv.lock holds no langchain entry. CI takes the skip branch, so upstream the Langchain hunk is the one with nothing on it; the other four tests pin the two dict-literal hunks only. langchain and langchain-classic in the core dev group would buy the coverage and pull langgraph and langsmith into every dev install. That is a packaging call for core, not something to add here.

@logan-markewich you merged eleven of the changes that touched prompts/base.py. Three one-line hunks and six tests. Anything blocking it going in?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants