Repository navigation
Conversation
acall passed the formatted prompt messages straight to apredict_and_call without applying llm._extend_messages, so any function-calling LLM configured with a system_prompt silently lost it on the async structured-output path while the sync path kept it. Apply _extend_messages like __call__, stream_call and astream_call already do.
chrikrah
approved these changes
Sep 30, 2026
chrikrah
left a comment
There was a problem hiding this comment.
@BlueX888 I would merge this. The user-visible symptom is wider than acall: LLM.astructured_predict goes through FunctionCallingProgram.acall for a function-calling model, so on main at 7e2c60a it drops system_prompt while structured_predict keeps it.
Evidence at 450097d, llama-index-core from the branch on PYTHONPATH:
$ pytest tests/program -q
42 passed in 0.17s
# function_program.py from 7e2c60a, your tests kept
1 failed, 41 passed test_async_function_program_includes_llm_system_prompt
$ python probe.py # CapturingMockFunctionCallingLLM(system_prompt=...), roles it receives
branch: structured_predict ['system', 'user'] astructured_predict ['system', 'user']
7e2c60a: structured_predict ['system', 'user'] astructured_predict ['user']
In FunctionCallingProgram, acall was the only one of the four call paths without _extend_messages.
non-blocking: @BlueX888, could the description name astructured_predict? That is what users will search for.
@logan-markewich, four-line fix, verified at 450097d. Anything else you want checked before you merge?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
FunctionCallingProgram.acallpassed the formatted prompt messages straight toapredict_and_callwithout applyingllm._extend_messages, so a function-calling LLM configured with asystem_promptsilently lost that system message on the async structured-output path, while the sync path kept it. The user-visible symptom is wider thanacall:LLM.astructured_predictgoes throughFunctionCallingProgram.acallfor a function-calling model, so on main it silently dropssystem_promptwhilestructured_predictkeeps it.Why (root cause): every sibling path applies
self._llm._extend_messages(...)to the formatted prompt messages before handing them to the LLM:FunctionCallingProgram.__call__—llama-index-core/llama_index/core/program/function_program.py:158-159FunctionCallingProgram.stream_call—llama-index-core/llama_index/core/program/function_program.py:340-341FunctionCallingProgram.astream_call—llama-index-core/llama_index/core/program/function_program.py:391-392LLMTextCompletionProgram(all four sync/async/stream paths) —llama-index-core/llama_index/core/program/llm_program.py:107,134,167,203LLM._extend_messagesis documented as "Add system prompt to chat message list." (llama-index-core/llama_index/core/llms/llm.py:295-296).acallwas the one path that skipped it.Changes:
acallnow mirrors__call__: format the prompt messages, extend them with the LLM's system prompt, and pass the result aschat_history(llama-index-core/llama_index/core/program/function_program.py:189-190).SYSTEMfor both__call__andacallwhen the LLM has asystem_prompt.No linked issue — I could not find an existing issue for this drift.
New Package?
Did I fill in the
tool.llamahubsection in thepyproject.tomland provide a detailed README.md for my new integration or package?Version Bump?
Did I bump the version in the
pyproject.tomlfile of the package I am updating? (Except for thellama-index-corepackage)Type of Change
How Has This Been Tested?
New tests in
llama-index-core/tests/program/test_function_program.py:test_async_function_program_includes_llm_system_prompt— fails on unpatchedmainwithAssertionError: assert <MessageRole.USER: 'user'> == <MessageRole.SYSTEM: 'system'>, passes with the fix.test_function_program_includes_llm_system_prompt— guards the sync behavior.Commands and results (run from
llama-index-core/):pytest tests/program/test_function_program.py -k system_prompt— 2 passed (before the fix: 1 failed, 1 passed)pytest tests/program/— 42 passedpytest tests/llms/test_structured_llm.py tests/agent/workflow/test_agent_with_structured_output.py— 16 passed, 2 skipped (identical result on the unpatched tree; the 182 warnings are pre-existing)pre-commit run --files llama-index-core/llama_index/core/program/function_program.py llama-index-core/tests/program/test_function_program.py— all hooks passed (ruff, ruff-format, mypy, codespell, prettier)Suggested Checklist:
uv run make format; uv run make lintto appease the lint gods (ran the same pre-commit hooks on the changed files)