Skip to content

fix(core): include LLM system prompt in FunctionCallingProgram.acall - #23250

Open
BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-function-program-acall-drops-system-prompt
Open

BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-function-program-acall-drops-system-prompt

Conversation

@BlueX888

@BlueX888 BlueX888 commented Sep 25, 2026 •

Copy link
Copy Markdown

Description

FunctionCallingProgram.acall passed the formatted prompt messages straight to apredict_and_call without applying llm._extend_messages, so a function-calling LLM configured with a system_prompt silently lost that system message on the async structured-output path, while the sync path kept it. The user-visible symptom is wider than acall: LLM.astructured_predict goes through FunctionCallingProgram.acall for a function-calling model, so on main it silently drops system_prompt while structured_predict keeps it.

Why (root cause): every sibling path applies self._llm._extend_messages(...) to the formatted prompt messages before handing them to the LLM:

  • FunctionCallingProgram.__call__ — llama-index-core/llama_index/core/program/function_program.py:158-159
  • FunctionCallingProgram.stream_call — llama-index-core/llama_index/core/program/function_program.py:340-341
  • FunctionCallingProgram.astream_call — llama-index-core/llama_index/core/program/function_program.py:391-392
  • LLMTextCompletionProgram (all four sync/async/stream paths) — llama-index-core/llama_index/core/program/llm_program.py:107,134,167,203

LLM._extend_messages is documented as "Add system prompt to chat message list." (llama-index-core/llama_index/core/llms/llm.py:295-296). acall was the one path that skipped it.

Changes:

  • acall now mirrors __call__: format the prompt messages, extend them with the LLM's system prompt, and pass the result as chat_history (llama-index-core/llama_index/core/program/function_program.py:189-190).
  • Added regression tests asserting the first message sent to the LLM has role SYSTEM for both __call__ and acall when the LLM has a system_prompt.

No linked issue — I could not find an existing issue for this drift.

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

  • Yes
  • No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

  • Yes
  • No

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • I added new unit tests to cover this change

New tests in llama-index-core/tests/program/test_function_program.py:

  • test_async_function_program_includes_llm_system_prompt — fails on unpatched main with AssertionError: assert <MessageRole.USER: 'user'> == <MessageRole.SYSTEM: 'system'>, passes with the fix.
  • test_function_program_includes_llm_system_prompt — guards the sync behavior.

Commands and results (run from llama-index-core/):

  • pytest tests/program/test_function_program.py -k system_prompt — 2 passed (before the fix: 1 failed, 1 passed)
  • pytest tests/program/ — 42 passed
  • pytest tests/llms/test_structured_llm.py tests/agent/workflow/test_agent_with_structured_output.py — 16 passed, 2 skipped (identical result on the unpatched tree; the 182 warnings are pre-existing)
  • pre-commit run --files llama-index-core/llama_index/core/program/function_program.py llama-index-core/tests/program/test_function_program.py — all hooks passed (ruff, ruff-format, mypy, codespell, prettier)

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods (ran the same pre-commit hooks on the changed files)

acall passed the formatted prompt messages straight to
apredict_and_call without applying llm._extend_messages, so any
function-calling LLM configured with a system_prompt silently lost
it on the async structured-output path while the sync path kept it.
Apply _extend_messages like __call__, stream_call and astream_call
already do.

@chrikrah chrikrah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BlueX888 I would merge this. The user-visible symptom is wider than acall: LLM.astructured_predict goes through FunctionCallingProgram.acall for a function-calling model, so on main at 7e2c60a it drops system_prompt while structured_predict keeps it.

Evidence at 450097d, llama-index-core from the branch on PYTHONPATH:

$ pytest tests/program -q
42 passed in 0.17s
# function_program.py from 7e2c60a, your tests kept
1 failed, 41 passed    test_async_function_program_includes_llm_system_prompt

$ python probe.py   # CapturingMockFunctionCallingLLM(system_prompt=...), roles it receives
branch:       structured_predict ['system', 'user']   astructured_predict ['system', 'user']
7e2c60a:      structured_predict ['system', 'user']   astructured_predict ['user']

In FunctionCallingProgram, acall was the only one of the four call paths without _extend_messages.

non-blocking: @BlueX888, could the description name astructured_predict? That is what users will search for.

@logan-markewich, four-line fix, verified at 450097d. Anything else you want checked before you merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants