Skip to content

fix(core): honor str eval_template in FaithfulnessEvaluator - #23484

Open
BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-faithfulness-str-eval-template-ignored
Open

BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-faithfulness-str-eval-template-ignored

Conversation

@BlueX888

Copy link
Copy Markdown

Description

Summary

FaithfulnessEvaluator silently ignored a custom eval_template passed as a plain string: the constructor wrapped it into a PromptTemplate, then immediately overwrote it with the catalog/default template.

Why

The constructor used two independent if statements instead of if/elif (llama-index-core/llama_index/core/evaluation/faithfulness.py:129-137, before this change):

if isinstance(eval_template, str):
    self._eval_template = PromptTemplate(eval_template)
if isinstance(eval_template, BasePromptTemplate):  # ran even for str inputs
    self._eval_template = eval_template
else:
    self._eval_template = TEMPLATES_CATALOG.get(model_name, DEFAULT_EVAL_TEMPLATE)

For a str input, the second isinstance check was False, so its else branch replaced the just-assigned custom template with TEMPLATES_CATALOG.get(model_name, DEFAULT_EVAL_TEMPLATE). This contradicts the class docstring, which declares eval_template as Optional[Union[str, BasePromptTemplate]] (faithfulness.py:110-111), and the sibling evaluators, which already handle the str case correctly (e.g. evaluation/relevancy.py:67-71, evaluation/multi_modal/faithfulness.py:108-111). No existing issue covers this; found by direct inspection.

Changes

  • Make the second branch an elif so the three cases are exclusive: str -> PromptTemplate(eval_template), BasePromptTemplate -> used as-is, None -> catalog/default lookup (unchanged).
  • Add llama-index-core/tests/evaluation/test_faithfulness.py covering all three constructor paths via get_prompts().

New Package?

  • Yes
  • No

Version Bump?

  • Yes
  • No

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • I added new unit tests to cover this change
  • I believe this change is already covered by existing unit tests

Run in llama-index-core:

  • New tests, on unpatched main: pytest tests/evaluation/test_faithfulness.py -> test_faithfulness_eval_template_from_str FAILED (custom template replaced by the default template); with this change: 3 passed.
  • Existing module tests: pytest tests/evaluation/ -> 50 passed.
  • Repo lint/format: make lint and make format at the repo root -> all hooks passed (ruff, ruff-format, mypy, codespell), no files modified.

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods

A str eval_template was wrapped into a PromptTemplate and then
immediately overwritten by the catalog/default template, since the
isinstance check that followed ran independently of the str branch.
Make the branches exclusive so a str input keeps the custom template,
matching the other evaluators.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant