Skip to content

fix(core): initialize refine_template in AnswerRelevancyEvaluator - #23489

Open
BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-answer-relevancy-get-prompts-attributeerror
Open

BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-answer-relevancy-get-prompts-attributeerror

Conversation

@BlueX888

Copy link
Copy Markdown

Description

AnswerRelevancyEvaluator.get_prompts() raised AttributeError on every invocation. _get_prompts() returns a "refine_template" entry read from self._refine_template (llama-index-core/llama_index/core/evaluation/answer_relevancy.py:94 on main), but __init__ never initialized that attribute, and the constructor accepted no refine_template argument — even though the class docstring already documents one (answer_relevancy.py:62 on main) and _update_prompts() already handles the key (answer_relevancy.py:101-102).

Since PromptMixin.get_prompts() is the public prompt-management API, this evaluator was the only prompt-bearing evaluator in the package whose get_prompts() crashed unconditionally:

>>> AnswerRelevancyEvaluator().get_prompts()
AttributeError: 'AnswerRelevancyEvaluator' object has no attribute '_refine_template'. Did you mean: '_eval_template'?

The fix adds the documented refine_template constructor parameter and initializes self._refine_template with a new DEFAULT_REFINE_TEMPLATE fallback, mirroring how every sibling evaluator in the same package already wires up its templates:

  • FaithfulnessEvaluator (faithfulness.py:122, 139-143)
  • RelevancyEvaluator (relevancy.py:65, 77-81)
  • ContextRelevancyEvaluator (context_relevancy.py:89, 107-111)

The new parameter is optional and defaults to None, so existing callers are unaffected. The only construction site inside the repo (tools/eval_query_engine.py:75) passes no arguments.

No linked issue — reproduced directly on current main.

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

  • Yes
  • No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

  • Yes
  • No

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • I added new unit tests to cover this change

New llama-index-core/tests/evaluation/test_answer_relevancy.py covers get_prompts(), the refine_template constructor parameter, and update_prompts(). On unpatched main the first two tests fail — test_get_prompts with AttributeError: 'AnswerRelevancyEvaluator' object has no attribute '_refine_template' and test_init_refine_template with TypeError: ... unexpected keyword argument 'refine_template'. With this change all three pass.

Run from llama-index-core/:

.venv/bin/python -m pytest tests/evaluation/test_answer_relevancy.py -v   # 3 passed
.venv/bin/python -m pytest tests/evaluation/                               # 50 passed (3 new + 47 existing)
.venv/bin/python -m pytest tests/prompts/                                  # 29 passed
pre-commit run --files llama_index/core/evaluation/answer_relevancy.py tests/evaluation/test_answer_relevancy.py   # all hooks passed
git ls-files | xargs pre-commit run --show-diff-on-failure --files          # full make-lint scope: all hooks passed (ruff, ruff-format, mypy, codespell, ...)

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods

AnswerRelevancyEvaluator.__init__ never set self._refine_template,
but _get_prompts() reads it, so the public PromptMixin API
get_prompts() raised AttributeError on every invocation.

Add the documented refine_template parameter and initialize it
with a default template, matching FaithfulnessEvaluator,
RelevancyEvaluator, and ContextRelevancyEvaluator.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant