Repository navigation
Description
[x] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
FewShotPydanticPrompt.generate_multiple never forwards self.threshold_for_examples to ExampleStore.get_examples (src/ragas/prompt/few_shot_pydantic_prompt.py:120), so InMemoryExampleStore always filters examples with its own default threshold=0.7. The public DemonstrationConfig.threshold field (src/ragas/config.py:19) — which MetricWithLLM._optimize_demonstration assigns to few_shot_prompt.threshold_for_examples (src/ragas/metrics/base.py:295) — is silently ignored. A user who lowers DemonstrationConfig.threshold to include more similar examples (or raises it to be stricter) gets no effect: in the repro below, with threshold_for_examples=0.4, an example with cosine similarity 0.5 is still dropped. The field is dead in both directions (setting it to 0.9 still keeps the 0.866-similarity example).
Ragas version: 0.4.4.dev8+g298b68274 (main @ 298b682)
Python version: 3.12.14
OS: macOS 26.6.2 (arm64)
Code to Reproduce
import asyncio
import math
import os
os.environ["RAGAS_DO_NOT_TRACK"] = "true"
from pydantic import BaseModel
from ragas.embeddings.base import BaseRagasEmbeddings
from ragas.prompt.few_shot_pydantic_prompt import (
FewShotPydanticPrompt,
InMemoryExampleStore,
)
from ragas.prompt.pydantic_prompt import PydanticPrompt
class QIn(BaseModel):
question: str
class QOut(BaseModel):
answer: str
class BaseQ(PydanticPrompt[QIn, QOut]):
instruction = "answer the question"
input_model = QIn
output_model = QOut
examples = [
(QIn(question="alpha"), QOut(answer="A")),
(QIn(question="beta"), QOut(answer="B")),
]
class FakeEmb(BaseRagasEmbeddings):
# deterministic 2-D embeddings: cos(query, alpha) = 0.5, cos(query, beta) = 0.866
def _emb(self, text):
if "alpha" in text:
return [1.0, 0.0]
if "beta" in text:
return [0.0, 1.0]
if "query" in text:
return [0.5, math.sqrt(3) / 2]
return [1.0, 0.0]
def embed_query(self, text):
return self._emb(text)
def embed_documents(self, texts):
return [self._emb(t) for t in texts]
async def aembed_query(self, text):
return self._emb(text)
async def aembed_documents(self, texts):
return [self._emb(t) for t in texts]
class StubLLM:
async def generate(self, *a, **k):
raise RuntimeError("STOP-BEFORE-LLM")
calls = []
_orig = InMemoryExampleStore.get_examples
def spy(self, data, top_k=5, threshold=0.7):
calls.append({"top_k_received": top_k, "threshold_received": threshold})
return _orig(self, data, top_k, threshold)
InMemoryExampleStore.get_examples = spy
async def main():
fs = FewShotPydanticPrompt.from_pydantic_prompt(BaseQ(), FakeEmb())
fs.threshold_for_examples = 0.4 # include the 0.5-similarity example
fs.top_k_for_examples = 5
try:
await fs.generate_multiple(StubLLM(), QIn(question="query"))
except RuntimeError:
pass # examples are selected before the LLM is called
print("threshold_for_examples configured on prompt:", fs.threshold_for_examples)
print("get_examples call(s) received by store: ", calls)
print("examples actually selected: ",
[(i.question, o.answer) for i, o in fs.examples])
asyncio.run(main())Error trace
No exception — this is a silent misbehaviour. Output of the script above (no error raised):
threshold_for_examples configured on prompt: 0.4
get_examples call(s) received by store: [{'top_k_received': 5, 'threshold_received': 0.7}]
examples actually selected: [('beta', 'B')]
EXPECTED: threshold_received=0.4 -> both examples (sim 0.5 and 0.866 both >= 0.4)
ACTUAL: threshold_received=0.7 -> only 'beta' (0.5 < 0.7 is dropped)
Reproduced independently (same output), and also with threshold_for_examples=0.9: the store still received (5, 0.7) and the 0.866-similarity example was still kept — so the field has no effect in either direction.
Expected behavior
The threshold configured via DemonstrationConfig.threshold should reach the example store: get_examples should be called with threshold=0.4, and both examples (similarity 0.5 and 0.866, both >= 0.4) should be selected.
Basis:
DemonstrationConfig.thresholdis a public config field (src/ragas/config.py:19, default 0.7) thatMetricWithLLM._optimize_demonstrationexplicitly assigns to the prompt:few_shot_prompt.threshold_for_examples = demonstration_config.threshold(src/ragas/metrics/base.py:295), after wrapping the metric's prompt in aFewShotPydanticPrompt(src/ragas/metrics/base.py:288).InMemoryExampleStore.get_examplesexplicitly accepts and filters on athresholdparameter (src/ragas/prompt/few_shot_pydantic_prompt.py:53-60), whichget_nearest_examplesuses assimilarities >= threshold(few_shot_pydantic_prompt.py:81).- The sibling
DynamicFewShotPromptforwards its threshold to the same store API:self.example_store.get_examples(kwargs, self.max_similar_examples, self.similarity_threshold)(src/ragas/prompt/dynamic_few_shot.py:179-181). - This flow is the documented training path:
metric.train(path, instruction_config=..., demonstration_config=DemonstrationConfig(embedding=...))(e.g.docs/howtos/applications/vertexai_alignment.md, "train your metric" section).
Additional context
Root cause: src/ragas/prompt/few_shot_pydantic_prompt.py:120 calls self.example_store.get_examples(data, self.top_k_for_examples) without the threshold argument, so the field FewShotPydanticPrompt.threshold_for_examples (few_shot_pydantic_prompt.py:101) is never read and the store applies its own default 0.7. Note the ExampleStore ABC itself does not declare a threshold parameter (few_shot_pydantic_prompt.py:27-31), only the concrete InMemoryExampleStore does.
Happy to open a PR forwarding self.threshold_for_examples to get_examples — either by adding threshold to the ExampleStore ABC signature or by passing it as a keyword with a fallback for custom stores that don't accept it.
I searched issues and PRs for threshold_for_examples and DemonstrationConfig threshold and found no existing report of this.
Activity
BlueX888 commentedon Oct 11, 2026
Reproduced this. Root cause: src/ragas/prompt/few_shot_pydantic_prompt.py:120 — generate_multiple calls self.example_store.get_examples(data, self.top_k_for_examples) without the threshold, so threshold_for_examples (set from DemonstrationConfig.threshold) is never read and InMemoryExampleStore filters with its own default 0.7.
Minimal reproduction:
Observed output:
Fix is ready on
BlueX888:fix/prep-fewshot-threshold-config-ignored(commitd06755c2). Forwards self.threshold_for_examples to example_store.get_examples in generate_multiple, and adds threshold to the ExampleStore.get_examples signature so custom example stores receive it too.Verified with:
pytest tests/unit/prompt/test_few_shot_pydantic_prompt.pyHappy to open the PR once you confirm the approach or assign this to me.