Skip to content

FewShotPydanticPrompt never forwards threshold_for_examples to ExampleStore.get_examples — DemonstrationConfig.threshold silently ignored (store default 0.7 always used) #3056

Description

@BlueX888

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug

FewShotPydanticPrompt.generate_multiple never forwards self.threshold_for_examples to ExampleStore.get_examples (src/ragas/prompt/few_shot_pydantic_prompt.py:120), so InMemoryExampleStore always filters examples with its own default threshold=0.7. The public DemonstrationConfig.threshold field (src/ragas/config.py:19) — which MetricWithLLM._optimize_demonstration assigns to few_shot_prompt.threshold_for_examples (src/ragas/metrics/base.py:295) — is silently ignored. A user who lowers DemonstrationConfig.threshold to include more similar examples (or raises it to be stricter) gets no effect: in the repro below, with threshold_for_examples=0.4, an example with cosine similarity 0.5 is still dropped. The field is dead in both directions (setting it to 0.9 still keeps the 0.866-similarity example).

Ragas version: 0.4.4.dev8+g298b68274 (main @ 298b682)
Python version: 3.12.14
OS: macOS 26.6.2 (arm64)

Code to Reproduce

import asyncio
import math
import os

os.environ["RAGAS_DO_NOT_TRACK"] = "true"

from pydantic import BaseModel

from ragas.embeddings.base import BaseRagasEmbeddings
from ragas.prompt.few_shot_pydantic_prompt import (
    FewShotPydanticPrompt,
    InMemoryExampleStore,
)
from ragas.prompt.pydantic_prompt import PydanticPrompt


class QIn(BaseModel):
    question: str


class QOut(BaseModel):
    answer: str


class BaseQ(PydanticPrompt[QIn, QOut]):
    instruction = "answer the question"
    input_model = QIn
    output_model = QOut
    examples = [
        (QIn(question="alpha"), QOut(answer="A")),
        (QIn(question="beta"), QOut(answer="B")),
    ]


class FakeEmb(BaseRagasEmbeddings):
    # deterministic 2-D embeddings: cos(query, alpha) = 0.5, cos(query, beta) = 0.866

    def _emb(self, text):
        if "alpha" in text:
            return [1.0, 0.0]
        if "beta" in text:
            return [0.0, 1.0]
        if "query" in text:
            return [0.5, math.sqrt(3) / 2]
        return [1.0, 0.0]

    def embed_query(self, text):
        return self._emb(text)

    def embed_documents(self, texts):
        return [self._emb(t) for t in texts]

    async def aembed_query(self, text):
        return self._emb(text)

    async def aembed_documents(self, texts):
        return [self._emb(t) for t in texts]


class StubLLM:
    async def generate(self, *a, **k):
        raise RuntimeError("STOP-BEFORE-LLM")


calls = []
_orig = InMemoryExampleStore.get_examples


def spy(self, data, top_k=5, threshold=0.7):
    calls.append({"top_k_received": top_k, "threshold_received": threshold})
    return _orig(self, data, top_k, threshold)


InMemoryExampleStore.get_examples = spy


async def main():
    fs = FewShotPydanticPrompt.from_pydantic_prompt(BaseQ(), FakeEmb())
    fs.threshold_for_examples = 0.4  # include the 0.5-similarity example
    fs.top_k_for_examples = 5
    try:
        await fs.generate_multiple(StubLLM(), QIn(question="query"))
    except RuntimeError:
        pass  # examples are selected before the LLM is called
    print("threshold_for_examples configured on prompt:", fs.threshold_for_examples)
    print("get_examples call(s) received by store:      ", calls)
    print("examples actually selected:                  ",
          [(i.question, o.answer) for i, o in fs.examples])


asyncio.run(main())

Error trace

No exception — this is a silent misbehaviour. Output of the script above (no error raised):

threshold_for_examples configured on prompt: 0.4
get_examples call(s) received by store:       [{'top_k_received': 5, 'threshold_received': 0.7}]
examples actually selected:                   [('beta', 'B')]

EXPECTED: threshold_received=0.4 -> both examples (sim 0.5 and 0.866 both >= 0.4)
ACTUAL:   threshold_received=0.7 -> only 'beta' (0.5 < 0.7 is dropped)

Reproduced independently (same output), and also with threshold_for_examples=0.9: the store still received (5, 0.7) and the 0.866-similarity example was still kept — so the field has no effect in either direction.

Expected behavior

The threshold configured via DemonstrationConfig.threshold should reach the example store: get_examples should be called with threshold=0.4, and both examples (similarity 0.5 and 0.866, both >= 0.4) should be selected.

Basis:

  • DemonstrationConfig.threshold is a public config field (src/ragas/config.py:19, default 0.7) that MetricWithLLM._optimize_demonstration explicitly assigns to the prompt: few_shot_prompt.threshold_for_examples = demonstration_config.threshold (src/ragas/metrics/base.py:295), after wrapping the metric's prompt in a FewShotPydanticPrompt (src/ragas/metrics/base.py:288).
  • InMemoryExampleStore.get_examples explicitly accepts and filters on a threshold parameter (src/ragas/prompt/few_shot_pydantic_prompt.py:53-60), which get_nearest_examples uses as similarities >= threshold (few_shot_pydantic_prompt.py:81).
  • The sibling DynamicFewShotPrompt forwards its threshold to the same store API: self.example_store.get_examples(kwargs, self.max_similar_examples, self.similarity_threshold) (src/ragas/prompt/dynamic_few_shot.py:179-181).
  • This flow is the documented training path: metric.train(path, instruction_config=..., demonstration_config=DemonstrationConfig(embedding=...)) (e.g. docs/howtos/applications/vertexai_alignment.md, "train your metric" section).

Additional context

Root cause: src/ragas/prompt/few_shot_pydantic_prompt.py:120 calls self.example_store.get_examples(data, self.top_k_for_examples) without the threshold argument, so the field FewShotPydanticPrompt.threshold_for_examples (few_shot_pydantic_prompt.py:101) is never read and the store applies its own default 0.7. Note the ExampleStore ABC itself does not declare a threshold parameter (few_shot_pydantic_prompt.py:27-31), only the concrete InMemoryExampleStore does.

Happy to open a PR forwarding self.threshold_for_examples to get_examples — either by adding threshold to the ExampleStore ABC signature or by passing it as a keyword with a fallback for custom stores that don't accept it.

I searched issues and PRs for threshold_for_examples and DemonstrationConfig threshold and found no existing report of this.

Activity

  1. BlueX888 commented on Oct 11, 2026

    @BlueX888
    Author

    Reproduced this. Root cause: src/ragas/prompt/few_shot_pydantic_prompt.py:120 — generate_multiple calls self.example_store.get_examples(data, self.top_k_for_examples) without the threshold, so threshold_for_examples (set from DemonstrationConfig.threshold) is never read and InMemoryExampleStore filters with its own default 0.7.

    Minimal reproduction:

    script from the issue body, trimmed to the operative lines
    (embeddings chosen so cos(query,"alpha")=0.5 and cos(query,"beta")=0.866):
    
      fs = FewShotPydanticPrompt.from_pydantic_prompt(BaseQ(), FakeEmb())
      fs.threshold_for_examples = 0.4   # the 0.5-similarity example should qualify too
      fs.top_k_for_examples = 5
      await fs.generate_multiple(StubLLM(), QIn(question="query"))
      # a spy on InMemoryExampleStore.get_examples records what the store received
    

    Observed output:

    threshold_for_examples configured on prompt: 0.4
    get_examples call(s) received by store:       [{'top_k_received': 5, 'threshold_received': 0.7}]
    examples actually selected:                   [('beta', 'B')]
    

    Fix is ready on BlueX888:fix/prep-fewshot-threshold-config-ignored (commit d06755c2). Forwards self.threshold_for_examples to example_store.get_examples in generate_multiple, and adds threshold to the ExampleStore.get_examples signature so custom example stores receive it too.
    Verified with: pytest tests/unit/prompt/test_few_shot_pydantic_prompt.py

    Happy to open the PR once you confirm the approach or assign this to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions