AgentGoalAccuracyWithoutReference never sets output_type, so metric.train() instruction optimization aborts with ValueError #3014

• I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug

AgentGoalAccuracyWithoutReference never declares output_type, so it keeps the base default output_type: t.Optional[MetricOutputType] = None (src/ragas/metrics/base.py:167). Calling the public metric.train(path, instruction_config=InstructionConfig(llm=...)) therefore raises

ValueError: Output type for metric 'agent_goal_accuracy' is not defined. Please set the output type in the metric or in the instruction config.

from src/ragas/metrics/base.py:212 — the output-type check that runs before the optimizer is constructed (base.py:211-213). Instruction/prompt optimization is impossible for this metric; it aborts at the check regardless of dataset size or LLM.

Its sibling AgentGoalAccuracyWithReference, defined in the same module, judges the same Literal["0", "1"] verdict (CompareOutcomeOutput) and does declare output_type: t.Optional[MetricOutputType] = MetricOutputType.BINARY (src/ragas/metrics/_goal_accuracy.py:114). The identical flow on the sibling passes that check and proceeds into the optimizer (on a 1-sample toy dataset it then fails later for an unrelated reason: Number of annotations should be greater than 10).

Ragas version: 0.4.4.dev8+g298b68274
Python version: 3.9.6
OS: macOS (Darwin 25.6.0)
Commit: 298b682

Code to Reproduce

import json, tempfile, os
from ragas.config import InstructionConfig
from ragas.llms.base import BaseRagasLLM
from ragas.run_config import RunConfig
from ragas.metrics._goal_accuracy import (
    AgentGoalAccuracyWithReference, AgentGoalAccuracyWithoutReference)

class FakeLLM(BaseRagasLLM):
    run_config = RunConfig()
    def generate_text(self, *a, **k): raise NotImplementedError
    async def agenerate_text(self, *a, **k): raise NotImplementedError
    def is_finished(self, response): return True

path = os.path.join(tempfile.mkdtemp(), "train.json")
json.dump({"agent_goal_accuracy": [{
    "metric_input": {"user_input": [{"content": "book a flight to Paris", "type": "human"}],
                     "reference": "a flight to Paris is booked"},
    "metric_output": 1.0,
    "prompts": {},
    "is_accepted": True,
    "target": 1.0,
}]}, open(path, "w"))

for cls in (AgentGoalAccuracyWithReference, AgentGoalAccuracyWithoutReference):
    m = cls(llm=FakeLLM())
    print(f"{cls.__name__}.output_type = {m.output_type!r}")
    try:
        m.train(path, instruction_config=InstructionConfig(llm=FakeLLM()))
    except Exception as e:
        print(f"  train() -> {type(e).__name__}: {e}")
    else:
        print("  train() -> completed")

Error trace

Two independent runs (Python 3.9.6, commit 298b682) produced the same result. Verbatim output:

AgentGoalAccuracyWithReference.output_type = <MetricOutputType.BINARY: 'binary'>
  train() -> ValueError: Number of annotations should be greater than 10. Please annotate 9 more samples
AgentGoalAccuracyWithoutReference.output_type = None
  train() -> ValueError: Output type for metric 'agent_goal_accuracy' is not defined. Please set the output type in the metric or in the instruction config.

A second run against a patched worktree (line added, 12 annotations; log lines truncated at the capture boundary):

### SRC=<unpatched tree>  -> ValueError: Output type for metric 'agent_goal_accuracy' is not defined. Please set the output type in the metric or in th
### SRC=<patched worktree> Initializing Population Step 1/4 ... Feedback Mutation Step 2/4 (optimiser runs)

The full traceback for the unpatched case is the ValueError raised in BaseMetric._optimize_instruction:

File "src/ragas/metrics/base.py", line 212, in _optimize_instruction
    raise ValueError(
ValueError: Output type for metric 'agent_goal_accuracy' is not defined. Please set the output type in the metric or in the instruction config.

Expected behavior

AgentGoalAccuracyWithoutReference should declare output_type: t.Optional[MetricOutputType] = MetricOutputType.BINARY, so that train(..., instruction_config=InstructionConfig(llm=...)) reaches the optimizer instead of raising, exactly as its sibling does.

Concrete basis:

• The sibling class in the same module, AgentGoalAccuracyWithReference, declares output_type: t.Optional[MetricOutputType] = MetricOutputType.BINARY at src/ragas/metrics/_goal_accuracy.py:114. The two classes are otherwise near-identical copies and both score the same Literal["0", "1"] verdict, so BINARY is the intended value.
• The base default is None: src/ragas/metrics/base.py:167.
• Commit 9bd1402 (PR feat: add output type to metrics #1722, "feat: add output type to metrics") added output_type to LLM-based metrics "to derive the loss required for optimising the metric". Its diff for src/ragas/metrics/_goal_accuracy.py adds exactly one line — the one at line 114 — to AgentGoalAccuracyWithReference, while AgentGoalAccuracyWithoutReference already existed in the same file and was skipped.
• Every other LLM-backed metric in the package declares an output type (_aspect_critic.py:98 BINARY, _simple_criteria.py DISCRETE, _context_recall.py:108, _faithfulness.py:145, _answer_correctness.py:163, ...). This class is the only multi-turn LLM metric left at the base default.
• The value is enforced independently downstream: src/ragas/optimizers/genetic.py:549 raises ValueError("No output type provided for the metric.").

Side note: the error message is misleading. InstructionConfig (src/ragas/optimizers/config.py:29-36) has no output_type field, only loss, so "or in the instruction config" gives the user no way to clear the error.

Additional context

Root cause: missing field in the dataclass body of AgentGoalAccuracyWithoutReference — src/ragas/metrics/_goal_accuracy.py:157, between _required_columns (line 150) and workflow_prompt (line 157). MetricOutputType is already imported in the file, and adding the line right after _required_columns mirrors line 114.

Related references:

• PR feat: add output type to metrics #1722 / commit 9bd1402 (merged) — introduced output_type; added it to the sibling only.
• PR docs: fix correct typos in source code and documentation #2987 (open, "docs: fix correct typos") also edits src/ragas/metrics/_goal_accuracy.py, but only line 86, so there is no textual conflict with a fix here.
• Issue [Bug] AgentGoalAccuracyWithoutReference is not implementing _ascore method #1565 is unrelated (AgentGoalAccuracyWithoutReference not implementing _ascore). No existing issue or PR covers the missing output_type.

Happy to open a PR adding the one-line output_type declaration plus a regression test asserting output_type is MetricOutputType.BINARY for both goal-accuracy classes.