Skip to content

fuzzy match gives the wrong answer in eval #139

@cheng-tan

Description

@cheng-tan

for task 361:

our agent gave the answer: Order number 170 is Canceled, order number 189 is Pending

the evaluator is using fuzzy match and evaluated our answer as wrong:

        "eval_types": [
            "string_match"
        ],
        "reference_answers": {
            "fuzzy_match": [
                "170: cancelled",
                "189: pending"
            ]
        },
        "reference_url": "",
        "program_html": [],
        "string_note": "",
        "reference_answer_raw_annotation": "170: cancelled, 189: pending"
    },

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions