Skip to content

Behavior in cases the template is not applicable #749

Open
@kasnerz

Description

@kasnerz

If the template is not applicable for a particular example, the prompt.apply() method returns only a single-element list with an empty string instead of a two-element list with the prompted input and the reference answer. This contradicts the docstring which says that two elements are returned.

Should we just update the docstring, or handle this case with more care?

Example: the e2e_nlg_cleaned dataset with QA-style templates such as food_qa (1acabbc3-c9b9-4624-a684-29faeccff46f). This manifests in the web interface by displaying "∅∅∅ Blank result" instead of an input-target pair.

screen-2022-04-27-11-30-16

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions