How to return multiple scores in one evaluator

Sometimes it is useful for a custom evaluator or summary evaluator to return multiple metrics. For example, if you have multiple metrics being generated by an LLM judge, you can save time and money by making a single LLM call that generates multiple metrics instead of making multiple LLM calls. To return multiple scores using the Python SDK, simply return a list of dictionaries/objects of the following form:

[
    # 'key' is the metric name
    # 'score' is the value of a numerical metric
    {"key": string, "score": number},
    # 'value' is the value of a categorical metric
    {"key": string, "value": string},
    ... # You may log as many as you wish
]

To do so with the JS/TS SDK, return an object with a ‘results’ key and then a list of the above form

{results: [{ key: string, score: number }, ...]};

Each of these dictionaries can contain any or all of the feedback fields; check out the linked document for more information. Example:

Python: Requires langsmith>=0.2.0
TypeScript: Support for multiple scores is available in langsmith@0.1.32 and higher

def multiple_scores(outputs: dict, reference_outputs: dict) -> list[dict]:
    # Replace with real evaluation logic.
    precision = 0.8
    recall = 0.9
    f1 = 0.85
    return [
        {"key": "precision", "score": precision},
        {"key": "recall", "score": recall},
        {"key": "f1", "score": f1},
    ]

import type { Run, Example } from "langsmith/schemas";

function multipleScores(rootRun: Run, example: Example) {
  // Your evaluation logic here
  return {
      results: [
          { key: "precision", score: 0.8 },
          { key: "recall", score: 0.9 },
          { key: "f1", score: 0.85 },
      ],
  };
}

Rows from the resulting experiment will display each of the scores.

Return categorical vs numerical metrics

Edit the source of this page on GitHub.

Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.

Datasets

Set up evaluations

Analyze experiment results

Annotation & human feedback

Common data types

How to return multiple scores in one evaluator

​Related

Related