> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> An evaluation that assesses traces from a session using a metric

> [!WARNING]
>
> You are reading the docs of Galtea 5.13. [Read this page in the latest version](https://docs.galtea.ai/concepts/product/version/session/evaluation.md).

## What is an Evaluation?

An evaluation in Galtea represents the assessment of traces from a [session](https://docs.galtea.ai/v5.13/concepts/product/version/session.md) using the evaluation criteria of a [metric](https://docs.galtea.ai/v5.13/concepts/metric.md). Evaluations are now directly linked to sessions, allowing for comprehensive evaluation of full sessions containing multiple traces for multi-turn dialogues.

> [!NOTE]
>
> Evaluations do not call your AI product — they score outputs that have already been generated. Run your product first to produce outputs, then trigger the evaluation to score them.

You can create evaluations from the [Galtea dashboard](https://platform.galtea.ai/) (using [Endpoint Connections](https://docs.galtea.ai/v5.13/concepts/product/endpoint-connection.md)) or programmatically using the [Galtea SDK](https://docs.galtea.ai/v5.13/sdk/api/evaluation/service.md).

Every launch you trigger creates a [Run](https://docs.galtea.ai/v5.13/concepts/product/run.md), and each evaluation of that launch points back at it. An evaluation produced by a [Monitor](https://docs.galtea.ai/v5.13/concepts/product/monitor.md) has no Run, because it comes from a rule instead of a launch.

## Results Excluded from Analytics

A metric revision can be excluded from analytics. Its evaluations stay available and keep their original score, reason, and status. They stop contributing to scores, coverage, insights, and monitor aggregates.

The evaluation list hides these results by default. To reveal them, open **Filters** and turn on **Show excluded from analytics**. Add the **Analytics** column from the **Columns** menu to see which results are excluded. An excluded evaluation's detail page explains who excluded its metric revision and when, then links to the metric where you can restore it.

Exclusion belongs to the metric revision, not to one evaluation. Restoring that revision returns all its results to analytics.

## Filtering by Score

Every evaluation list has a **Judge Score** filter and a **Human Score** filter. Open **Filters**, then choose one of them to find the evaluations that scored low. Pick a condition (is below, is at most, equals, is at least, is above) and type a percentage from 0 to 100, with at most one decimal. The pill in the filter row shows the condition, for example `≥ 70%`.

* **Judge Score** reads the score the metric's judge gave. It ignores a human review.
* **Human Score** reads the reviewer's score only.
* **Confidence** reads the confidence of a [Classifier (Jev)](https://docs.galtea.ai/v5.13/concepts/metric/classifier.md) result, from 0 to 1 with two decimals. The column and the filter (open **Filters**, then choose **Confidence**) show only on a deployment where classifier metrics are on. Add the column from the **Columns** menu.
* **Result** (open **Filters**, then choose **Result**) keeps the evaluations that passed or failed their metric's [pass threshold](https://docs.galtea.ai/v5.13/concepts/metric.md#pass-threshold). Pick both options, or none, and the filter does nothing. It follows the reported score, so a human review counts, unlike the Judge Score filter. The **Result** column shows PASS, FAIL or a dash, is hidden by default, and is included in the CSV export.
* An evaluation with no score of that kind never matches the filter. An evaluation with no confidence never matches a Confidence bound.
* Every condition compares the score as it displays. For example, a stored score of 0.4252 shows as 42.5%, so it matches `= 42.5%` and `≤ 42.5%`, and not `< 42.5%`.

The API takes the same bounds on `GET /evaluations` as `scoreGt`, `scoreGte`, `scoreLt`, `scoreLte` and `humanScoreGt`, `humanScoreGte`, `humanScoreLt`, `humanScoreLte`, on the stored 0 to 1 scale. It also takes `confidenceGt`, `confidenceGte`, `confidenceLt` and `confidenceLte`, and can sort by `confidence`, with evaluations that have no confidence last in both directions. These bounds compare the stored value exactly, and a bound outside 0 to 1 returns a 400 error. To match one exact score, send the same value as both inclusive bounds, for example `scoreGte=0.5&scoreLte=0.5`. It also takes `passed=true` or `passed=false`, which sends the same verdict filter as the Result filter. The SDK takes `passed` and the score bounds as keyword arguments of [`evaluations.list()`](https://docs.galtea.ai/v5.13/sdk/api/evaluation/list). It does not take the confidence bounds.

## Evaluation Lifecycle

Evaluations follow a specific lifecycle:

**1. Creation**

[Trigger an evaluation](https://docs.galtea.ai/v5.13/sdk/tutorials/run-dataset-based-evaluations.md). It will appear in the session's details page with the status "pending"

**2. Processing**

Galtea's evaluation system processes the evaluation using the evaluation criteria of the selected [metric](https://docs.galtea.ai/v5.13/concepts/metric.md)

**3. Completion**

Once processed, the status changes to "completed" and the [results](https://docs.galtea.ai/v5.13/concepts/product/version/session/evaluation.md#evaluation-properties) are available

## SDK Integration

The Galtea SDK allows you to create, view, and manage evaluations programmatically. This is particularly useful for organizations that want to automate their evaluation process or integrate it into their CI/CD pipeline.

> [card] [Evaluation Service SDK](https://docs.galtea.ai/v5.13/sdk/api/evaluation/service.md)
>
> Manage evaluations using the Python SDK

> [card] [GitHub Actions](https://docs.galtea.ai/v5.13/sdk/integrations/github-actions.md)
>
> Learn how to set up GitHub Actions to automatically evaluate new versions

## Evaluation Properties

### Result Properties

Once an evaluation completes, the following fields are available:

**Status** `Enum`

The current status of the evaluation.
Possible values:

* **Pending**: The evaluation has been created but not yet processed.
* **Pending Human**: The evaluation is waiting for a human annotator to review and score it. This status is used for metrics with `source: "human_evaluation"`.
* **Success**: The evaluation was processed successfully.
* **Failed**: The evaluation encountered an error during processing. Check the `canRetry` field to determine if this evaluation can be retried. An evaluation that stays **Pending** for longer than 24 hours (configurable with `EVALUATION_PENDING_TIMEOUT_HOURS` on an on-premise deployment) is marked **Failed** with `canRetry` set, so you can retry it.
* **Skipped**: The evaluation could not produce a score. The error message names the cause. Common causes are:

  * Metric validation failed, for example a required parameter is missing.
  * The organization ran out of credits.
  * The metric could not read the actual output, for example plain text where [JSON Field Match](https://docs.galtea.ai/v5.13/concepts/metric/json-field-match.md) expects a JSON object. `canRetry` is `true`, but a retry reads the same stored output.
  * A [JSON Field Match](https://docs.galtea.ai/v5.13/concepts/metric/json-field-match.md) metric could not use the test case's expected output: it is not valid JSON, it is nested too deeply or holds a number too long to read, it is not a JSON object, or a [field rule](https://docs.galtea.ai/v5.13/concepts/metric/json-field-match.md#field-rules) in it is invalid. The error message names the field and the problem. Fix the test case and retry: a retry reads the current test case.
  * The judge model was unavailable.
  * The evaluator model's content safety filter blocked the request. This covers a metric that an LLM judge scores with a full prompt or a partial prompt. A retry uses the same model, so evaluate the session with a metric that uses a different evaluator model. For G-Eval based built-in metrics, a blocked request also ends as Skipped, and a filtered reply can end as Failed. Neither shows this advice. [Classifier](https://docs.galtea.ai/v5.13/concepts/metric/classifier.md) metrics do not detect a block yet.
  * A [classifier](https://docs.galtea.ai/v5.13/concepts/metric/classifier.md) question's **Does not apply** level was the most probable answer, or the input was too long for the classifier.
  * The [applicability check](https://docs.galtea.ai/v5.13/concepts/metric.md#applicability-check) found that the metric does not apply to the session. The message gives the probability that it applies and the metric's threshold. No credits are charged, and `canRetry` is `false`, because a retry asks the same question about the same session.

  Check the `canRetry` field to see whether the evaluation is eligible for a retry. A retry re-scores the output already stored on the session. When the cause is that output, send a new inference result instead.

  A credit shortage skips only the metrics the judge scores, and you can retry them. A self-hosted score you send in the same request is still stored. A mistake in the request itself, such as a metric of another organization or a malformed body, is not a skipped evaluation. A request for one session, and a batch request that names its metrics, answer with a 4xx error and create nothing. A batch request that takes its metrics from each session's specification lists the mistake for that session in its answer instead.
* **Cancelled**: The evaluation was deleted while it was still waiting to be scored, so the platform stopped it before it produced a score. Deleting the evaluation, or something it belongs to such as its session, its run or its metric, cancels every pending evaluation the delete removes. The same step deletes the evaluation, so it no longer appears in lists. Cancelling a run does not cancel its evaluations.
* **Outdated**: The evaluation was produced by a [Monitor](https://docs.galtea.ai/v5.13/concepts/product/monitor.md), and the session then reopened and gained a new turn. The score describes an older version of the conversation, so it no longer counts as current: charts and averages leave it out, and the monitor scores the session again once it closes. The row keeps its score and its reason, so you can still read what it found.

**Score** `Number`

The score assigned to the output by the metric's evaluation criteria. **Example**: 0.85

**Passed** `Boolean`

Whether the reported score reaches the metric's [pass threshold](https://docs.galtea.ai/v5.13/concepts/metric.md#pass-threshold). Null when the evaluation has no reported score, when it is outdated or deleted, or when it is a classifier result marked **Long input**. In the SDK it is `evaluation.passed`.

**Metric Pass Threshold** `Number`

The pass threshold of the metric revision that scored this evaluation, from 0 to 1. In the SDK it is `evaluation.metric_pass_threshold`.

**Reason** `Text`

The explanation of the score assigned to the output by the [metric's](https://docs.galtea.ai/v5.13/concepts/metric.md) evaluation criteria.

**Error** `Text`

The error message if the evaluation failed during processing.

**Retries Attempted** `Number`

The number of retry attempts made for this evaluation. Starts at 0 and increments each time the evaluation is retried. Used to track retry history and enforce retry limits.

**Can Retry** `Boolean` (default: false)

Indicates whether the evaluation is eligible for the retry operation. When `false`, it cannot be retried at all. When `true`, it can (only a `FAILED` or `SKIPPED` evaluation carries it), but a retry does not always help: `canRetry` is a permission, not a promise. A retry re-scores the output already stored on the session, so it succeeds for a temporary cause (an evaluation processing error, insufficient credits, a judge model that was unavailable) and skips again when the cause is the stored output itself. Defaults to `false`.

A row for a metric with `source: "self_hosted"` is never eligible for retry, whatever its status: a self-hosted score comes from you, not the judge, so there is nothing stored to re-send. Send the score again in a new evaluation instead.

**Launched By** `User`

The user who launched this evaluation, read-only and set by the API. For an evaluation launched through the SDK, this is the owner of the API key used. Null for a [Monitor](https://docs.galtea.ai/v5.13/concepts/product/monitor.md)-dispatched evaluation.

**Run ID** `Text`

The ID of the [Run](https://docs.galtea.ai/v5.13/concepts/product/run.md) that launched this evaluation (`runId` in the API). It is empty for an evaluation produced by a [Monitor](https://docs.galtea.ai/v5.13/concepts/product/monitor.md), because an evaluation carries a Run or a Monitor, never both. A retried evaluation keeps the Run it was created with.

**Product ID** `Text`

The ID of the [product](https://docs.galtea.ai/v5.13/concepts/product.md) whose session produced this evaluation, read-only and set by the API. Useful when a list spans several products, for example listing every evaluation for one metric.

**Judge Input Tokens** `Number`

The tokens the judge model received to score this evaluation, read-only and set by the API. It covers the metric prompt and the session content. Null when the judge model reported no usage, which is different from 0.

**Judge Output Tokens** `Number`

The tokens the judge model produced to score this evaluation, read-only and set by the API. The judge's total is `judgeInputTokens` plus `judgeOutputTokens`. Null when the judge model reported no usage.

**Judge Reasoning Tokens** `Number`

The part of `judgeOutputTokens` the judge model spent on reasoning, read-only and set by the API. It is a subset of the output tokens, so do not add it to the total. A judge model that does not reason reports 0, not null. Null means no reasoning count was recorded, which almost always means the whole measurement is missing and the other two fields are null too.

> [!NOTE]
>
> The API returns `creditsUsed` and the three judge token fields as `null` to a user without the `evaluation.readInference` permission, such as a Reader. See [Roles and permissions](https://docs.galtea.ai/v5.13/security/roles-and-permissions.md).

> [!NOTE]
>
> These token counts measure the judge model, not your AI product. They are not Galtea credits: Galtea charges a flat credit cost per evaluation. Use them to predict what the same judge costs on your own model provider. For a [classifier](https://docs.galtea.ai/v5.13/concepts/metric/classifier.md) metric, these token fields count Jev's tokens.

**Score Breakdown** `Object`

How a [classifier](https://docs.galtea.ai/v5.13/concepts/metric/classifier.md) metric reached its score, read-only and set by the API. Present only for `CLASSIFIER` metrics. The fields depend on the question type:

* `version`: the breakdown schema version, `2`.
* `model`: the classifier model that answered, such as `jev-1.13.0`.
* `type`: `yes_no` or `scale`.
* `value`: the score.
* `confidence`: the same value as the evaluation's **Confidence**.
* For `yes_no`: `pYes` (the probability of "yes") and `goodAnswer` (`yes` or `no`).
* For `scale`: `probabilities` (the probability of each level, keyed by its index as a string, worst level first, with the Does not apply level last when the question has one), `levelScores` (the score of each scored level, worst first), and `doesNotApply` (whether the question has a Does not apply level).

In the SDK it is `evaluation.score_breakdown`, with snake\_case keys (`p_yes`, `good_answer`, `level_scores`, `does_not_apply`).

**Confidence** `Number`

How sure the classifier was of its answer, from 0 to 1, read-only and set by the API. Present only for `CLASSIFIER` metrics, and null for every other source. Confidence values do not compare across question types. In the SDK it is `evaluation.confidence`.

**Long Input** `Boolean`

True for a `CLASSIFIER` evaluation whose input is longer than the classifier reads reliably (about 2,000 tokens) and that no reviewer has scored. The evaluation keeps its score, but analytics leave it out of pass rates and score averages. False for every other evaluation. In the SDK it is `evaluation.long_input`.

**Human Evaluator ID** `Text`

The ID of the human evaluator assigned to this evaluation, if applicable.

**Human Score** `Number`

The human-provided annotation score, separate from the AI-generated score. When a human annotates an AI-evaluated metric, this field stores their score without overwriting the original AI score. Once set, this score replaces the AI score in every reported number: analytics, the product score, coverage, and monitor results. A reviewer can remove the annotation again, which clears this field. Where the AI produced a score, every reported number goes back to it. Where the AI produced none, as on a failed, skipped or cancelled evaluation, the evaluation stops being reported at all. The removal is permanent: the score is deleted, not archived, so getting it back means annotating the evaluation again.

**Human Reason** `Text`

The human-provided annotation reason, separate from the AI-generated reason. When a human annotates an AI-evaluated metric, this field stores their reasoning without overwriting the original AI reason. Galtea reports this reason wherever it reports the human score. Leave it blank, and the reported reason is blank too; it never falls back to the AI reason. Removing the annotation clears this field with the human score. When the reviewer adds evidence from the conversation, the reason ends with an **Evidence from the conversation** footer that lists each quoted phrase and its note, and holds only that footer when no reason was typed.

**Human Evaluator Started At** `Text`

The timestamp when the human evaluator started reviewing this evaluation, if applicable.

**Human Evaluator Finished At** `Text`

The timestamp when the human evaluator submitted their evaluation or annotation.

## Related

> [card] [Concepts overview](https://docs.galtea.ai/v5.13/concepts/overview.md)
>
> How Galtea's concepts connect — diagram + per-entity quick reference.

> [card] [Dataset](https://docs.galtea.ai/v5.13/concepts/product/dataset.md)
>
> A dataset of test cases for evaluating product performance

> [card] [Test Case](https://docs.galtea.ai/v5.13/concepts/product/dataset/test-case.md)
>
> Each challenge in a dataset for evaluating product performance

> [card] [Session](https://docs.galtea.ai/v5.13/concepts/product/version/session.md)
>
> A full conversation between a user and an AI system.

> [card] [Metric](https://docs.galtea.ai/v5.13/concepts/metric.md)
>
> Ways to evaluate and score product performance

> [card] [Run](https://docs.galtea.ai/v5.13/concepts/product/run.md)
>
> The record of one evaluation launch


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.