> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Case

> A single challenge for evaluating product performance

> [!WARNING]
>
> You are reading the docs of Galtea 5.16. [Read this page in the latest version](https://docs.galtea.ai/concepts/product/dataset/test-case.md).

## What is a Test Case?

A test case in Galtea is a challenge designed to evaluate the performance of a particular [version](https://docs.galtea.ai/v5.16/concepts/product/version.md) of a [product](https://docs.galtea.ai/v5.16/concepts/product.md).

It represents a specific set of *inputs* that must be sent to the product's AI model to generate an output. It also can define an *expected output* that can be used to assess the product's capabilities.

> [!NOTE]
>
> Test cases are part of a [dataset](https://docs.galtea.ai/v5.16/concepts/product/dataset.md), so you'll need to create a dataset first. You can do this in the Galtea dashboard or using the SDK.

You can **create**, view and manage your test cases on the [Galtea dashboard](https://platform.galtea.ai/) or programmatically using the [Galtea SDK](https://docs.galtea.ai/v5.16/sdk/api/test-case/service.md). A dataset created with the *Add test cases manually* origin starts empty and is filled this way, one test case at a time.

## Using Test Cases in Evaluations

Test cases are used with [evaluations](https://docs.galtea.ai/v5.16/concepts/product/version/session/evaluation.md). Sessions are created automatically when running evaluations.

> [card] [Create an Evaluation](https://docs.galtea.ai/v5.16/concepts/product/version/session/evaluation.md)
>
> Learn how to use datasets in evaluations

## SDK Integration

The Galtea SDK allows you to create, view, and manage test cases programmatically.

> [card] [Test Case Service SDK](https://docs.galtea.ai/v5.16/sdk/api/test-case/service.md)
>
> Manage test cases using the Python SDK

## Test Case Properties

**Input** `Text` *required*

The input data used for inference on the LLM product's version. This is the question or prompt that will be sent to your AI model. **Example**: "What is the operating system of the Samsung Galaxy A8?"

When an [Input Schema](https://docs.galtea.ai/v5.16/concepts/product/endpoint-connection-input-schema.md) is defined on the endpoint connection, inputs become structured JSON objects with multiple typed fields instead of a single string.

An input can also carry uploaded files, for products that read documents. See [File inputs](#file-inputs) below.

### File inputs

A test case input can attach one or more files beside the text, or carry only files and no text at all. Use this when the thing you are testing is a document: a scanned contract, an invoice, a report.

With the SDK, name the local files and it uploads them for you:

```python theme={"system"}
test_case = galtea.test_cases.create(
    dataset_id=dataset.id,
    input="Summarize this contract",
    input_file_paths=["path/to/lease-agreement.pdf"],
)
```

Leave `input` out to create a test case that is only a document:

```python theme={"system"}
document_only = galtea.test_cases.create(
    dataset_id=dataset.id,
    input_file_paths=["path/to/lease-agreement.pdf"],
)
```

To have Galtea run the inference for you, name the files in your [endpoint connection template](https://docs.galtea.ai/v5.16/concepts/product/endpoint-connection-template-syntax.md#attached-files-and-audio). Read the files back from `test_case.input_files` instead when you run your own pipeline, and fetch the bytes with [`galtea.storage.download()`](https://docs.galtea.ai/v5.16/sdk/api/storage/download), passing the `AttachedFile` straight from `test_case.input_files`. A [file part of your own](https://docs.galtea.ai/v5.16/concepts/product/version/session/content-parts.md#your-own-parts) is listed there too; fetch it with your own client. [Evaluate Document Inputs](https://docs.galtea.ai/v5.16/sdk/tutorials/evaluate-document-inputs.md) walks that whole round trip, from the CSV to a scored answer.

To attach files to many test cases at once, add an `input_file_paths` column to your dataset CSV. Separate several paths with `;` or `|`. The SDK uploads each file and rewrites the row before the file is sent, so the column never reaches the platform and the paths stay private:

```csv theme={"system"}
input,expected_output,input_file_paths
Summarize this contract,A 12-month lease at 900 EUR,lease-agreement.pdf
,Total due 1450 EUR,invoice.pdf;annex.pdf
```

A cell may also hold a Galtea file reference from an earlier upload instead of a local path. That file is attached as it is, so one document can serve several datasets without being uploaded again. The SDK refuses any other URL in the cell, before it uploads anything.

```python theme={"system"}
dataset = galtea.datasets.create(
    name=f"document-dataset-{run_identifier}",
    type="ACCURACY",
    product_id=product_id,
    dataset_file_path="path/to/document_dataset.csv",
)
```

Through the API instead, upload each file first and send the storage URI the upload returned. The files ride the input as a `content` array:

```json theme={"system"}
{
  "user_message": "Summarize this contract",
  "content": [
    { "type": "file", "uri": "s3://bucket/files/org_123/2f6c0a9e41b84d7f9c3e5a1b2d4f6e80.pdf" }
  ]
}
```

Galtea reads each element of `content` on its own. A Galtea file is a `file` part whose `uri` names the key an upload returned. Galtea checks it and stores it without its upload signature. Every other element is yours and is stored as sent, such as a file URL on your own host, an `image_url` part or a `file` part with an OpenAI `file_id`. A Galtea file of another organization, or one that no longer exists, is refused with a `400`. See [Content Parts](https://docs.galtea.ai/v5.16/concepts/product/version/session/content-parts.md).

`mimeType` is filled from the stored file. `filename` is filled too on Galtea Cloud, so the download offers the name you uploaded rather than a random id. A self-hosted deployment on Azure Blob Storage cannot keep that name, so send `filename` yourself there. Your own values override the filled ones, with one exception: `sizeBytes` comes back measured from the stored file, and a value you send for it is ignored. The 20 MB limit below is read from that measurement. The SDK sends `filename` and `mimeType`, read from the local file, so they are right on every deployment.

You can also attach a file from the dashboard, on the test case create and update forms. An attached image shows a thumbnail; click it to view the image at full size. If you have no dataset to add the test case to, create one with the [manual origin](https://docs.galtea.ai/v5.16/concepts/product/dataset.md#dataset-origin).

These limits apply to Galtea files, and each one is refused with a clear error rather than failing later:

* **20 MB in total, and at most 20 files, per test case input.** There is no per-file size limit.
* **These file types only**: `pdf`, `docx`, `xlsx`, `pptx`, `rtf`, `png`, `jpg`, `jpeg`, `tiff`, `tif`, `bmp`, `webp`, `heic`, `heif`, `gif`, `txt`, `csv`, `md`, `html`, `xml`, `json`, `eml`. The type is read from the file name. Audio belongs on a voice test case instead.
* **Single turn only.** The document is the first turn, so a scenario that would keep a simulated user going has nothing to say after it.
* **Galtea runs the inference only when your template asks for the file.** Add `{{ input_files[0].url }}` or `{{ input_files[0].base64 }}` to the [endpoint connection template](https://docs.galtea.ai/v5.16/concepts/product/endpoint-connection-template-syntax.md#attached-files-and-audio), and Galtea delivers each uploaded file to your endpoint. Your own `file` parts, such as a URL on your own host, are not delivered (see [Content Parts](https://docs.galtea.ai/v5.16/concepts/product/version/session/content-parts.md#your-own-parts)). Without it, the run is refused rather than sent without the document.

**Inline file bytes never reach a judge.** A file sent inline in a test case or a trace, as a `data:` URI of any length, reaches an LLM-as-a-judge in one of two ways:

* **A text file**, such as CSV, JSON or plain text, arrives as its text, between `[inline file: <type>, <size> bytes]` and `[end of inline file]`. Galtea decides it is text from its bytes, not its declared type, and reads the `charset` it declares, UTF-8 by default. Inside JSON, including JSON written inside a reply, it arrives escaped as a JSON string. It arrives as the marker below instead when that escaping would make it over four times longer.
* **Any other file**, such as an image or a compressed PDF, arrives as `[inline bytes dropped: <type>, <size> bytes]`.

Metrics that compare values exactly still read the stored value.

A file test case runs as a single turn, so a metric that reads the input reads the file.

> [!NOTE]
>
> **A metric scores a file only when its judge model reads that kind of file.** Only a [Galtea file](https://docs.galtea.ai/v5.16/concepts/product/version/session/content-parts.md#galtea-files), one you uploaded to Galtea, is sent to a judge. A file on your own host is stored as sent and never reaches a judge, so it never skips an evaluation: the judge reads the text beside it. The kinds a judge reads depend on its model, and the model picker of the metric form shows them for each model. A judge is sent PNG, JPEG, WebP, GIF and PDF files only. The other accepted types are stored, and a Galtea input file of those types still reaches your endpoint, but no judge reads them. The rule is the same for the input and the output, and a metric that reads `conversation_turns` reads both sides of every turn.
>
> Before an evaluation runs, Galtea checks every Galtea file on each side the metric reads. One file the judge would not read skips the evaluation, whatever text sits beside it. A skipped evaluation costs no credits, and its reason names the file, its turn and what refused it:
>
> * The judge model cannot read that kind of file.
> * The model catalog does not say whether the judge model reads it.
> * The metric names no judge model.
> * The metric is DeepEval-based: those judges are sent text only.
> * The metric is deterministic: it compares text without a model.
> * The metric is a [classifier](https://docs.galtea.ai/v5.16/concepts/metric/classifier.md) metric: it is sent text only.
> * The file is a type no judge reads, such as DOCX, CSV or TIFF, or an audio file: a judge reads audio only through its transcript.
>
> When the evaluator then reads a Galtea file to send it, it skips the evaluation if the file no longer exists in storage, is empty, or is larger than the 20 MB a judge can receive. The reason names the file.
>
> Audio input with no transcript is a separate case. A metric that reads the input is skipped when no turn of the session has input text or a Galtea input file, and a turn's audio carries no transcript. Text or a Galtea file on another turn is enough, and the reason, `Evaluators read an audio input through its transcript, and no turn carries one.`, names no file.
>
> Human evaluation and self-hosted metrics are never skipped for a file or for audio. On deployments that do not send files to the judge, the evaluator also skips a metric whose judge would read the file, and the reason names the file.
>
> To have the content scored, pick a metric whose judge model reads that kind of file, or log the text your pipeline extracted as the turn input or output, without the file. See [Troubleshooting skipped evaluations](https://docs.galtea.ai/v5.16/concepts/metric/evaluation-parameters.md#troubleshooting-skipped-evaluations).

You can also score the run with a metric that reads only the output, such as a JSON Field Match metric comparing your output field by field.

Editing a test case that has files follows the usual [revision](https://docs.galtea.ai/v5.16/concepts/product/dataset/test-case.md#revisions) behaviour: a revision may add or replace files, and the superseded revision keeps its own. Send `input` as the whole object, including the files you want to keep. **The edit must leave at least one `file` part.** Your own `file` parts count; an `image_url` part does not. Plain text is refused, and so is an object with an empty `content`, because either one would turn a document test case into a text one. To get the text-only test case, create it and delete this one. A Galtea file you keep unchanged is not read from storage again, so the edit succeeds even when the file was deleted after you attached it.

**Expected Output** `Text`

The expected output for the evaluation. This represents the ideal response you want your AI model to provide. **Example**: "The operating system of the Samsung Galaxy A8 is Android 8.1." For a JSON object, a field can hold a [field rule](https://docs.galtea.ai/v5.16/concepts/metric/json-field-match.md#field-rules) (`$norm`, `$regex`, `$num` or `$date`) instead of a value. Rules work only in the four JSON Field Match metrics. Every other metric reads a rule as plain JSON text.

**Context** `Text`

Test case-specific context that provides context to the model. It can be the past conversation or any other relevant information. Do not mistake for the system prompt/few-shot examples; those should be defined in the product's [version](https://docs.galtea.ai/v5.16/concepts/product/version.md) unless they change on an interaction basis.

**Variant** `Text`

A label that helps categorize the test case (e.g., "original", "paraphrased", "incorrect"). This corresponds to the `tag` column in an uploaded dataset CSV file and is used for organizing and filtering test cases.

**Source** `Text`

The original source text used to create the test case. This can be helpful for tracing where test cases originated from, especially when generating test cases from documentation or other reference materials.

**Source File** `Text`

The name of the file from which the test case was extracted. This is populated when test cases are generated from uploaded documents (e.g., ZIP or PDF files containing ground truth data).

**Human Reviewed** `Boolean`

Indicates if a test case has been manually reviewed and approved by a user. This is useful for tracking which test cases are considered reliable and validated. The system stores the ID of the user who reviewed the test case, upon creation or update, the one who created or updated the test case is set as the reviewer.

**Is Augmented** `Boolean`

Indicates if a test case was generated through the augmentation process. Test cases created via the generate-from-few-shots augmentation feature will have this flag set to `true`, while manually created test cases will have it set to `false`.

**User Score** `Integer`

User vote for the test case quality. Possible values are `1` (upvote), `-1` (downvote), or `0` (unreviewed).

**User Score Reason** `Text`

A justification for the user score, providing context on why a test case was downvoted (or upvoted).

**Confidence** `Float`

The confidence level of the test case.

**Confidence Reason** `Text`

A justification for the confidence level of the test case.

**Gender** `Text`

The persona's gender, either `MALE` or `FEMALE`. In voice (phone) evaluations it selects the male or female synthesized caller voice; text evaluations ignore it, and when unset the voice uses the default for the language. For scenario test cases produced by the generator, it is inferred from the user persona.

**Background Noise Profile** `Enum`

Background noise mixed into the simulated caller's audio when this test case runs over a [phone connection](https://docs.galtea.ai/v5.16/concepts/product/phone-connection.md). Only applies to voice datasets; has no effect otherwise.
Possible values:

* Clip-backed: `office`, `street`, `car`
* Synthetic: `white`, `pink`

If not set explicitly, a test case generated by Galtea or ingested from a CSV file inherits its parent [dataset](https://docs.galtea.ai/v5.16/concepts/product/dataset.md)'s background noise at creation time. Must be set together with **Background Noise Level**.

**Background Noise Level** `Enum`

How loud the background noise is relative to the caller's speech.
Possible values: `light`, `medium`, `heavy`. Must be set together with **Background Noise Profile**.

## Scenario-Based Test Case Properties

The following properties apply specifically to test cases of type `BEHAVIOR` used with the [Conversation Simulator](https://docs.galtea.ai/v5.16/concepts/product/dataset/test-case/conversation-simulator.md):

**Goal** `Text`

The objective the simulated user is trying to achieve in a conversation scenario. This defines what the simulated user wants to accomplish during the interaction. **Example**: "Book a flight to New York"

**User Persona** `Text`

The personality of the simulated user for a conversation scenario. This shapes how the simulated user will behave and communicate during the conversation. **Example**: "A busy professional who values efficiency"

**Scenario** `Text`

A description of the specific scenario or situation in which the conversation takes place. This provides additional context for the interaction. **Example**: "Flight booking scenario"

**Input** `Text`

The first message from the simulated user that starts the conversation. If not provided, the simulator will generate an appropriate opening message based on the goal and persona. **Example**: "I need to book a flight"

**Stopping Criterias** `Text`

Conditions that determine when the conversation should end, separated by `;` or `|`. The conversation will stop when any of these criteria are met. **Example**: "Booking confirmed|Unable to fulfill request"

**Live Turns to Run** `Number`

The maximum number of turns the simulator runs, on top of the predefined conversation if the test case has one. This prevents conversations from running indefinitely. A 4-turn predefined conversation with **Live Turns to Run** 1 runs one new turn and the session ends with 5. In the CSV file and in the SDK the field is named `max_iterations`. **Example**: 10

## Starting a Scenario from a Conversation

A scenario test case can start from a conversation that already happened. The simulator then continues from that point instead of opening the dialogue itself. Give the conversation in one of two ways, never both in the same request:

**conversationHistory** `Array`

The turns to store, written out. Each turn holds the user message, the agent answer, or both. Galtea stores them as a **seed session** and returns that session's id in `seedSessionId`. The array itself is never echoed back: read the predefined conversation through the seed session. Sending the array again replaces the predefined conversation. In the SDK the parameter is named `conversation_history`.

**seedSessionId** `Text`

The id of a session to copy the conversation from. The copy takes that session's turns **and its traces**, so a seed built this way carries the retrieval context and tool calls that the written-out form cannot. The session must belong to the same product as the test case, and every turn in it must be finished. Removed turns at the end of the session are left out of the copy, so a session the agent ended can be copied. A removed turn in the middle of the session is refused.

In the SDK the parameter and the test case field are both named `seed_session_id`.

On a response, this field names the test case's own seed session, not the session you copied. Sending a test case body back unchanged is therefore safe: the API reads its own seed id as "leave the conversation alone".

Both fields are only accepted on `SCENARIOS` test cases whose product has at least one version.

### How the run uses it

When you run the test case, Galtea copies the predefined conversation into the run's own session before
your product answers anything. Your product therefore sees that conversation as history and answers
the turn after it. The predefined conversation itself is never changed, so every run of the test case
starts from the same point.

This works the same way whether Galtea calls your product for you, or your own code drives the
conversation with `galtea.simulator.simulate()`, `galtea.evaluations.run(agent=...)` or
`galtea sessions generate-message`.

Two rules follow:

* **Live Turns to Run counts only the new turns.** The copied turns are already answered, so they widen
  the budget instead of spending it.
* **A seeded test case needs `userPersona`, `goal` and `scenario`.** The run is never on its first
  turn, so the simulator writes every new user message and cannot fall back to the test case
  `input`.

Start the run on a session that holds no turn. Galtea refuses a session that already holds turns of
its own, because the conversation can no longer be placed in front of them. That is why
`simulate(agent_goes_first=True)` cannot be used with a predefined conversation: it records your agent's
opening turn first.

## Revisions

Editing a test case's evaluation content (for example its input, expected output, or context) creates a new **revision** instead of changing the existing row. This keeps your evaluation history accurate:

* **Past evaluations stay pinned** to the exact content that produced them. An old score always reflects the test case as it was when that evaluation ran.
* **The list shows the latest revision** of each test case by default. To also see superseded revisions, open **Filters** and turn on **Include legacy**. They are marked with a `Legacy` status.
* **The details page links back** with an **Evolved from** link to the revision it was created from.
* **Outdated evaluations are flagged.** When an evaluation ran against a revision that has since been replaced, it shows an outdated marker next to its score. Run a new evaluation against the current revision for an up-to-date score.
* **Replay past evaluations in one step.** Right after an edit creates a new revision, a **Re-evaluate with this Test Case** dialog lists the products whose earlier evaluations of this test case can be re-run, all selected by default. Confirming generates fresh responses from your product for the new revision and evaluates them with the same metrics as before. The dialog only appears when at least one product has previous evaluations to replay.

Only edits to content used during evaluation create a revision. Annotation edits (such as your quality vote or marking a test case reviewed) change the row in place and do not create a revision.

## Related

> [card] [Concepts overview](https://docs.galtea.ai/v5.16/concepts/overview.md)
>
> How Galtea's concepts connect — diagram + per-entity quick reference.

> [card] [Input Schema](https://docs.galtea.ai/v5.16/concepts/product/endpoint-connection-input-schema.md)
>
> Define structured, multi-field inputs for test case generation.

> [card] [Conversation Simulator](https://docs.galtea.ai/v5.16/concepts/product/dataset/test-case/conversation-simulator.md)
>
> Run multi-turn conversations against scenario-based test cases.

> [card] [Dataset](https://docs.galtea.ai/v5.16/concepts/product/dataset.md)
>
> The datasets that contain these test cases.

> [!NOTE]
>
> For Scenarios test cases, only `Goal` and `User Persona` are mandatory. All other scenario properties are optional but can help create more realistic and controlled conversation simulations.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.