> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Case

> A single challenge for evaluating product performance

> [!WARNING]
>
> You are reading the docs of Galtea 5.13. [Read this page in the latest version](https://docs.galtea.ai/concepts/product/dataset/test-case.md).

## What is a Test Case?

A test case in Galtea is a challenge designed to evaluate the performance of a particular [version](https://docs.galtea.ai/v5.13/concepts/product/version.md) of a [product](https://docs.galtea.ai/v5.13/concepts/product.md).

It represents a specific set of *inputs* that must be sent to the product's AI model to generate an output. It also can define an *expected output* that can be used to assess the product's capabilities.

> [!NOTE]
>
> Test cases are part of a [dataset](https://docs.galtea.ai/v5.13/concepts/product/dataset.md), so you'll need to create a dataset first. You can do this in the Galtea dashboard or using the SDK.

You can **create**, view and manage your test cases on the [Galtea dashboard](https://platform.galtea.ai/) or programmatically using the [Galtea SDK](https://docs.galtea.ai/v5.13/sdk/api/test-case/service.md). A dataset created with the *Add test cases manually* origin starts empty and is filled this way, one test case at a time.

## Using Test Cases in Evaluations

Test cases are used with [evaluations](https://docs.galtea.ai/v5.13/concepts/product/version/session/evaluation.md). Sessions are created automatically when running evaluations.

> [card] [Create an Evaluation](https://docs.galtea.ai/v5.13/concepts/product/version/session/evaluation.md)
>
> Learn how to use datasets in evaluations

## SDK Integration

The Galtea SDK allows you to create, view, and manage test cases programmatically.

> [card] [Test Case Service SDK](https://docs.galtea.ai/v5.13/sdk/api/test-case/service.md)
>
> Manage test cases using the Python SDK

## Test Case Properties

**Input** `Text` *required*

The input data used for inference on the LLM product's version. This is the question or prompt that will be sent to your AI model. **Example**: "What is the operating system of the Samsung Galaxy A8?"

When an [Input Schema](https://docs.galtea.ai/v5.13/concepts/product/endpoint-connection-input-schema.md) is defined on the endpoint connection, inputs become structured JSON objects with multiple typed fields instead of a single string.

An input can also carry uploaded files, for products that read documents. See [File inputs](#file-inputs) below.

### File inputs

A test case input can attach one or more files beside the text, or carry only files and no text at all. Use this when the thing you are testing is a document: a scanned contract, an invoice, a report.

With the SDK, name the local files and it uploads them for you:

```python theme={"system"}
test_case = galtea.test_cases.create(
    dataset_id=dataset.id,
    input="Summarize this contract",
    input_file_paths=["path/to/lease-agreement.pdf"],
)
```

Leave `input` out to create a test case that is only a document:

```python theme={"system"}
document_only = galtea.test_cases.create(
    dataset_id=dataset.id,
    input_file_paths=["path/to/lease-agreement.pdf"],
)
```

To have Galtea run the inference for you, name the files in your [endpoint connection template](https://docs.galtea.ai/v5.13/concepts/product/endpoint-connection-template-syntax.md#attached-files-and-audio). Read the files back from `test_case.input_files` instead when you run your own pipeline, and fetch the bytes with [`galtea.storage.download()`](https://docs.galtea.ai/v5.13/sdk/api/storage/download), passing the `AttachedFile` straight from `test_case.input_files`. [Evaluate Document Inputs](https://docs.galtea.ai/v5.13/sdk/tutorials/evaluate-document-inputs.md) walks that whole round trip, from the CSV to a scored answer.

To attach files to many test cases at once, add an `input_file_paths` column to your dataset CSV. Separate several paths with `;` or `|`. The SDK uploads each file and rewrites the row before the file is sent, so the column never reaches the platform and the paths stay private:

```csv theme={"system"}
input,expected_output,input_file_paths
Summarize this contract,A 12-month lease at 900 EUR,lease-agreement.pdf
,Total due 1450 EUR,invoice.pdf;annex.pdf
```

A cell may also hold a storage URI from an earlier upload instead of a local path. That file is attached as it is, so one document can serve several datasets without being uploaded again.

```python theme={"system"}
dataset = galtea.datasets.create(
    name=f"document-dataset-{run_identifier}",
    type="ACCURACY",
    product_id=product_id,
    dataset_file_path="path/to/document_dataset.csv",
)
```

Through the API instead, upload each file first and send the storage URI the upload returned. The files ride the input as a `content` array:

```json theme={"system"}
{
  "user_message": "Summarize this contract",
  "content": [
    { "type": "file", "uri": "s3://bucket/files/org_123/2f6c….pdf" }
  ]
}
```

`mimeType` is filled from the stored file. `filename` is filled too on Galtea Cloud, so the download offers the name you uploaded rather than a random id. A self-hosted deployment on Azure Blob Storage cannot keep that name, so send `filename` yourself there. Your own values override the filled ones, with one exception: `sizeBytes` comes back measured from the stored file, and a value you send for it is ignored. The 20 MB limit below is read from that measurement. The SDK sends `filename` and `mimeType`, read from the local file, so they are right on every deployment.

You can also attach a file from the dashboard, on the test case create and update forms. An attached image shows a thumbnail; click it to view the image at full size. If you have no dataset to add the test case to, create one with the [manual origin](https://docs.galtea.ai/v5.13/concepts/product/dataset.md#dataset-origin).

These limits apply, and each one is refused with a clear error rather than failing later:

* **20 MB in total, and at most 20 files, per test case input.** There is no per-file size limit.
* **These file types only**: `pdf`, `docx`, `xlsx`, `pptx`, `rtf`, `png`, `jpg`, `jpeg`, `tiff`, `tif`, `bmp`, `webp`, `heic`, `heif`, `gif`, `txt`, `csv`, `md`, `html`, `xml`, `json`, `eml`. The type is read from the file name. Audio belongs on a voice test case instead.
* **Single turn only.** The document is the first turn, so a scenario that would keep a simulated user going has nothing to say after it.
* **Galtea runs the inference only when your template asks for the file.** Add `{{ input_files[0].url }}` or `{{ input_files[0].base64 }}` to the [endpoint connection template](https://docs.galtea.ai/v5.13/concepts/product/endpoint-connection-template-syntax.md#attached-files-and-audio), and Galtea delivers each attachment to your endpoint. Without it, the run is refused rather than sent without the document.

**Evaluators cannot read files yet, so what the judge is given is the text of your conversation turns, with every file stripped out.** A metric that reads the input is skipped when no turn is left with text of its own, and it costs no credits. Text sent beside the file does not count: it is usually a prompt about the document, so scoring it would give you a confident number about a document the judge never read. A file test case runs as a single turn, so a run of one always skips those metrics.

To have the document scored, log the text your pipeline extracted as the turn input, without the file. A later turn that carries text of its own is also enough, so one attachment in a long session does not stop the metric. Metrics that read `conversation_turns` are never skipped for a file: the turns are scored as you logged them. You can also score the run with a metric that reads only the output, such as a JSON Field Match metric comparing your output field by field. See [Troubleshooting skipped evaluations](https://docs.galtea.ai/v5.13/concepts/metric/evaluation-parameters.md#troubleshooting-skipped-evaluations).

Editing a test case that has files follows the usual [revision](https://docs.galtea.ai/v5.13/concepts/product/dataset/test-case.md#revisions) behaviour: a revision may add or replace files, and the superseded revision keeps its own. Send `input` as the whole object, including the files you want to keep. **The edit must leave at least one file.** Plain text is refused, and so is an object with an empty `content`, because either one would turn a document test case into a text one. To get the text-only test case, create it and delete this one.

**Expected Output** `Text`

The expected output for the evaluation. This represents the ideal response you want your AI model to provide. **Example**: "The operating system of the Samsung Galaxy A8 is Android 8.1." For a JSON object, a field can hold a [field rule](https://docs.galtea.ai/v5.13/concepts/metric/json-field-match.md#field-rules) (`$norm`, `$regex`, `$num` or `$date`) instead of a value. Rules work only in the four JSON Field Match metrics. Every other metric reads a rule as plain JSON text.

**Context** `Text`

Test case-specific context that provides context to the model. It can be the past conversation or any other relevant information. Do not mistake for the system prompt/few-shot examples; those should be defined in the product's [version](https://docs.galtea.ai/v5.13/concepts/product/version.md) unless they change on an interaction basis.

**Variant** `Text`

A label that helps categorize the test case (e.g., "original", "paraphrased", "incorrect"). This corresponds to the `tag` column in an uploaded dataset CSV file and is used for organizing and filtering test cases.

**Source** `Text`

The original source text used to create the test case. This can be helpful for tracing where test cases originated from, especially when generating test cases from documentation or other reference materials.

**Source File** `Text`

The name of the file from which the test case was extracted. This is populated when test cases are generated from uploaded documents (e.g., ZIP or PDF files containing ground truth data).

**Human Reviewed** `Boolean`

Indicates if a test case has been manually reviewed and approved by a user. This is useful for tracking which test cases are considered reliable and validated. The system stores the ID of the user who reviewed the test case, upon creation or update, the one who created or updated the test case is set as the reviewer.

**Is Augmented** `Boolean`

Indicates if a test case was generated through the augmentation process. Test cases created via the generate-from-few-shots augmentation feature will have this flag set to `true`, while manually created test cases will have it set to `false`.

**User Score** `Integer`

User vote for the test case quality. Possible values are `1` (upvote), `-1` (downvote), or `0` (unreviewed).

**User Score Reason** `Text`

A justification for the user score, providing context on why a test case was downvoted (or upvoted).

**Confidence** `Float`

The confidence level of the test case.

**Confidence Reason** `Text`

A justification for the confidence level of the test case.

**Gender** `Text`

The persona's gender, either `MALE` or `FEMALE`. In voice (phone) evaluations it selects the male or female synthesized caller voice; text evaluations ignore it, and when unset the voice uses the default for the language. For scenario test cases produced by the generator, it is inferred from the user persona.

**Background Noise Profile** `Enum`

Background noise mixed into the simulated caller's audio when this test case runs over a [phone connection](https://docs.galtea.ai/v5.13/concepts/product/phone-connection.md). Only applies to voice datasets; has no effect otherwise.
Possible values:

* Clip-backed: `office`, `street`, `car`
* Synthetic: `white`, `pink`

If not set explicitly, a test case generated by Galtea or ingested from a CSV file inherits its parent [dataset](https://docs.galtea.ai/v5.13/concepts/product/dataset.md)'s background noise at creation time. Must be set together with **Background Noise Level**.

**Background Noise Level** `Enum`

How loud the background noise is relative to the caller's speech.
Possible values: `light`, `medium`, `heavy`. Must be set together with **Background Noise Profile**.

## Scenario-Based Test Case Properties

The following properties apply specifically to test cases of type `BEHAVIOR` used with the [Conversation Simulator](https://docs.galtea.ai/v5.13/concepts/product/dataset/test-case/conversation-simulator.md):

**Goal** `Text`

The objective the simulated user is trying to achieve in a conversation scenario. This defines what the simulated user wants to accomplish during the interaction. **Example**: "Book a flight to New York"

**User Persona** `Text`

The personality of the simulated user for a conversation scenario. This shapes how the simulated user will behave and communicate during the conversation. **Example**: "A busy professional who values efficiency"

**Scenario** `Text`

A description of the specific scenario or situation in which the conversation takes place. This provides additional context for the interaction. **Example**: "Flight booking scenario"

**Input** `Text`

The first message from the simulated user that starts the conversation. If not provided, the simulator will generate an appropriate opening message based on the goal and persona. **Example**: "I need to book a flight"

**Stopping Criterias** `Text`

Conditions that determine when the conversation should end, separated by `;` or `|`. The conversation will stop when any of these criteria are met. **Example**: "Booking confirmed|Unable to fulfill request"

**Live Turns to Run** `Number`

The maximum number of turns the simulator runs, on top of the predefined conversation if the test case has one. This prevents conversations from running indefinitely. A 4-turn predefined conversation with **Live Turns to Run** 1 runs one new turn and the session ends with 5. In the CSV file and in the SDK the field is named `max_iterations`. **Example**: 10

## Starting a Scenario from a Conversation

A scenario test case can start from a conversation that already happened. The simulator then continues from that point instead of opening the dialogue itself. Give the conversation in one of two ways, never both in the same request:

**conversationHistory** `Array`

The turns to store, written out. Each turn holds the user message, the agent answer, or both. Galtea stores them as a **seed session** and returns that session's id in `seedSessionId`. The array itself is never echoed back: read the predefined conversation through the seed session. Sending the array again replaces the predefined conversation. In the SDK the parameter is named `conversation_history`.

**seedSessionId** `Text`

The id of a session to copy the conversation from. The copy takes that session's turns **and its traces**, so a seed built this way carries the retrieval context and tool calls that the written-out form cannot. The session must belong to the same product as the test case, and every turn in it must be finished. Removed turns at the end of the session are left out of the copy, so a session the agent ended can be copied. A removed turn in the middle of the session is refused.

In the SDK the parameter and the test case field are both named `seed_session_id`.

On a response, this field names the test case's own seed session, not the session you copied. Sending a test case body back unchanged is therefore safe: the API reads its own seed id as "leave the conversation alone".

Both fields are only accepted on `SCENARIOS` test cases whose product has at least one version.

### How the run uses it

When you run the test case, Galtea copies the predefined conversation into the run's own session before
your product answers anything. Your product therefore sees that conversation as history and answers
the turn after it. The predefined conversation itself is never changed, so every run of the test case
starts from the same point.

This works the same way whether Galtea calls your product for you, or your own code drives the
conversation with `galtea.simulator.simulate()`, `galtea.evaluations.run(agent=...)` or
`galtea sessions generate-message`.

Two rules follow:

* **Live Turns to Run counts only the new turns.** The copied turns are already answered, so they widen
  the budget instead of spending it.
* **A seeded test case needs `userPersona`, `goal` and `scenario`.** The run is never on its first
  turn, so the simulator writes every new user message and cannot fall back to the test case
  `input`.

Start the run on a session that holds no turn. Galtea refuses a session that already holds turns of
its own, because the conversation can no longer be placed in front of them. That is why
`simulate(agent_goes_first=True)` cannot be used with a predefined conversation: it records your agent's
opening turn first.

## Revisions

Editing a test case's evaluation content (for example its input, expected output, or context) creates a new **revision** instead of changing the existing row. This keeps your evaluation history accurate:

* **Past evaluations stay pinned** to the exact content that produced them. An old score always reflects the test case as it was when that evaluation ran.
* **The list shows the latest revision** of each test case by default. To also see superseded revisions, open **Filters** and turn on **Include legacy**. They are marked with a `Legacy` status.
* **The details page links back** with an **Evolved from** link to the revision it was created from.
* **Outdated evaluations are flagged.** When an evaluation ran against a revision that has since been replaced, it shows an outdated marker next to its score. Run a new evaluation against the current revision for an up-to-date score.
* **Replay past evaluations in one step.** Right after an edit creates a new revision, a **Re-evaluate with this Test Case** dialog lists the products whose earlier evaluations of this test case can be re-run, all selected by default. Confirming generates fresh responses from your product for the new revision and evaluates them with the same metrics as before. The dialog only appears when at least one product has previous evaluations to replay.

Only edits to content used during evaluation create a revision. Annotation edits (such as your quality vote or marking a test case reviewed) change the row in place and do not create a revision.

## Related

> [card] [Concepts overview](https://docs.galtea.ai/v5.13/concepts/overview.md)
>
> How Galtea's concepts connect — diagram + per-entity quick reference.

> [card] [Input Schema](https://docs.galtea.ai/v5.13/concepts/product/endpoint-connection-input-schema.md)
>
> Define structured, multi-field inputs for test case generation.

> [card] [Conversation Simulator](https://docs.galtea.ai/v5.13/concepts/product/dataset/test-case/conversation-simulator.md)
>
> Run multi-turn conversations against scenario-based test cases.

> [card] [Dataset](https://docs.galtea.ai/v5.13/concepts/product/dataset.md)
>
> The datasets that contain these test cases.

> [!NOTE]
>
> For Scenarios test cases, only `Goal` and `User Persona` are mandatory. All other scenario properties are optional but can help create more realistic and controlled conversation simulations.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.