Create Test Case

Returns

Returns a TestCase object for the given parameters.

Example

test_case = galtea.test_cases.create(
    dataset_id=dataset_id,
    input="What is the capital of France?",
    expected_output="Paris",
    context="Geography facts",
    variant="original",
)

Starting from a predefined conversation

Write the turns out, or copy them from a session that already holds them. See Starting a Scenario from a Conversation.

conversation_history = [
    {"role": "user", "content": "Hi, I am checking what this analytics workspace can do."},
    {"role": "assistant", "content": "I can help with dashboards, reports and data sources."},
    {"role": "user", "content": "Does it handle orders and payments too?"},
    {"role": "assistant", "content": "No. It reports on your data, it does not process payments."},
]

test_case = galtea.test_cases.create(
    dataset_id=dataset.id,
    user_persona="A team lead evaluating the product",
    goal="Get a clear answer about what the product does and does not cover",
    scenario="The user asks about an unrelated service",
    conversation_history=conversation_history,
    # Turns the simulator runs after the predefined conversation.
    max_iterations=2,
)

reused = galtea.test_cases.create(
    dataset_id=dataset.id,
    user_persona="A team lead evaluating the product",
    goal="Ask for a feature the product does not have",
    scenario="The user asks about an unrelated service",
    seed_session_id=test_case.seed_session_id,
    max_iterations=2,
)

Parameters

string

required

The ID of the dataset you want to create the test case for.

Deprecated alias: test_id.

string

The input for the test case. For Accuracy and Safety datasets, this is the question or prompt you want to send to your model for evaluation. For Behavior datasets, this is the first user message to use in the conversation (bypassing the simulator for the first turn).

list[string]

Files to attach to the input, for a product that reads documents. Each entry is a local path the SDK uploads, or a storage URI from an earlier upload, which is attached without uploading the file again. input stays optional, so a test case may hold files and no text at all. Galtea sends each attachment to your endpoint when the endpoint connection template names input_files.At most 20 files totalling 20 MB, of these types: pdf, docx, xlsx, pptx, rtf, png, jpg, jpeg, tiff, tif, bmp, webp, heic, heif, gif, txt, csv, md, html, xml, json, eml. To attach one already-uploaded file to several test cases, use Upload Input File instead.

Evaluators cannot read files yet, so the judge is given the text of your conversation turns with the files stripped out. A metric that reads the input is skipped when no turn is left with text of its own, which is what a single-turn document run always looks like. This is true for both run styles: your own pipeline, and Galtea calling your endpoint. Log the text your pipeline extracted as the turn input, without the file, to have it scored, or use a metric that reads only the output. See File inputs.

string

The expected output for the input given. This is the correct answer or response you want to evaluate against.

list[string]

The expected tools for the input given. This is the list of tools that the model should use to generate the expected output.

string

The context for the test case. This is the additional information that can help the model generate the expected output.

string

An optional variant to categorize or label the test case (e.g., “original”, “paraphrased”).

string

The ID of the user who has reviewed and validated this test case.

string

Language for the test case. Accepts a BCP-47 language tag: a bare ISO 639-1 code ("es") or a code with a region subtag ("es-MX", "en-GB"). The region subtag selects the regional variety in the simulated user’s messages and in voice synthesis (the accent the simulated caller speaks with); generated test text keeps the base language. The value is stored as the full tag. A bare code for ca, de, es, eu, fr, gl, it, nl or pt defaults to that language’s European variety in the simulated user’s messages only (pt → pt-PT); an explicit region always overrides it, and neither voice synthesis nor the stored value is affected.

string

Persona gender, "MALE" or "FEMALE", for voice evaluations: it selects the male or female synthesized voice for the simulated caller. A voice pinned for the call’s locale takes precedence. Text evaluations ignore it. When unset, the voice uses the default for the language.

string

Background noise mixed into the simulated caller’s audio for voice evaluations run over a phone connection. Ignored for test cases that don’t run over a phone connection.

Clip-backed: office, street, car
Synthetic: white, pink

This method creates the test case directly, so it does not inherit the parent dataset’s background noise — omitting this parameter leaves the test case with no noise. (Test cases generated by Galtea from a ground truth file or a CSV upload do inherit the dataset’s noise automatically; this only applies to cases created one-by-one through this method.)Must be set together with background_noise_level — passing only one of the two is rejected by the API.

string

How loud the background noise is relative to the caller’s speech, as a fixed signal-to-noise ratio per level. One of light, medium, or heavy. Must be set together with background_noise_profile.

int

User vote for the test case (1 for good quality, -1 for bad quality, 0 for unreviewed).

string

Reason for the user vote given.

float

Confidence for the test case.

string

Reason for the confidence given.

string

The user’s objective for behavior datasets. Used when creating BEHAVIOR test cases.

string

Description of the user persona used to generate scenario-based test cases.

string

Description of the scenario context for scenario-based test cases.

list[string]

List of criteria to end the conversation in scenario-based simulations.

list[dict]

A predefined conversation the simulation starts from, written out. Each entry is one message, {"role": "user" | "assistant", "content": "..."}, and at least one message is required. Galtea stores the conversation as a separate session and names it in the created test case’s seed_session_id. See Starting a Scenario from a Conversation.An assistant message directly after a user message joins that user message to form one turn. Any other message opens a turn of its own, so a conversation may open with an assistant greeting or end on an unanswered question. The four messages above alternate, so they are two turns, which is the unit max_iterations counts.

A message carries text only. retrieval_context on a message is refused, so use seed_session_id when the conversation must carry retrieval context or tool calls. Cannot be sent together with seed_session_id. Both are only accepted on a Behavior test case whose product has at least one version.

string

The ID of an existing session to copy the predefined conversation from. The copy takes that session’s turns and its traces, so a conversation built this way carries the retrieval context and tool calls that the written-out form cannot. The session must belong to the same product as the test case, and every turn in it must be finished. Removed turns at the end of the session are left out of the copy, so a session the agent ended can be copied. A removed turn in the middle of the session is refused.

On the created test case, seed_session_id names the copy Galtea made, not the session you passed. Cannot be sent together with conversation_history.

int

The maximum number of turns the simulator runs before the conversation ends. It counts the live turns only: a test case that starts from a predefined conversation runs this many turns on top of it. A predefined conversation of 4 turns with max_iterations 1 runs one new turn and the session ends with 5.A turn is one exchange. A conversation_history of 8 alternating messages starting with the user is therefore 4 turns, not 8. Messages that do not alternate that way make more turns, so count them with the pairing rule above rather than halving.

Introduction

SDK

CLI

Concepts

Returns

Example

Starting from a predefined conversation

Parameters

​Returns

​Example

​Starting from a predefined conversation

​Parameters

Returns

Example

Starting from a predefined conversation

Parameters