Skip to main content
You are reading the docs of Galtea 5.15. Read this page in the latest version.

Returns

Returns a TestCase object for the given parameters.

Example

Starting from a predefined conversation

Write the turns out, or copy them from a session that already holds them. See Starting a Scenario from a Conversation.

Parameters

string
required
The ID of the dataset you want to create the test case for.
Deprecated alias: test_id.
string
The input for the test case. For Accuracy and Safety datasets, this is the question or prompt you want to send to your model for evaluation. For Behavior datasets, this is the first user message to use in the conversation (bypassing the simulator for the first turn).
list[string]
Files to attach to the input, for a product that reads documents. Each entry is a local path the SDK uploads, or a Galtea file reference from an earlier upload, which is attached without uploading the file again. Any other URL raises a ValueError before anything is uploaded. input stays optional, so a test case may hold files and no text at all. Galtea sends each attachment to your endpoint when the endpoint connection template names input_files.At most 20 files totalling 20 MB, of these types: pdf, docx, xlsx, pptx, rtf, png, jpg, jpeg, tiff, tif, bmp, webp, heic, heif, gif, txt, csv, md, html, xml, json, eml. To attach one already-uploaded file to several test cases, use Upload Input File instead.The rule below holds for both run styles: your own pipeline, and Galtea calling your endpoint. See File inputs.
A metric scores a file only when its judge model reads that kind of file. Only a Galtea file, one you uploaded to Galtea, is sent to a judge. A file on your own host is stored as sent and never reaches a judge, so it never skips an evaluation: the judge reads the text beside it. The kinds a judge reads depend on its model, and the model picker of the metric form shows them for each model. A judge is sent PNG, JPEG, WebP, GIF and PDF files only. The other accepted types are stored, and a Galtea input file of those types still reaches your endpoint, but no judge reads them. The rule is the same for the input and the output, and a metric that reads conversation_turns reads both sides of every turn.Before an evaluation runs, Galtea checks every Galtea file on each side the metric reads. One file the judge would not read skips the evaluation, whatever text sits beside it. A skipped evaluation costs no credits, and its reason names the file, its turn and what refused it:
  • The judge model cannot read that kind of file.
  • The model catalog does not say whether the judge model reads it.
  • The metric names no judge model.
  • The metric is DeepEval-based: those judges are sent text only.
  • The metric is deterministic: it compares text without a model.
  • The metric is a classifier metric: it is sent text only.
  • The file is a type no judge reads, such as DOCX, CSV or TIFF, or an audio file: a judge reads audio only through its transcript.
When the evaluator then reads a Galtea file to send it, it skips the evaluation if the file no longer exists in storage, is empty, or is larger than the 20 MB a judge can receive. The reason names the file.Audio input with no transcript is a separate case. A metric that reads the input is skipped when no turn of the session has input text or a Galtea input file, and a turn’s audio carries no transcript. Text or a Galtea file on another turn is enough, and the reason, Evaluators read an audio input through its transcript, and no turn carries one., names no file.Human evaluation and self-hosted metrics are never skipped for a file or for audio. On deployments that do not send files to the judge, the evaluator also skips a metric whose judge would read the file, and the reason names the file.To have the content scored, pick a metric whose judge model reads that kind of file, or log the text your pipeline extracted as the turn input or output, without the file. See Troubleshooting skipped evaluations.
string
The expected output for the input given. This is the correct answer or response you want to evaluate against. For a JSON Field Match metric, a field of a JSON object can hold a field rule. Rules travel inside this string, so write each backslash twice.
list[string]
The expected tools for the input given. This is the list of tools that the model should use to generate the expected output.
string
The context for the test case. This is the additional information that can help the model generate the expected output.
string
An optional variant to categorize or label the test case (e.g., “original”, “paraphrased”).
string
The ID of the user who has reviewed and validated this test case.
string
Language for the test case. Accepts a BCP-47 language tag: a bare ISO 639-1 code ("es") or a code with a region subtag ("es-MX", "en-GB"). The region subtag selects the regional variety in the simulated user’s messages and in voice synthesis (the accent the simulated caller speaks with); generated test text keeps the base language. The value is stored as the full tag. A bare code for ca, de, es, eu, fr, gl, it, nl or pt defaults to that language’s European variety in the simulated user’s messages only (pt → pt-PT); an explicit region always overrides it, and neither voice synthesis nor the stored value is affected.
string
Persona gender, "MALE" or "FEMALE", for voice evaluations: it selects the male or female synthesized voice for the simulated caller. A voice pinned for the call’s locale takes precedence. Text evaluations ignore it. When unset, the voice uses the default for the language.
string
Background noise mixed into the simulated caller’s audio for voice evaluations run over a phone connection. Ignored for test cases that don’t run over a phone connection.
  • Clip-backed: office, street, car
  • Synthetic: white, pink
This method creates the test case directly, so it does not inherit the parent dataset’s background noise — omitting this parameter leaves the test case with no noise. (Test cases generated by Galtea from a ground truth file or a CSV upload do inherit the dataset’s noise automatically; this only applies to cases created one-by-one through this method.)Must be set together with background_noise_level — passing only one of the two is rejected by the API.
string
How loud the background noise is relative to the caller’s speech, as a fixed signal-to-noise ratio per level. One of light, medium, or heavy. Must be set together with background_noise_profile.
int
User vote for the test case (1 for good quality, -1 for bad quality, 0 for unreviewed).
string
Reason for the user vote given.
float
Confidence for the test case.
string
Reason for the confidence given.
string
The user’s objective for behavior datasets. Used when creating BEHAVIOR test cases.
string
Description of the user persona used to generate scenario-based test cases.
string
Description of the scenario context for scenario-based test cases.
list[string]
List of criteria to end the conversation in scenario-based simulations.
list[dict]
A predefined conversation the simulation starts from, written out. Each entry is one message, {"role": "user" | "assistant", "content": "..."}, and at least one message is required. Galtea stores the conversation as a separate session and names it in the created test case’s seed_session_id. See Starting a Scenario from a Conversation.An assistant message directly after a user message joins that user message to form one turn. Any other message opens a turn of its own, so a conversation may open with an assistant greeting or end on an unanswered question. The four messages above alternate, so they are two turns, which is the unit max_iterations counts.
A message carries text only. retrieval_context on a message is refused, so use seed_session_id when the conversation must carry retrieval context or tool calls. Cannot be sent together with seed_session_id. Both are only accepted on a Behavior test case whose product has at least one version.
string
The ID of an existing session to copy the predefined conversation from. The copy takes that session’s turns and its traces, so a conversation built this way carries the retrieval context and tool calls that the written-out form cannot. The session must belong to the same product as the test case, and every turn in it must be finished. Removed turns at the end of the session are left out of the copy, so a session the agent ended can be copied. A removed turn in the middle of the session is refused.
On the created test case, seed_session_id names the copy Galtea made, not the session you passed. Cannot be sent together with conversation_history.
int
The maximum number of turns the simulator runs before the conversation ends. It counts the live turns only: a test case that starts from a predefined conversation runs this many turns on top of it. A predefined conversation of 4 turns with max_iterations 1 runs one new turn and the session ends with 5.A turn is one exchange. A conversation_history of 8 alternating messages starting with the user is therefore 4 turns, not 8. Messages that do not alternate that way make more turns, so count them with the pairing rule above rather than halving.