Create Test

Returns

Returns a Test object for the given parameters.

Examples

Accuracy Test
Security Test
Behavior Test
Accuracy Custom Test from CSV

Create an accuracy test to evaluate the quality and correctness of outputs.

test = galtea.tests.create(
    name=test_name,
    type="ACCURACY",
    product_id=product_id,
    ground_truth_file_path="path/to/knowledge.md",
    language="english",
    max_test_cases=5,
)

Create security and safety tests to identify potential vulnerabilities in your product.

security_test = galtea.tests.create(
    name=f"security-test-{run_identifier}",
    type="SECURITY",
    product_id=product_id,
    variants=["toxicity"],
    strategies=["original", "role_play", "base64"],
    max_test_cases=50,
)

Generate conversation scenarios tailored to specific user personas and use cases.

behavior_test = galtea.tests.create(
    name=f"behavior-test-{run_identifier}",
    type="BEHAVIOR",
    product_id=product_id,
    custom_user_focus="A medical professional specialized in dementia with more than 15 years in the field",
    language="english",
    max_test_cases=25,
    max_iterations=10,
    strategies=["written"],
)

Upload your own test cases using a CSV file for complete control over test content.

csv_test = galtea.tests.create(
    name=f"csv-accuracy-test-{run_identifier}",
    type="ACCURACY",
    product_id=product_id,
    test_file_path="path/to/accuracy_test.csv",
)

See a complete example of creating custom tests in our Create a Custom Test example.

Parameters

name

string

required

The name of the test.

type

string | TestType

The type of test. Required unless specification_id is provided (in which case the type is auto-derived from the specification).

ACCURACY: Tests that evaluate the quality and correctness of outputs
SECURITY: Tests that evaluate security, safety, and bias aspects
BEHAVIOR: Tests that use conversation simulation to evaluate multi-turn dialogue interactions

You can also use the TestType enum: TestType.ACCURACY, TestType.SECURITY, TestType.BEHAVIOR.

product_id

string

required

The ID of the product you want to evaluate.

specification_id

string

The ID of a Specification to derive the test type from. When provided, type is auto-resolved and becomes optional.

ground_truth_file_path

string

Path to a local file containing the knowledge base. This file is uploaded to Galtea and used to build the knowledge base for your test.

Supported formats: .pdf, .txt, .json, .html, .md, .yaml, .yml, .xml, .docx, and .zipMaximum file size: 100 MB Maximum ZIP contents: 1,000 files

few_shot_examples

string

Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases. Example:

Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of Germany?
A: The capital of Germany is Berlin.

This field only applies if tests are generated by Galtea and are of type ACCURACY.

custom_user_focus

string

Narrow down the scope of generated scenarios by describing a specific type of user, context, or situation. This helps ensure test cases align with your most relevant goals and flows. Example: “A medical professional specialized in dementia with more than 15 years on the field.”

This field only applies if tests are generated by Galtea and are of type BEHAVIOR.

language

string

The language for generating synthetic test cases if ground_truth_file_path is provided. Defaults to the language detected in the ground truth file. Accepts a BCP-47 language tag: a bare ISO 639-1 code ("es") or a code with a region subtag ("es-MX", "en-GB"). The region subtag selects the regional variety in the simulated user’s messages and in voice synthesis (the accent the simulated caller speaks with); generated test text keeps the base language. The value is stored as the full tag.

background_noise_profile

string

Background noise mixed into the simulated caller’s audio for voice evaluations run over a phone connection. Ignored for tests that don’t run over a phone connection.

Clip-backed: office, street, car
Synthetic: white, pink

Every test case Galtea generates or ingests from a CSV file inherits this value at creation time. Test cases added afterward with test_cases.create() do not inherit it automatically — pass the same values there to match.

Must be set together with background_noise_level — passing only one of the two is rejected by the API. Leave both unset for a clean recording (the default).

background_noise_level

string

How loud the background noise is relative to the caller’s speech, as a fixed signal-to-noise ratio per level. One of light, medium, or heavy. Must be set together with background_noise_profile.

variants

list[string]

A list of strings that specifies how to generate test cases. The meaning of this parameter depends on the test type:

For Security tests (SECURITY): Specifies which threat category to generate test cases for. Exactly one threat must be provided (e.g., ["toxicity"]). Available threats: toxicity, data_leakage, misuse, custom, financial_attacks, illegal_activities.

Required when generating Security tests (not needed when uploading a custom test file via test_file_path). Not applicable for Behavior tests.

strategies

list[string]

A list of strings that specifies how to generate test cases related to its style.

For Security tests (SECURITY): Strategies are techniques for modifying or obfuscating prompts generated for each threat. The original strategy must always be included. See the full list of security strategies.
For Behavior tests (BEHAVIOR): Strategies define the conversation style. Currently written and spoken are supported, which influence the tone and formality of the generated dialogues. If not specified, the default strategy is written.

custom_variant_description

str

Description for guiding synthetic data generation for security tests. This parameter defines the behavior of the custom threat when used as the variant for Security tests (SECURITY).

Required when variants is set to ["custom"]. The API will reject the request if this field is missing for custom threats.

max_test_cases

int

Maximum number of test cases to generate from the knowledge base.Default values:

Accuracy/Security tests: 50 test cases
Behavior tests: 20 test cases

Maximum limits:

Accuracy/Security tests: 1000 test cases
Behavior tests: 100 test cases

Generating large numbers of test cases can take significant time and consume more credits. Start with the default values and increase as needed.

If you need more than the maximum limit, consider:

Splitting your knowledge base into multiple tests
Creating test cases manually with test_cases.create()
Uploading a CSV file with predefined test cases using test_file_path

max_iterations

int

Maximum number of conversation turns per generated scenario for Behavior tests (BEHAVIOR).Only applies when scenarios are generated by Galtea. The value is stamped on every generated test case and consumed by the conversation simulator at evaluation time (read as max_turns).Ignored for Accuracy (ACCURACY) and Security (SECURITY) tests, and ignored when a custom test file is uploaded via test_file_path (no generation runs).Constraints: must be >= 1.Default: if omitted, the scenario-generator default is used.

test_file_path

string

Path to a local CSV file containing predefined test cases. This file is uploaded to Galtea.File Format by Test Type:

Accuracy/Security tests: Standard format with input, expected_output, tag, source columns
Behavior tests: Conversation simulator format with goal, user_persona, input, stopping_criterias, max_iterations, scenario columns

See the Conversation Simulator Tutorial for detailed Behavior test CSV format examples.

metadata

Any

An open field where you can store any value for tracking or organizational purposes. Accepts plain text, numbers, JSON objects, arrays, or booleans.Example: {"environment": "staging", "team": "ml-ops"} or "my tracking note"

Introduction

SDK

CLI

Concepts

Returns

Examples

Parameters

​Returns

​Examples

​Parameters

Returns

Examples

Parameters