Create Dataset

Returns

Returns a Dataset object for the given parameters.

Examples

Accuracy Dataset
Security Dataset
Behavior Dataset
Accuracy Custom Dataset from CSV

Create an accuracy dataset to evaluate the quality and correctness of outputs.

dataset = galtea.datasets.create(
    name=dataset_name,
    type="ACCURACY",
    product_id=product_id,
    ground_truth_file_path="path/to/knowledge.md",
    language="english",
    max_test_cases=5,
)

Create security and safety datasets to identify potential vulnerabilities in your product.

security_dataset = galtea.datasets.create(
    name=f"security-test-{run_identifier}",
    type="SECURITY",
    product_id=product_id,
    variants=["toxicity"],
    strategies=["original", "role_play", "base64"],
    max_test_cases=50,
)

Generate conversation scenarios tailored to specific user personas and use cases.

behavior_dataset = galtea.datasets.create(
    name=f"behavior-test-{run_identifier}",
    type="BEHAVIOR",
    product_id=product_id,
    custom_user_focus="A medical professional specialized in dementia with more than 15 years in the field",
    language="english",
    max_test_cases=25,
    max_iterations=10,
    strategies=["written"],
)

Upload your own test cases using a CSV file for complete control over dataset content.

csv_dataset = galtea.datasets.create(
    name=f"csv-accuracy-test-{run_identifier}",
    type="ACCURACY",
    product_id=product_id,
    dataset_file_path="path/to/accuracy_dataset.csv",
)

See a complete example of creating custom datasets in our Create a Custom Dataset example.

Parameters

string

required

The name of the dataset.

string | DatasetType

The type of dataset. Required unless specification_id is provided (in which case the type is auto-derived from the specification).

ACCURACY: Datasets that evaluate the quality and correctness of outputs
SECURITY: Datasets that evaluate security, safety, and bias aspects
BEHAVIOR: Datasets that use conversation simulation to evaluate multi-turn dialogue interactions

You can also use the DatasetType enum: DatasetType.ACCURACY, DatasetType.SECURITY, DatasetType.BEHAVIOR.

string

required

The ID of the product you want to evaluate.

string

The ID of a Specification to derive the dataset type from. When provided, type is auto-resolved and becomes optional.

string

Path to a local file containing the knowledge base. This file is uploaded to Galtea and used to build the knowledge base for your dataset.

Supported formats: .pdf, .txt, .json, .html, .md, .yaml, .yml, .xml, .docx, and .zipMaximum file size: 100 MB Maximum ZIP contents: 1,000 files

string

Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases. Example:

Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of Germany?
A: The capital of Germany is Berlin.

This field only applies if datasets are generated by Galtea and are of type ACCURACY.

string

Narrow down the scope of generated scenarios by describing a specific type of user, context, or situation. This helps ensure test cases align with your most relevant goals and flows. Example: “A medical professional specialized in dementia with more than 15 years on the field.”

This field only applies if datasets are generated by Galtea and are of type BEHAVIOR.

string

The language for generating synthetic test cases if ground_truth_file_path is provided. Defaults to the language detected in the ground truth file. Accepts a BCP-47 language tag: a bare ISO 639-1 code ("es") or a code with a region subtag ("es-MX", "en-GB"). The region subtag selects the regional variety in the simulated user’s messages and in voice synthesis (the accent the simulated caller speaks with); generated test text keeps the base language. The value is stored as the full tag.

string

Background noise mixed into the simulated caller’s audio for voice evaluations run over a phone connection. Ignored for datasets that don’t run over a phone connection.

Clip-backed: office, street, car
Synthetic: white, pink

Every test case Galtea generates or ingests from a CSV file inherits this value at creation time. Test cases added afterward with test_cases.create() do not inherit it automatically — pass the same values there to match.

Must be set together with background_noise_level — passing only one of the two is rejected by the API. Leave both unset for a clean recording (the default).

string

How loud the background noise is relative to the caller’s speech, as a fixed signal-to-noise ratio per level. One of light, medium, or heavy. Must be set together with background_noise_profile.

list[string]

A list of strings that specifies how to generate test cases. The meaning of this parameter depends on the dataset type:

For Security datasets (SECURITY): Specifies which threat category to generate test cases for. Exactly one threat must be provided (e.g., ["toxicity"]). Available threats: toxicity, data_leakage, misuse, custom, financial_attacks, illegal_activities.

Required when generating Security datasets (not needed when uploading a custom dataset file via dataset_file_path). Not applicable for Behavior datasets.

list[string]

A list of strings that specifies how to generate test cases related to its style.

For Security datasets (SECURITY): Strategies are techniques for modifying or obfuscating prompts generated for each threat. The original strategy must always be included. See the full list of security strategies.
For Behavior datasets (BEHAVIOR): Strategies define the conversation style. Currently written and spoken are supported, which influence the tone and formality of the generated dialogues. If not specified, the default strategy is written.

str

Description for guiding synthetic data generation for security datasets. This parameter defines the behavior of the custom threat when used as the variant for Security datasets (SECURITY).

Required when variants is set to ["custom"]. The API will reject the request if this field is missing for custom threats.

int

Maximum number of test cases to generate from the knowledge base.Default values:

Accuracy/Security datasets: 50 test cases
Behavior datasets: 20 test cases

Maximum limits:

Accuracy/Security datasets: 1000 test cases
Behavior datasets: 100 test cases

Generating large numbers of test cases can take significant time and consume more credits. Start with the default values and increase as needed.

If you need more than the maximum limit, consider:

Splitting your knowledge base into multiple datasets
Creating test cases manually with test_cases.create()
Uploading a CSV file with predefined test cases using dataset_file_path

int

Maximum number of conversation turns per generated scenario for Behavior datasets (BEHAVIOR).Only applies when scenarios are generated by Galtea. The value is stamped on every generated test case and consumed by the conversation simulator at evaluation time (read as max_turns).Ignored for Accuracy (ACCURACY) and Security (SECURITY) datasets, and ignored when a custom dataset file is uploaded via dataset_file_path (no generation runs).Constraints: must be >= 1.Default: if omitted, the scenario-generator default is used.

string

Path to a local CSV file containing predefined test cases. This file is uploaded to Galtea.File Format by Dataset Type:

Accuracy/Security datasets: Standard format with input, expected_output, tag, source columns
Behavior datasets: Conversation simulator format with goal, user_persona, input, stopping_criterias, max_iterations, scenario columns

See the Conversation Simulator Tutorial for detailed Behavior dataset CSV format examples.

Deprecated alias: test_file_path.

Any

An open field where you can store any value for tracking or organizational purposes. Accepts plain text, numbers, JSON objects, arrays, or booleans.Example: {"environment": "staging", "team": "ml-ops"} or "my tracking note"

Introduction

SDK

CLI

Concepts

Returns

Examples

Parameters

​Returns

​Examples

​Parameters

Returns

Examples

Parameters