Skip to main content

​
What is a Dataset?

A dataset in Galtea is a group of test cases designed to evaluate the performance of a product. A dataset file provides simulations of interactions with the product (and, in Accuracy datasets, expected outcomes for each interaction). You can create, view and manage your datasets on the Galtea dashboard or programmatically using the Galtea SDK.
This entity was called Test before SDK 5.0.0. galtea.tests still works and warns; the new name is galtea.datasets. See Dataset, Trace, and Span Renames.

​
Dataset Origin

When creating a dataset in the Galtea dashboard, you’ll be asked to specify the dataset origin:

Generated

Galtea will take the knowledge base file and generate a set of test cases that will define the dataset.

Uploaded

The dataset is uploaded by you as a complete set of test cases.
Your selection will determine whether you need to provide a Knowledge Base File or a Dataset File.
The SDK parameter variants is used to specify “Threats” for Security datasets. Similarly, the strategies parameter is used for Security datasets to apply different attack modifications.
More information on how to create datasets can be found in the Create Accuracy Datasets and Create Security Datasets documentation.
It is important to note that the information you provide during product onboarding, such as the product’s description and intended use, plays a valuable role when generating test cases. Galtea can leverage this metadata to generate more targeted and context-aware test cases when creating both Accuracy and Security datasets, leading to more effective and insightful evaluations.

​
Dataset Types

Galtea supports three main types of datasets:

Accuracy Datasets

Datasets that evaluate the quality and correctness of outputs.

Security & Safety Datasets

Datasets that evaluate security, safety, and bias aspects, often by generating adversarial inputs based on defined threats and applying various strategies to make them more challenging.

Behavior Datasets

Datasets that evaluate multi-turn dialogue capabilities of an agent, through the use of scenarios which are based on user personas and specific goals.

​
Using Datasets in Evaluations

The Test Cases of a Dataset are used in evaluations to assess the performance of specific versions of your product. These evaluations are grouped within sessions.
The Test Cases of a Dataset should be reused across multiple sessions of distinct versions to ensure consistent comparison between different product versions.

Using Datasets in Evaluations

Learn how to use datasets with evaluations

​
Dataset File Formats

When uploading a dataset file, the required columns depend on the dataset type:

Accuracy Dataset Format

Required: input, expected_output. Optional: context, tag, source. The input column accepts plain strings or JSON objects for structured inputs.

Security & Safety Dataset Format

Required: input. Optional: expected_output, tag, source.

Behavior Dataset Format

Required: goal, user_persona. Optional: input, stopping_criterias, max_iterations.

​
SDK Integration

The Galtea SDK allows you to create, view, and manage datasets programmatically.

Dataset Service SDK

Manage datasets using the Python SDK

Create a Custom Dataset

See how to create and upload custom datasets using the SDK.
Already have a dataset and want more cases? If Galtea generated the dataset, use Extend to add more cases the same way it was created. For any dataset with cases, use Data Augmentation to expand it from its existing examples.

​
Dataset Properties

​
Text
required
The name of the dataset. Example: “Legal Document Accuracy Test” or “Customer Support Safety Evaluation”
​
Enum
The current status of the dataset. Possible values:
  • PENDING: Test cases are being generated
  • SUCCESS: Dataset is ready
  • FAILED: Dataset generation failed
  • AUGMENTING: Test cases are being augmented with additional generated cases
  • EXTENDING: Test cases are being extended by re-running the dataset’s original generator
  • CANCELLED: Dataset generation was cancelled
​
Text
Why generation failed. Set only when the status is FAILED.
​
Enum
required
The type of the dataset. Possible values:
  • Accuracy (ACCURACY): Datasets that evaluate the quality and correctness of outputs
  • Security & Safety (SECURITY): Datasets that evaluate security, safety, and bias aspects
  • Behavior (BEHAVIOR): Datasets that use conversation simulation to evaluate multi-turn dialogue interactions
​
Any
An open field where you can store any value for tracking or organizational purposes. Accepts plain text, numbers, JSON objects, arrays, or booleans.Example: {"environment": "staging", "team": "ml-ops"} or "my tracking note"
​
Enum
Background noise mixed into the simulated caller’s audio for voice datasets run over a phone connection. Only applies to voice datasets; has no effect otherwise. Possible values:
  • Clip-backed: office, street, car
  • Synthetic: white, pink
Every test case Galtea generates or ingests from a CSV file inherits this value at creation time. Must be set together with Background Noise Level.
​
Enum
How loud the background noise is relative to the caller’s speech. Possible values: light, medium, heavy. Must be set together with Background Noise Profile.
​
Text
Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases. Example:
This field only applies if datasets are generated by Galtea and are of type Accuracy.
​
Text
The language for generating synthetic test cases if Knowledge Base File is provided (e.g., ‘english’, ‘spanish’). This should be the English name of the language. If not provided, Galtea attempts to infer the language from the knowledge base file. Supported languages include English, Spanish, Catalan, French, German, Portuguese, Italian, Dutch, Polish, Chinese, Korean, and Japanese.
This field only applies if datasets are generated by Galtea (using Knowledge Base File).
​
Number
The maximum number of test cases generated by Galtea. This helps control the size of the dataset and associated costs.
​
File
required
The path to a local file (e.g., PDF, TXT, JSON, HTML, Markdown) containing the knowledge base. This file is uploaded to Galtea, which then generates test cases based on its content. Required if the test cases are to be generated by Galtea. Example: “path/to/your/knowledge_base.pdf”

Concepts overview

How Galtea’s concepts connect — diagram + per-entity quick reference.

Create a Dataset

Tutorial on creating datasets via the SDK.

Specification-Driven Datasets

Auto-derive dataset types from specifications.