> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset

> A dataset of test cases for evaluating product performance

> [!WARNING]
>
> You are reading the docs of Galtea 5.13. [Read this page in the latest version](https://docs.galtea.ai/concepts/product/dataset.md).

## What is a Dataset?

**A dataset in Galtea is a group of test cases** designed to evaluate the performance of a [product](https://docs.galtea.ai/v5.13/concepts/product.md). A dataset file provides simulations of interactions with the product (and, in [Accuracy datasets](https://docs.galtea.ai/v5.13/concepts/product/dataset/accuracy-datasets.md), expected outcomes for each interaction).

You can **create**, view and manage your datasets on the [Galtea dashboard](https://platform.galtea.ai/) or programmatically using the [Galtea SDK](https://docs.galtea.ai/v5.13/sdk/api/dataset/service.md).

> [!NOTE]
>
> **This entity was called Test before SDK 5.0.0.** `galtea.tests` still works and warns; the new
> name is `galtea.datasets`. See
> [Dataset, Trace, and Span Renames](https://docs.galtea.ai/v5.13/sdk/migration-guides/entity-renames.md).

## Dataset Origin

When creating a dataset in the Galtea dashboard, you'll be asked to specify the dataset origin:

> [card] Generated
>
> Galtea will take the knowledge base file and generate a set of test cases that will define the dataset.

> [card] Uploaded
>
> The dataset is uploaded by you as a complete set of test cases.

> [card] Manual
>
> Start with an empty dataset and add each test case yourself. Accuracy and Security test cases can carry file attachments.

Behavior datasets have a fourth origin, "From an existing Accuracy Dataset". It uses the test cases of an Accuracy Dataset as the basis for new behavior test cases.

Your selection will determine whether you need to provide a *Knowledge Base File*, a *Dataset File*, or no file at all. The manual origin needs no file: you add the test cases one by one after the dataset is created.

<video controls muted playsInline src="https://mintcdn.com/galtea/dTqmcvyz2pQyIXAj/videos/test-generation-demo.mp4?fit=max&auto=format&n=dTqmcvyz2pQyIXAj&q=85&s=0b468d8549d330f78e1ca41b8b0f5d2d" />

> [!NOTE]
>
> The SDK parameter `variants` is used to specify "Threats" for Security datasets. Similarly, the `strategies` parameter is used for Security datasets to apply different attack modifications.

> [!NOTE]
>
> More information on how to create datasets can be found in the [Create Accuracy Datasets](https://docs.galtea.ai/v5.13/concepts/product/dataset/accuracy-datasets.md) and [Create Security Datasets](https://docs.galtea.ai/v5.13/concepts/product/dataset/security-datasets.md) documentation.

> [!NOTE]
>
> It is important to note that the information you provide during **product onboarding**, such as the product's description and intended use, plays a valuable role when generating test cases. Galtea can leverage this metadata to generate more targeted and context-aware test cases when creating both Accuracy and Security datasets, leading to more effective and insightful evaluations.

## Dataset Types

Galtea supports three main types of datasets:

> [card] [Accuracy Datasets](https://docs.galtea.ai/v5.13/concepts/product/dataset/accuracy-datasets.md)
>
> Datasets that evaluate the quality and correctness of outputs.

> [card] [Security & Safety Datasets](https://docs.galtea.ai/v5.13/concepts/product/dataset/security-datasets.md)
>
> Datasets that evaluate security, safety, and bias aspects, often by generating adversarial inputs based on defined threats and applying various strategies to make them more challenging.

> [card] [Behavior Datasets](https://docs.galtea.ai/v5.13/concepts/product/dataset/behavior-datasets.md)
>
> Datasets that evaluate multi-turn dialogue capabilities of an agent, through the use of scenarios which are based on user personas and specific goals.

## Using Datasets in Evaluations

The [Test Cases](https://docs.galtea.ai/v5.13/concepts/product/dataset/test-case.md) of a Dataset are used in [evaluations](https://docs.galtea.ai/v5.13/concepts/product/version/session/evaluation.md) to assess the performance of specific [versions](https://docs.galtea.ai/v5.13/concepts/product/version.md) of your product. These evaluations are grouped within [sessions](https://docs.galtea.ai/v5.13/concepts/product/version/session.md).

> [!WARNING]
>
> The [Test Cases](https://docs.galtea.ai/v5.13/concepts/product/dataset/test-case.md) of a Dataset should be reused across multiple [sessions](https://docs.galtea.ai/v5.13/concepts/product/version/session.md) of distinct [versions](https://docs.galtea.ai/v5.13/concepts/product/version.md) to ensure consistent comparison between different [product](https://docs.galtea.ai/v5.13/concepts/product.md) [versions](https://docs.galtea.ai/v5.13/concepts/product/version.md).

> [card] [Using Datasets in Evaluations](https://docs.galtea.ai/v5.13/concepts/product/version/session/evaluation.md)
>
> Learn how to use datasets with evaluations

## Dataset File Formats

When uploading a dataset file, the required columns depend on the dataset type:

> [card] [Accuracy Dataset Format](https://docs.galtea.ai/v5.13/concepts/product/dataset/accuracy-datasets.md#example-accuracy-datasets-and-file-format)
>
> Required: `input`, `expected_output`. Optional: `context`, `tag`, `source`. The `input` column accepts plain strings or JSON objects for structured inputs.

> [card] [Security & Safety Dataset Format](https://docs.galtea.ai/v5.13/concepts/product/dataset/security-datasets.md#example-security--safety-datasets-and-file-format)
>
> Required: `input`. Optional: `expected_output`, `tag`, `source`.

> [card] [Behavior Dataset Format](https://docs.galtea.ai/v5.13/concepts/product/dataset/behavior-datasets.md#example-behavior-datasets-and-file-format)
>
> Required: `goal`, `user_persona`, `scenario`. Optional: `input`, `stopping_criterias`, `max_iterations`, `conversation_history`.

## SDK Integration

The Galtea SDK allows you to create, view, and manage datasets programmatically.

> [card] [Dataset Service SDK](https://docs.galtea.ai/v5.13/sdk/api/dataset/service.md)
>
> Manage datasets using the Python SDK

> [card] [Create a Custom Dataset](https://docs.galtea.ai/v5.13/sdk/tutorials/create-dataset.md)
>
> See how to create and upload custom datasets using the SDK.

> [!NOTE]
>
> Already have a dataset and want more cases? If Galtea generated the dataset, use [Extend](https://docs.galtea.ai/v5.13/concepts/product/dataset/extend-test-cases.md) to add more cases the same way it was created. For a dataset with text-only cases, use [Data Augmentation](https://docs.galtea.ai/v5.13/concepts/product/dataset/data-augmentation.md) to expand it from its existing examples.

## Dataset Properties

**Dataset Name** `Text` *required*

The name of the dataset. **Example**: "Legal Document Accuracy Test" or "Customer Support Safety Evaluation"

**Status** `Enum`

The current status of the dataset.
Possible values:

* `PENDING`: Test cases are being generated
* `SUCCESS`: Dataset is ready
* `FAILED`: Dataset generation failed
* `AUGMENTING`: Test cases are being augmented with additional generated cases
* `EXTENDING`: Test cases are being extended by re-running the dataset's original generator
* `CANCELLED`: Dataset generation was cancelled

**Error** `Text`

Why generation failed. Set only when the status is `FAILED`.

**Type** `Enum` *required*

The type of the dataset.
Possible values:

* [Accuracy](https://docs.galtea.ai/v5.13/concepts/product/dataset/accuracy-datasets.md) (`ACCURACY`): Datasets that evaluate the quality and correctness of outputs
* [Security & Safety](https://docs.galtea.ai/v5.13/concepts/product/dataset/security-datasets.md) (`SECURITY`): Datasets that evaluate security, safety, and bias aspects
* [Behavior](https://docs.galtea.ai/v5.13/concepts/product/dataset/behavior-datasets.md) (`BEHAVIOR`): Datasets that use conversation simulation to evaluate multi-turn dialogue interactions

**Specifications** `List[Text]`

The [Specifications](https://docs.galtea.ai/v5.13/concepts/product/specification.md) the dataset links to. A dataset can link to several specifications of its product. An evaluation that names several of them runs the dataset once and scores it with the metrics of all of them. In the SDK this is `specification_ids`; the older `specification_id` is deprecated and holds the oldest link.

**Metadata** `Any`

An open field where you can store any value for tracking or organizational purposes. Accepts plain text, numbers, JSON objects, arrays, or booleans.

**Example**: `{"environment": "staging", "team": "ml-ops"}` or `"my tracking note"`

**Background Noise Profile** `Enum`

Background noise mixed into the simulated caller's audio for voice datasets run over a [phone connection](https://docs.galtea.ai/v5.13/concepts/product/phone-connection.md). Only applies to voice datasets; has no effect otherwise.
Possible values:

* Clip-backed: `office`, `street`, `car`
* Synthetic: `white`, `pink`

Every test case Galtea generates or ingests from a CSV file inherits this value at creation time. Must be set together with **Background Noise Level**.

**Background Noise Level** `Enum`

How loud the background noise is relative to the caller's speech.
Possible values: `light`, `medium`, `heavy`. Must be set together with **Background Noise Profile**.

> [tab] Generated
>
> **Few Shot Examples** `Text`
>
> Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases.
> **Example**:
>
> ```
> Q: What is the capital of France?
> A: The capital of France is Paris.
> Q: What is the capital of Germany?
> A: The capital of Germany is Berlin.
> ```
>
> > [!NOTE]
> >
> > This field only applies if datasets are generated by Galtea and are of type Accuracy.
>
> **Language** `Text`
>
> The language for generating synthetic test cases if `Knowledge Base File` is provided (e.g., 'english', 'spanish'). This should be the English name of the language. If not provided, Galtea attempts to infer the language from the knowledge base file. Supported languages include English, Spanish, Catalan, French, German, Portuguese, Italian, Dutch, Polish, Chinese, Korean, and Japanese.
> \[!NOTE] This field only applies if datasets are generated by Galtea (using `Knowledge Base File`).
>
> **Max Test Cases** `Number`
>
> The maximum number of test cases generated by Galtea. This helps control the size of the dataset and associated costs.
>
> > [tab] Accuracy
> >
> > **Knowledge Base File** `File` *required*
> >
> > The path to a local file (e.g., PDF, TXT, JSON, HTML, Markdown) containing the knowledge base. This file is uploaded to Galtea, which then generates test cases based on its content. Required if the test cases are to be generated by Galtea. **Example**: "path/to/your/knowledge\_base.pdf"
>
> > [tab] Security & Safety
> >
> > **Threats** `List[Enum]` *required*
> >
> > Specifies which [threat categories](https://docs.galtea.ai/v5.13/concepts/product/dataset/security-threats.md) to generate test cases for. This corresponds to the `variants` parameter in the SDK.
> >
> > **Strategies** `List[Enum]`
> >
> > A list of [security strategies](https://docs.galtea.ai/v5.13/concepts/product/dataset/security-strategies.md) to modify prompts for each threat. This corresponds to the `strategies` parameter in the SDK.
>
> > [tab] Behavior
> >
> > **Knowledge Base File** `File`
> >
> > Optional file containing context or domain knowledge to help generate more realistic conversation scenarios. **Example**: "path/to/your/domain\_context.pdf"
> >
> > **custom\_user\_focus** `Text`
> >
> > Narrow down the scope of generated scenarios by describing a specific type of user, context, or situation. This helps ensure test cases align with your most relevant goals and flows. **Example**: "A medical professional specialized in dementia with more than 15 years in the field."
> >
> > **Max Test Cases** `Number`
> >
> > The maximum number of conversation scenarios generated by Galtea. This helps control the size of the dataset.

> [tab] Uploaded
>
> **Dataset File** `File` *required*
>
> The path to a local CSV file containing predefined test cases. This file is uploaded to Galtea. Required if you are providing your own set of test cases instead of having Galtea generate them. **Example**: "path/to/your/test\_file.csv"
>
> **File Format Requirements:**
>
> * **Accuracy/Security Datasets**: Must include `input`, `expected_output`, `tag`, `source` columns
> * **Behavior Datasets**: Must include conversation simulation columns (see Behavior Dataset File Format below)

> [tab] Manual
>
> No file or generation settings are needed. The dataset is created with no test cases, so add at least one before you run an evaluation. Add each test case from the dataset's *Test Cases* tab with *New Test Case*. For Accuracy and Security, a test case has an input, an optional expected output and optional file attachments. For Behavior, a test case has a persona, a scenario, a goal and a first input. *New Test Case* starts each test case with the dataset's background noise, and you can change it there. A test case created through the API or the SDK has no background noise unless the request sets it. This origin is available in the dashboard.

## Related

> [card] [Concepts overview](https://docs.galtea.ai/v5.13/concepts/overview.md)
>
> How Galtea's concepts connect — diagram + per-entity quick reference.

> [card] [Create a Dataset](https://docs.galtea.ai/v5.13/sdk/tutorials/create-dataset.md)
>
> Tutorial on creating datasets via the SDK.

> [card] [Specification-Driven Datasets](https://docs.galtea.ai/v5.13/sdk/tutorials/specification-driven-evaluations.md)
>
> Auto-derive dataset types from specifications.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.