Create a Custom Dataset

Looking for the fastest way to create datasets? If you’ve defined Specifications, Galtea can generate datasets automatically — including type, variants, and test cases. Use this tutorial when you need full control over dataset creation or want to upload your own test data.

You can create Datasets by uploading your own test data or by having Galtea generate test cases from a knowledge file.

Creating Datasets with the SDK

The Galtea SDK provides methods to create Accuracy Datasets, Security & Safety Datasets, and Behavior Datasets.

Upload Existing Dataset File
Generate Dataset from Knowledge Base

If you have a prepared dataset file in CSV format, you can upload it directly.

# Upload a pre-existing dataset file to the Galtea Platform
dataset = galtea.datasets.create(
    name="financial-qa-test-" + run_identifier,
    type="ACCURACY",
    product_id=product_id,
    dataset_file_path="path/to/accuracy_dataset.csv",
)

The dataset file must follow the structure specified for:

Accuracy Datasets: Standard format with input, expected_output, etc.
Security & Safety Datasets: Standard format with adversarial inputs
Behavior Datasets: Conversation simulator format with goal, user_persona, etc.

If the file is not correctly formatted, test cases will not be created automatically, but you can still add them manually.

Behavior Datasets: Use type="BEHAVIOR" to create conversation simulation datasets that enable multi-turn dialogue evaluation with simulated users. See the Conversation Simulator Tutorial for complete examples.

You can provide a knowledge base file (e.g., PDF, TXT) and have Galtea generate the test cases for you. This is a powerful feature for creating comprehensive datasets with minimal effort.

# Generate a dataset from a knowledge base file (PDF, TXT, etc.)
generated_dataset = galtea.datasets.create(
    name="generated-financial-qa-test-" + run_identifier,
    type="ACCURACY",
    product_id=product_id,
    ground_truth_file_path="path/to/knowledge.md",
    language="english",
    max_test_cases=50,  # Limit the number of generated test cases
)

Getting Started

Core Workflows

Production & Monitoring

Advanced

Playbooks

Integrations

Security & Administration

Migration Guides

Create a Custom Dataset

Creating Datasets with the SDK

Next Steps

Run Dataset-Based Evaluations

API Reference

​Creating Datasets with the SDK

​Next Steps

Run Dataset-Based Evaluations

API Reference

Creating Datasets with the SDK

Next Steps