Create a Custom Test

While Galtea can automatically generate tests from a knowledge file, you can also create custom Tests by providing your own set of Test Cases.

Creating Tests with the SDK

Tests can be generated automatically by Galtea providing a knowledge file via the SDK or the dashboard.

The Galtea SDK provides methods to create both Quality Tests and Red Teaming Tests.

Upload Existing Test File
Generate Test from Knowledge Base

If you have a prepared test file in CSV format, you can upload it directly.

# Upload a pre-existing test file to the Galtea Platform
test = galtea.tests.create(
    name="financial-qa-test-" + run_identifier,
    type="QUALITY",
    product_id=product_id,
    test_file_path="path/to/quality_test.csv",
)

The test file must follow the structure specified for:

Quality Tests: Standard format with input, expected_output, etc.
Red Teaming Tests: Standard format with adversarial inputs
Scenarios Tests: Conversation simulator format with goal, user_persona, etc.

If the file is not correctly formatted, test cases will not be created automatically, but you can still add them manually.

New Test Type: Use type="SCENARIOS" to create conversation simulation tests that enable multi-turn dialogue evaluation with synthetic users. See the Conversation Simulator Tutorial for complete examples.

You can provide a knowledge base file (e.g., PDF, TXT) and have Galtea generate the test cases for you. This is a powerful feature for creating comprehensive tests with minimal effort.

# Generate a test from a knowledge base file (PDF, TXT, etc.)
generated_test = galtea.tests.create(
    name="generated-financial-qa-test-" + run_identifier,
    type="QUALITY",
    product_id=product_id,
    ground_truth_file_path="path/to/knowledge.md",
    language="english",
    max_test_cases=50,  # Limit the number of generated test cases
)

Getting Started

Tutorials

Integrations

Creating Tests with the SDK

Next Steps

Run Test-Based Evaluations

API Reference

Getting Started

Tutorials

Integrations

​Creating Tests with the SDK

​Next Steps

Run Test-Based Evaluations

API Reference

Creating Tests with the SDK

Next Steps