Create a Custom Test

While Galtea can automatically generate tests from a knowledge file, you can also create custom Tests by providing your own set of Test Cases.

Creating Tests with the SDK

Tests can be generated automatically by Galtea providing a knowledge file via the sdk or the dashboard.

The Galtea SDK provides methods to create both Quality Tests and Red Teaming Tests.

If you have a prepared test file in CSV format, you can upload it directly.

from galtea import Galtea
import os

galtea = Galtea(api_key=os.getenv("GALTEA_API_KEY"))

# Upload a pre-existing test file to the Galtea Platform
test = galtea.tests.create(
    name="financial-qa-test",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    test_file_path="path/to/your/finance_db_sample.csv"
)

print(f"Test created with ID: {test.id}")

The test file must follow the structure specified for:

Quality Tests: Standard format with input, expected_output, etc.
Red Teaming Tests: Standard format with adversarial inputs
Scenarios Tests: Conversation simulator format with goal, user_persona, etc.

If the file is not correctly formatted, test cases will not be created automatically, but you can still add them manually.

New Test Type: Use type="SCENARIOS" to create conversation simulation tests that enable multi-turn dialogue evaluation with synthetic users. See the Conversation Simulator Tutorial for complete examples.

If you have a prepared test file in CSV format, you can upload it directly.

from galtea import Galtea
import os

galtea = Galtea(api_key=os.getenv("GALTEA_API_KEY"))

# Upload a pre-existing test file to the Galtea Platform
test = galtea.tests.create(
    name="financial-qa-test",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    test_file_path="path/to/your/finance_db_sample.csv"
)

print(f"Test created with ID: {test.id}")

The test file must follow the structure specified for:

Quality Tests: Standard format with input, expected_output, etc.
Red Teaming Tests: Standard format with adversarial inputs
Scenarios Tests: Conversation simulator format with goal, user_persona, etc.

If the file is not correctly formatted, test cases will not be created automatically, but you can still add them manually.

You can provide a knowledge base file (e.g., PDF, TXT) and have Galtea generate the test cases for you. This is a powerful feature for creating comprehensive tests with minimal effort.

from galtea import Galtea
import os

galtea = Galtea(api_key=os.getenv("GALTEA_API_KEY"))

# Generate a test from a PDF knowledge base
test = galtea.tests.create(
    name="legal-document-test-from-pdf",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    ground_truth_file_path="path/to/your/legal_document.pdf",
    language="english",
    # Generate variations to improve test robustness
    variants=["paraphrased", "incorrect", "typos"], 
    max_test_cases=50 # Limit the number of generated test cases
)

print(f"Test generated from knowledge base with ID: {test.id}")

The variants parameter instructs Galtea to create evolved versions of the base test cases. See the Quality Test Evolutions documentation for more details. For Red Teaming, variants specify threats and you can also add strategies.

Getting Started

Tutorials

Integrations

Creating Tests with the SDK

Next Steps

Run a Test-Based Evaluation

API Reference

Getting Started

Tutorials

Integrations

​Creating Tests with the SDK

​Next Steps

Run a Test-Based Evaluation

API Reference

Creating Tests with the SDK

Next Steps