Run Dataset-Based Evaluations

Looking for the short path? galtea.evaluations.run() runs a whole evaluation in one call and is what most workflows should use — see Specification-Driven Evaluations. This page teaches the manual loop on purpose, for the cases where you need to control each test case yourself.

This page teaches the manual loop because it is the workflow that gives you per-test-case control: you decide what to send your product, what output to log, and which metrics score it. To evaluate a product version against a predefined Dataset, you can loop through its Test Cases and run an evaluation for each one. This workflow is ideal for regression testing and standardized quality checks.

A session is created implicitly the first time you run an evaluation for a specific version_id and dataset_id combination. You do not need to create it manually.

Workflow

Select a Dataset and Version

Identify the dataset_id and version_id you want to evaluate.

Iterate Through Test Cases

Fetch all test cases associated with the dataset using galtea.test_cases.list().

Generate and Evaluate Output

For each test case, call your product to get its output, create a session with galtea.sessions.create(), then use galtea.traces.create_and_evaluate() to log and evaluate the result.

Example

This example demonstrates how to run an evaluation on all test cases from a specific dataset.

# 1. Fetch the test cases for the specified test
test_cases = galtea.test_cases.list(dataset_id=dataset_id)
print(f"Found {len(test_cases)} test cases for Test ID: {dataset_id}")

# 2. Loop through each test case and run an evaluation
for test_case in test_cases:
    # Get your product's actual response to the input
    actual_output = your_product_function(test_case.input)

    # Create a session and evaluate
    session = galtea.sessions.create(version_id=version_id, test_case_id=test_case.id)
    galtea.traces.create_and_evaluate(
        session_id=session.id,
        output=actual_output,
        metrics=METRICS_TO_EVALUATE,
    )

print(f"\nAll evaluations submitted for Version ID: {version_id}")

A session is automatically created behind the scenes to link this version_id and dataset_id with the provided trace (the actual_output and the Test Case’s input).

Specification-Based Evaluation

Instead of passing metrics explicitly, you can evaluate against Specifications. Each specification has linked metrics that the API resolves automatically.

# 1. List the specifications for your product
specifications = galtea.specifications.list(product_id=product_id)
spec_ids = [spec.id for spec in specifications]
print(f"Found {len(spec_ids)} specifications for product {product_id}")

# 2. Create a session and record a trace
session = galtea.sessions.create(version_id=version_id, test_case_id=test_cases[0].id)
trace = galtea.traces.create(
    session_id=session.id,
    input=test_cases[0].input,
    output=your_product_function(test_cases[0].input),
)

# 3. Evaluate using specifications — the API resolves linked metrics automatically
evaluations = galtea.evaluations.create(
    trace_id=trace.id,
    specification_ids=spec_ids,
)
print(f"Created {len(evaluations)} evaluations from specifications")

If you omit both metrics and specification_ids, the API falls back to all metrics from every specification linked to the product.

Next Steps

Specification-Driven Evaluations

Automate dataset resolution, agent execution, and evaluation with specifications.

Evaluating Conversations

Evaluate multi-turn conversations and production sessions.

Getting Started

Core Workflows

Production & Monitoring

Advanced

Playbooks

Integrations

Security & Administration

Migration Guides

Run Dataset-Based Evaluations

Workflow

Example

Specification-Based Evaluation

Next Steps

Specification-Driven Evaluations

Evaluating Conversations

​Workflow

​Example

​Specification-Based Evaluation

​Next Steps

Specification-Driven Evaluations

Evaluating Conversations

Workflow

Example

Specification-Based Evaluation

Next Steps