> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Dataset-Based Evaluations

> Learn how to run evaluations for a single-turn, dataset-based workflow.

> [!WARNING]
>
> You are reading the docs of Galtea 5.15. [Read this page in the latest version](https://docs.galtea.ai/sdk/tutorials/run-dataset-based-evaluations.md).

> [!NOTE]
>
> **Looking for the short path?** `galtea.evaluations.run()` runs a whole evaluation in one call and is what most workflows should use — see [Specification-Driven Evaluations](https://docs.galtea.ai/v5.15/sdk/tutorials/specification-driven-evaluations.md). This page teaches the manual loop on purpose, for the cases where you need to control each test case yourself.

This page teaches the manual loop because it is the workflow that gives you per-test-case control: you decide what to send your product, what output to log, and which metrics score it.

To evaluate a product version against a predefined [Dataset](https://docs.galtea.ai/v5.15/concepts/product/dataset.md), you can loop through its [Test Cases](https://docs.galtea.ai/v5.15/concepts/product/dataset/test-case.md) and run an evaluation for each one. This workflow is ideal for regression testing and standardized quality checks.

> [!NOTE]
>
> A [session](https://docs.galtea.ai/v5.15/concepts/product/version/session.md) is created **implicitly** the first time you run an evaluation for a specific `version_id` and `dataset_id` combination. You do not need to create it manually.

### Workflow

**1. Select a Dataset and Version**

Identify the `dataset_id` and `version_id` you want to evaluate.

**2. Iterate Through Test Cases**

Fetch all test cases associated with the dataset using `galtea.test_cases.list()`.

**3. Generate and Evaluate Output**

For each test case, call your product to get its output, create a session with `galtea.sessions.create()`, then use `galtea.traces.create_and_evaluate()` to log and evaluate the result. You do not need to close the session: the product's auto-close setting closes it after 30 quiet minutes by default, and [`sessions.finish()`](https://docs.galtea.ai/v5.15/sdk/api/session/finish) closes it at once if you prefer.

### Example

This example demonstrates how to run an evaluation on all test cases from a specific dataset.

```python theme={"system"}
# 1. Fetch the test cases for the specified test
test_cases = galtea.test_cases.list(dataset_id=dataset_id)
print(f"Found {len(test_cases)} test cases for Test ID: {dataset_id}")

# 2. Loop through each test case and run an evaluation
for test_case in test_cases:
    # Get your product's actual response to the input
    actual_output = your_product_function(test_case.input)

    # Create a session and evaluate
    session = galtea.sessions.create(version_id=version_id, test_case_id=test_case.id)
    galtea.traces.create_and_evaluate(
        session_id=session.id,
        output=actual_output,
        metrics=METRICS_TO_EVALUATE,
    )

print(f"\nAll evaluations submitted for Version ID: {version_id}")
```

> [!NOTE]
>
> A [session](https://docs.galtea.ai/v5.15/concepts/product/version/session.md) is automatically created behind the scenes to link this `version_id` and `dataset_id` with the provided [trace](https://docs.galtea.ai/v5.15/concepts/product/version/session/trace.md) (the `actual_output` and the [Test Case](https://docs.galtea.ai/v5.15/concepts/product/dataset/test-case.md)'s input).

### Specification-Based Evaluation

Instead of passing metrics explicitly, you can evaluate against [Specifications](https://docs.galtea.ai/v5.15/concepts/product/specification.md). Each specification has linked metrics that the API resolves automatically.

```python theme={"system"}
# 1. List the specifications for your product
specifications = galtea.specifications.list(product_id=product_id)
spec_ids = [spec.id for spec in specifications]
print(f"Found {len(spec_ids)} specifications for product {product_id}")

# 2. Create a session and record a trace
session = galtea.sessions.create(version_id=version_id, test_case_id=test_cases[0].id)
trace = galtea.traces.create(
    session_id=session.id,
    input=test_cases[0].input,
    output=your_product_function(test_cases[0].input),
)

# 3. Evaluate using specifications — the API resolves linked metrics automatically
evaluations = galtea.evaluations.create(
    trace_id=trace.id,
    specification_ids=spec_ids,
)
print(f"Created {len(evaluations)} evaluations from specifications")
```

> [!NOTE]
>
> If you omit both `metrics` and `specification_ids`, the API falls back to all metrics from every specification linked to the product.

## Next Steps

> [card] [Specification-Driven Evaluations](https://docs.galtea.ai/v5.15/sdk/tutorials/specification-driven-evaluations.md)
>
> Automate dataset resolution, agent execution, and evaluation with specifications.

> [card] [Evaluating Conversations](https://docs.galtea.ai/v5.15/sdk/tutorials/evaluating-conversations.md)
>
> Evaluate multi-turn conversations and production sessions.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.