> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Evaluations

> Create evaluations for all inference results within a session, or evaluate a single inference result using specified metrics.

This method evaluates either:

1. An entire conversation stored in a [Session](/concepts/product/version/session) by creating evaluations for each of its [Inference Results](/concepts/product/version/session/inference-result)
2. A single [Inference Result](/concepts/product/version/session/inference-result) by providing its ID

## Returns

Returns a list of [Evaluation](/concepts/product/version/session/evaluation) objects, one for each metric provided.

## Usage

This method evaluates inference results using the specified metrics. It supports both Galtea-hosted metrics and self-hosted custom metrics.

<Note>
  You must provide **either** `session_id` **or** `inference_result_id`, but not both. For single-turn evaluations, you can also use `galtea.inference_results.create_and_evaluate()` which combines creating an inference result and evaluating it in one call.
</Note>

### Development Testing

Evaluate by passing [Specification](/concepts/product/specification) IDs so the API resolves their linked metrics automatically (recommended), or by listing metric names explicitly.

<Tabs>
  <Tab title="Using specification IDs">
    ```python theme={"system"}
    # Resolve metrics automatically from linked specifications instead of listing them
    evaluations = galtea.evaluations.create(
        session_id=session_id,
        specification_ids=[specification.id],
    )
    ```
  </Tab>

  <Tab title="Using metrics">
    ```python theme={"system"}
    evaluations = galtea.evaluations.create(
        session_id=session_id,
        metrics=[{"name": "Role Adherence"}, {"name": "Conversation Relevancy"}],
    )
    ```
  </Tab>
</Tabs>

**If you pre-computed the score (self-hosted metrics)**

```python theme={"system"}
evaluations = galtea.evaluations.create(
    session_id=session_id,
    metrics=[{"name": self_hosted_metric.name, "score": 0.85}],
)
```

### Evaluating a Single Inference Result

You can evaluate a specific inference result by providing its ID instead of a session ID. Pass [Specification](/concepts/product/specification) IDs so the API resolves their linked metrics automatically (recommended), or list metrics explicitly.

<Tabs>
  <Tab title="Using specification IDs">
    ```python theme={"system"}
    # Evaluate the inference result by resolving metrics from linked specifications
    evaluations = galtea.evaluations.create(
        inference_result_id=inference_result_id,
        specification_ids=[specification.id],
    )
    ```
  </Tab>

  <Tab title="Using metrics">
    ```python theme={"system"}
    # Evaluate a specific inference result by providing its ID
    evaluations = galtea.evaluations.create(
        inference_result_id=inference_result_id,
        metrics=[{"name": "Factual Accuracy"}, {"name": "Answer Relevancy"}],
    )
    ```
  </Tab>
</Tabs>

### Production Monitoring

In order to monitor your product in a production environment, you can create evaluations not linked to a specific test case, but you need to set the `is_production` flag of the [Session](/concepts/product/version/session) to `True`. Pass [Specification](/concepts/product/specification) IDs so the API resolves their linked metrics automatically (recommended), or list metrics explicitly.

<Tabs>
  <Tab title="Using specification IDs">
    ```python theme={"system"}
    # Create a production session and an inference result, then evaluate it by
    # resolving metrics from linked specifications (no need to list metrics)
    production_session = galtea.sessions.create(version_id=version_id, is_production=True)

    galtea.inference_results.create(
        session_id=production_session.id,
        input="Production user query",
        output="Production response",
    )

    evaluations = galtea.evaluations.create(
        session_id=production_session.id,
        specification_ids=[specification.id],
    )
    ```
  </Tab>

  <Tab title="Using metrics">
    ```python theme={"system"}
    production_session = galtea.sessions.create(version_id=version_id, is_production=True)

    # Create an inference result for the production session first
    production_inference_result = galtea.inference_results.create(
        session_id=production_session.id,
        input="Production user query",
        output="Production response",
    )

    evaluations = galtea.evaluations.create(
        session_id=production_session.id,
        metrics=[{"name": self_hosted_metric.name, "score": 0.85}],
    )
    ```
  </Tab>
</Tabs>

### Advanced Usage

You can also create evaluations using self-hosted metrics with dynamically calculated scores by utilizing the `CustomScoreEvaluationMetric` class, which allows for more complex evaluation scenarios.

```python theme={"system"}
# First, create a session, in this case it is a production session, so we do not need a test case
session = galtea.sessions.create(version_id=version_id, is_production=True)

# Then, add some inference results to the session
galtea.inference_results.create_batch(
    session_id=session.id,
    conversation_turns=[
        {"role": "user", "content": "Hi"},
        {"role": "assistant", "content": "Hello!"},
        {"role": "user", "content": "How are you?"},
        {"role": "assistant", "content": "I am fine, thank you."},
    ],
)


# Define scoring logic as a class
class PolitenessCheck(CustomScoreEvaluationMetric):
    def __init__(self):
        super().__init__(name="politeness-check")

    def measure(self, *args, actual_output: str | None = None, **kwargs) -> float:
        if actual_output is None:
            return 0.0
        polite_words = ["please", "thank you", "you're welcome"]
        return 1.0 if any(word in actual_output.lower() for word in polite_words) else 0.0


# Create the metric in the platform if it doesn't exist yet
# Note: This can be done via the Dashboard too
try:
    metric = galtea.metrics.get_by_name(name="politeness-check")
except Exception:
    metric = None
if metric is None:
    galtea.metrics.create(
        name="politeness-check",
        source="self_hosted",
        description="Checks if polite words appear in the output",
    )

# Now, evaluate the entire session
evaluations = galtea.evaluations.create(
    session_id=session.id,
    metrics=[
        {"name": "Role Adherence"},  # You can use galtea-hosted metrics simultaneously
        {"score": PolitenessCheck()},  # Self-hosted with dynamic scoring
        # Note: No 'name' or 'id' in dict - it comes from PolitenessCheck(name="...")
    ],
)
```

<Note>
  Both options are equally valid for self-hosted metrics. Choose based on your preference: pre-compute for simplicity, or use CustomScoreEvaluationMetric for encapsulation and reusability.
</Note>

<Info>
  When using `CustomScoreEvaluationMetric`, your `measure()` method receives an `inference_results` parameter containing `InferenceResult` objects — all turns for session evaluations, or a single-element list for single inference result evaluations. This enables conversation-level custom scoring. See [Evaluate with Custom Metrics](/sdk/tutorials/evaluate-with-custom-metrics#multi-turn-custom-metrics) for examples.
</Info>

## Parameters

<ResponseField name="session_id" type="string" optional>
  The ID of the session containing the inference results to be evaluated.

  <Note>Either `session_id` or `inference_result_id` must be provided, but not both.</Note>
</ResponseField>

<ResponseField name="inference_result_id" type="string" optional>
  The ID of a specific inference result to evaluate.

  <Note>Either `session_id` or `inference_result_id` must be provided, but not both.</Note>
</ResponseField>

<ResponseField name="specification_ids" type="List[str]" optional>
  A list of [Specification](/concepts/product/specification) IDs. When provided, the evaluation uses the metrics linked to these specifications.

  Can be combined with `metrics` — the API merges and deduplicates by metric ID. If neither `metrics` nor `specification_ids` is provided, the API falls back to metrics from all specifications linked to the product.

  <Note>
    This parameter allows you to evaluate against specific product specifications without manually listing all their associated metrics.
  </Note>
</ResponseField>

<ResponseField name="metrics" type="List[Union[str, CustomScoreEvaluationMetric, Dict]]" optional>
  A list of metrics to use for the evaluation.

  Optional when `specification_ids` is provided or when the product has specifications with linked metrics (in which case those metrics are used as a fallback).

  The `MetricInput` dictionary supports the following keys:

  * `id` (string, optional): The ID of an existing metric
  * `name` (string, optional): The name of the metric
  * `score` (float | CustomScoreEvaluationMetric, optional): For self-hosted metrics only
    * If `float`: Pre-computed score (0.0 to 1.0). Requires `id` or `name` in the dict.
    * If `CustomScoreEvaluationMetric`: Score will be calculated dynamically. The CustomScoreEvaluationMetric instance must be initialized with `name` or `id`. Do NOT provide `id` or `name` in the dict when using this option.

  <Note>
    For self-hosted metrics, both score options are equally valid: pre-compute as a float, or use CustomScoreEvaluationMetric for dynamic calculation. Galtea-hosted metrics automatically compute scores and should not include a `score` field.
  </Note>
</ResponseField>