> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation Types

> How metrics are scored: Self-Hosted, AI Evaluation, or Human Evaluation

> [!WARNING]
>
> You are reading the docs of Galtea 5.13. [Read this page in the latest version](https://docs.galtea.ai/concepts/metric/evaluation-types.md).

Every metric in Galtea has an **Evaluation Type** that determines how responses are scored. Choose the type that matches your scoring needs.

## Self-Hosted

This method is for deterministic metrics that you score locally. No evaluation prompt is sent to Galtea. Instead, your custom logic runs on your infrastructure, and the resulting score is uploaded to the platform for tracking and analysis.

You can provide the score in two ways via the SDK:

* **Pre-calculated score**: Pass a `float` value directly in the `score` field of a `MetricInput` dictionary. This is the simplest method.
  Example: `{'name': 'my-custom-metric', 'score': 0.85}`

* **Dynamic score calculation**: Use the SDK's `CustomScoreEvaluationMetric` class to encapsulate your scoring logic, which will be executed at runtime.
  Example: `{'score': MyCustomMetric(name='my-custom-metric')}`

> [card] [Custom Scores Tutorial](https://docs.galtea.ai/v5.13/sdk/tutorials/evaluate-with-custom-metrics.md)
>
> Learn how to implement and use custom scoring metrics with the SDK.

## AI Evaluation

An LLM scores responses using a scoring rubric you define. You provide the core evaluation logic (your criteria and rubric) in the `judge_prompt`, then select the necessary data fields (like `input`, `context`, etc.) from the [Evaluation Parameters](https://docs.galtea.ai/v5.13/concepts/metric/evaluation-parameters.md) list. Galtea dynamically constructs the final prompt by prepending the selected data to the content of your `judge_prompt`, ensuring a consistent structure for the evaluator model.

> [!NOTE]
>
> The non-deterministic metrics, powered by Large Language Models that act as Judges, utilize models that have demonstrated the best performance in our internal benchmarks and testing. We are committed to continuously evolving and improving these evaluator models to ensure the highest quality assessments over time.

### Choose the judge: LLM judge or Classifier (Jev)

The **Judge** field picks what scores the metric.

* **LLM judge**: an evaluator model reads your judge prompt and parameters and returns a score with its reasoning; the cost depends on the model.
* **Classifier (Jev)**: TypeSafe's Jev answers one typed question (Yes/No or Scale) with probabilities. It writes no reasoning and costs 1 credit. It is available only where the deployment enables it; SDK `source="classifier"`.

The steps below describe the LLM judge.

> [card] [Classifier (Jev)](https://docs.galtea.ai/v5.13/concepts/metric/classifier.md)
>
> Score a metric with one typed question instead of a judge prompt.

### How It Works

1. You write the evaluation criteria (what to check for)
2. You define the scoring rubrics (how to score)
3. You select which [evaluation parameters](https://docs.galtea.ai/v5.13/concepts/metric/evaluation-parameters.md) to include
4. Galtea automatically constructs the complete evaluation prompt

### Example Judge Prompt

```python theme={"system"}

judge_prompt = """
**Evaluation Criteria:**
Check if the ACTUAL_OUTPUT is good by comparing it to what was expected. Focus on:
1. Factual accuracy and correctness
2. Completeness of the ACTUAL_OUTPUT, regarding the user INPUT
3. Appropriate use of provided CONTEXT information to answer the user INPUT
4. Overall helpfulness and relevance to the user INPUT

**Rubric:**
Score 1 (Good): The ACTUAL_OUTPUT is accurate, complete, uses information properly, and truly helps the user.
Score 0 (Bad): The ACTUAL_OUTPUT has major errors, missing parts, ignores important info, or doesn't help the user.
"""
```

> [!NOTE]
>
> When designing judge prompts, be specific about your scoring criteria and reference the evaluation parameters explicitly. This ensures consistent and reliable evaluations.

### Best Suited For

Custom AI Evaluation metrics are ideal when standard built-in metrics don't cover your specific needs:

* **Behavioral Evaluation** — Ensuring model behavior aligns with defined product guidelines or safety constraints
* **Policy Adherence** — Checking compliance with brand tone, content rules, or moderation policies
* **Security Testing** — Validating that responses do not cross ethical, privacy, or safety boundaries
* **Retrieval Use** — Evaluating whether the model made appropriate use of retrieved context

### AI-Generated Metrics

You can also generate metrics directly from your [Specifications](https://docs.galtea.ai/v5.13/concepts/product/specification.md). Select one or more specifications, and the AI produces ready-to-use metric candidates — complete with judge prompt, evaluation parameters, tags, and the platform's default evaluator model (editable before saving). See [AI Metric Generation](https://docs.galtea.ai/v5.13/concepts/metric/ai-generation.md) for the full workflow.

Alternatively, if you already know what you want to measure, write the name and description yourself and let the AI fill in the judge prompt and evaluation parameters with **Complete with AI**. See [Complete a Metric with AI](https://docs.galtea.ai/v5.13/concepts/metric/ai-generation.md#complete-a-metric-with-ai).

## Human Evaluation

Human annotators score responses using the evaluation criteria you define. The metric's `judge_prompt` serves as the evaluation rubric for annotators, and `evaluation_params` determines which data fields are displayed to the evaluator.

When an evaluation is created for a human-evaluated metric, it enters a `PENDING_HUMAN` status and waits for a qualified human evaluator to provide a score.

> [!TIP]
>
> Don't want to write the annotator rubric by hand? Provide the metric's name and description, then use [Complete with AI](https://docs.galtea.ai/v5.13/concepts/metric/ai-generation.md#complete-a-metric-with-ai) to generate guidelines on the 0–100 scale.

### Key Features

* **User Groups**: Link [user groups](https://docs.galtea.ai/v5.13/concepts/user-group.md) to a metric to control which users can annotate evaluations for it. Any user who may edit the metric can link a group of the organization to it.
* **Group Membership**: Users in a linked group see pending evaluations for that metric in the Human Evaluations page. A READER or HUMAN\_EVALUATOR also sees only the evaluations of the products their groups grant. A metric with no linked group reaches nobody.
* **Claim & Submit**: Evaluators can claim pending evaluations to prevent concurrent editing, then submit their score with an optional reason. The UI opens on **Results** (Incorrect, Partially correct or Correct, stored as 0, 0.5 or 1), and also accepts a **Percentage** (0-100) or a **Fraction**. Every score is stored from 0 to 1.

### Use Cases

* Subjective quality assessments that require human judgment
* Edge cases where LLM evaluation may be unreliable
* Building gold standard datasets for training custom evaluators
* Compliance and audit requirements that mandate human review

### SDK Example

```python theme={"system"}
quality_reviewers_group = galtea.user_groups.get_by_name(quality_reviewers_group_name)
metric = galtea.metrics.create(
    name=metric_name,
    source="human_evaluation",
    judge_prompt="Evaluate the quality of the response based on accuracy, completeness, and clarity.",
    evaluation_params=["input", "actual_output", "expected_output"],
    user_group_ids=[quality_reviewers_group.id],
)
```

> [card] [Human Evaluation Tutorial](https://docs.galtea.ai/v5.13/sdk/tutorials/human-evaluation.md)
>
> Step-by-step guide to setting up human evaluation with user groups, from metric creation to annotation.

## Related

> [card] [Evaluation Parameters](https://docs.galtea.ai/v5.13/concepts/metric/evaluation-parameters.md)
>
> Full reference of data fields available to evaluators.

> [card] [Metrics Overview](https://docs.galtea.ai/v5.13/concepts/metric.md)
>
> Browse all available metrics and understand the two metric families.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.