Human Evaluation

What is Human Evaluation?

Human Evaluation lets your team manually review and score AI outputs. Instead of an LLM judge, evaluations enter a PENDING_HUMAN status and wait for a human annotator to submit a score through the dashboard. It is especially useful when you need subjective judgment, domain expertise, or a human-in-the-loop quality gate that automated scoring cannot provide.

How It Works

Create a user group — organize the team members who will review outputs
Link products to the group — choose which products the group members can review
Create a Human Evaluation metric — route evaluations to your group instead of an LLM judge
Run evaluations — evaluations enter PENDING_HUMAN status
Annotate in the dashboard — group members review outputs and submit scores

Step 1: Create a User Group

User Groups organize evaluators and control who can score evaluations for specific metrics. Only an OWNER or ADMIN can create a group or change its users. Any user who may edit a metric can link a group to that metric.

user_group = galtea.user_groups.create(
    name="quality-reviewers-" + run_identifier,
    description="Team responsible for reviewing output quality",
)

You can also create groups from the dashboard: navigate to your organization’s Groups tab.

Step 2: Add Users to the Group

Link users by their user IDs. Users in the group will see pending evaluations for linked metrics in their Human Evaluations dashboard page. A READER or HUMAN_EVALUATOR also sees only the evaluations of the products their groups grant.

galtea.user_groups.link_users(
    user_group_id=user_group_id,
    user_ids=[user_id_1],
)

You can find user IDs in the dashboard under Settings > Members, or by using the organization members API.

Step 3: Link Products to the Group

A READER or HUMAN_EVALUATOR sees only the products that one of its groups links. Link the products that the group must review. You can also create the group with all_products=True to link every product of the organization, current and future.

galtea.user_groups.link_products(
    user_group_id=user_group_id,
    product_ids=[product_id_1, product_id_2],
)

You can also link products from the dashboard: open the group and use its Products tab, or set the products (or All products) in the group’s create and edit forms.

Step 4: Create a Human Evaluation Metric

Create a metric with the Human Evaluation type and link it to your user group. When evaluations run against this metric, they enter PENDING_HUMAN status and appear in the linked group members’ dashboards. A metric with no linked group reaches nobody, so always link at least one group before running evaluations against it. A READER or HUMAN_EVALUATOR in that group also needs a group that links the evaluation’s product.

Option A: From the Dashboard

Go to the Metrics creation form and configure:

Evaluation Type — Select Human Evaluation
User Groups — Assign one or more user groups
Evaluation Parameters — Choose which parameters (Input, Expected Output, Actual Output, etc.) annotators will see
Evaluation Guidelines — Write the criteria annotators should follow when scoring

Option B: From the SDK

metric = galtea.metrics.create(
    name="domain-expert-review-" + run_identifier,
    source="human_evaluation",
    judge_prompt="Review the assistant's response for accuracy and helpfulness. Score 1 if the response is correct and useful, 0 if it contains errors or is unhelpful.",
    evaluation_params=["input", "actual_output", "expected_output"],
    user_group_ids=[user_group_id],
    description="Domain expert review of response quality",
)

# Link the metric to the user group
galtea.user_groups.link_metrics(
    user_group_id=user_group_id,
    metric_ids=[metric.id],
)

Step 5: Run Evaluations

Run evaluations using the SDK or from the dashboard just like any other metric. The only difference is that each evaluation will enter PENDING_HUMAN status instead of being scored by an LLM. Link the metric and a Dataset to a Specification once (see Writing Specifications), create a Version for the build you want reviewed, then let galtea.evaluations.run() do the rest: it resolves the specifications, runs your product on every Test Case, and submits the results for review. A specification missing either its metrics or a dataset is skipped silently, so the run evaluates nothing.

# Simulate your product's response
def your_product_function(user_message: str) -> str:
    return f"Model response to: {user_message}"


# One call resolves the specifications, their datasets and metrics, runs your product on
# every test case, and submits the results. Since the metric source is human_evaluation,
# each evaluation lands in PENDING_HUMAN instead of running an LLM judge.
result = galtea.evaluations.run(version_id=version_id, agent=your_product_function)

print(f"Submitted evaluations for {result['testCaseCount']} test cases")

See also Evaluating Conversations for multi-turn reviews, Run Dataset-Based Evaluations when you need per-test-case control, or Direct Inferences and Evaluations from the Platform.

Step 6: Annotate in the Dashboard

Navigate to the Human Evaluations page in the sidebar. Click Start Evaluating to open the annotation dialog. For each evaluation, review the conversation and context, submit a score (0–100, normalized to 0–1), and optionally add a reason.

Managing Groups

Update a Group

user_group = galtea.user_groups.update(
    user_group_id=user_group_id,
    name="senior-quality-reviewers-" + run_identifier,
    description="Senior team for quality reviews",
)

Remove Users or Metrics

galtea.user_groups.unlink_users(
    user_group_id=user_group_id,
    user_ids=[user_id_1],
)

galtea.user_groups.unlink_metrics(
    user_group_id=user_group_id,
    metric_ids=[metric_id_2],
)

Evaluation Types

Understand AI Evaluation, Human Evaluation, and Self-Hosted scoring.

User Group Concept

Learn more about user groups and their properties.

Getting Started

Core Workflows

Production & Monitoring

Advanced

Playbooks

Integrations

Security & Administration

Migration Guides

What is Human Evaluation?

How It Works

Step 1: Create a User Group

Step 2: Add Users to the Group

Step 3: Link Products to the Group

Step 4: Create a Human Evaluation Metric

Option A: From the Dashboard

Option B: From the SDK

Step 5: Run Evaluations

Step 6: Annotate in the Dashboard

Managing Groups

Update a Group

Remove Users or Metrics

Evaluation Types

User Group Concept

​What is Human Evaluation?

​How It Works

​Step 1: Create a User Group

​Step 2: Add Users to the Group

​Step 3: Link Products to the Group

​Step 4: Create a Human Evaluation Metric

​Option A: From the Dashboard

​Option B: From the SDK

​Step 5: Run Evaluations

​Step 6: Annotate in the Dashboard

​Managing Groups

​Update a Group

​Remove Users or Metrics

​Related

Evaluation Types

User Group Concept

What is Human Evaluation?

How It Works

Step 1: Create a User Group

Step 2: Add Users to the Group

Step 3: Link Products to the Group

Step 4: Create a Human Evaluation Metric

Option A: From the Dashboard

Option B: From the SDK

Step 5: Run Evaluations

Step 6: Annotate in the Dashboard

Managing Groups

Update a Group

Remove Users or Metrics

Related