Skip to main content
When creating or configuring a metric, you select which parameters are relevant for your evaluation. These parameters are made available to the evaluator during scoring.
  • For AI Evaluation, the selected parameters are automatically prepended to your judge_prompt.
  • For Human Evaluation, they determine which data fields are displayed to annotators.
  • For Self-Hosted metrics, evaluation parameters do not apply.
  • For Classifier metrics, you do not select parameters. They come from the {{ name }} placeholders in the question, and only a short list of names is allowed.
Only select parameters that your evaluation criteria actually reference. Including unnecessary parameters can reduce evaluation quality by adding noise to the evaluator’s context.

​
Parameter Reference

ParameterDescriptionAvailability
inputThe prompt or query sent to the model.Accuracy, Security & Safety, and Behavior
actual_outputThe actual output generated by the model.Accuracy, Security & Safety, and Behavior
expected_outputThe ideal answer for the given input. A field rule in a JSON expected output is checked only by the JSON Field Match metrics. Every other metric reads it as plain JSON text.Accuracy and Security & Safety
contextAdditional background information provided to the model alongside the input.All metrics
retrieval_contextThe context retrieved by your RAG system before sending the user query to your LLM. Derived from your RETRIEVER spans; you can also supply it directly and it is stored as a RETRIEVER span, not a directly-readable field.Accuracy, Security & Safety, and Behavior
tracesExecution traces from the agent, including tool calls, LLM invocations, and other internal operations.All metrics
expected_toolsList of tools expected to be used by the agent to accomplish the task.All metrics
tools_usedList of tools actually used by the agent during execution (automatically inferred from traces).All metrics
product_descriptionThe description of the product.All metrics
product_capabilitiesThe capabilities of the product.All metrics
product_inabilitiesThe product’s known inabilities or restrictions.All metrics
product_security_boundaries (deprecated)The security boundaries of the product. Deprecated — no longer accepted for new metrics. Use product_capabilities and product_inabilities instead.N/A (rejected by API for new metrics)
user_personaInformation about the user interacting with the agent.Behavior tests
goalThe user’s objective in the conversation.Behavior tests
scenarioThe context or situation for the conversation.Behavior tests
stopping_criteriasList of criteria that define when a conversation should end.Behavior tests
conversation_turnsAll turns in a conversation, including user and assistant messages.Behavior tests (Human Evaluation only)

​
Troubleshooting skipped evaluations

When an evaluation runs against a session that does not have all the data the metric needs, Galtea marks the evaluation as SKIPPED instead of producing a misleading score. The evaluation’s error field describes which parameters are missing, grouped by where you provide them. The categorized error message looks like this:
Each section points to a different place to look:
SectionWhere to fix it
Cannot be read by the judgeThe value is there, but the judge would not read it. A file that the metric’s judge model does not read skips the evaluation, whatever text sits beside it, and the reason names the file, its turn and what refused it. The Run Evaluation dialog greys those metrics out before the run, with the same reason under the name. The file rule on the Trace page lists every cause and the fix.
Missing from product settingsOpen the product and fill in the missing field (Description, Capabilities, or Inabilities). Capabilities and Inabilities can also be derived from the product’s Specifications.
Missing from test caseEdit the test case and provide the missing value. Some fields (Goal, User Persona, Scenario, Stopping Criteria) are only populated on SCENARIOS-type test cases. If your test cases use a different type, use a metric that does not require these parameters.
Missing from endpoint connection’s output mappingEdit the endpoint connection and add the missing key to the output mapping. actual_output is extracted from the output key. See Templates & Mapping for the JSONPath syntax used in mapping values.
Missing trace dataThe session has no conversation turns yet. Run the metric against a session that contains at least one turn.
Missing span dataThe metric needs span-derived data (traces, tools_used, or retrieval_context). Configure your product to send span data — see the Tracing Agent Operations tutorial for setup, and the Span concept page for what gets captured. tools_used (from TOOL spans) and retrieval_context (from RETRIEVER spans) are automatically extracted from your span data; you do not provide them directly. One turn with the data is enough: a retrieval_context metric is skipped only when no turn of the session has a RETRIEVER span with content.
A skipped evaluation consumes no credits. The skips listed above are decided before the evaluation reaches the evaluator. The evaluator can also skip one. Two examples are a metric that does not apply to the session where the applicability check is on, and, on deployments that do not send files to the judge, a metric whose judge model would read a Galtea file on its turns. Those skips consume no credits either. Spans can reach Galtea a few seconds after the turn they belong to. So on a run, Galtea waits up to about 9 seconds for late spans before it decides. It does not wait when every turn that misses a span finished more than about 9 seconds ago, because late spans for those turns would already have arrived. When it skips after a wait, the message ends with a line such as Waited ~9 s for late spans. A production monitor applies the same wait rule and the same other rules as a run, including the Cannot be read by the judge rule and the single-turn limit of deterministic metrics. A monitor decides a session up to about two minutes after the session closes, so it rarely waits. Once you have provided the missing data, re-run the evaluation against the same session. A Cannot be read by the judge section is the exception: no data is missing, so re-running changes nothing until you pick a different metric or log the turn with the text you want scored, without the file.

Evaluation Types

Understand AI Evaluation, Human Evaluation, and Self-Hosted scoring.

Metrics Overview

Browse all available metrics and create custom ones.