Returns
Returns a Metric object for the given parameters, orNone if an error occurs. When the platform stores it but cannot return it, create raises WrittenResourceUnreadableException instead: do not create it again.
Examples
- AI Evaluation
- Human Evaluation
- Self-Hosted
- Classifier
Parameters
The name of the model used to evaluate the metric. Required for metrics using
judge_prompt.Available models:"claude-opus-5-5""claude-sonnet-5-5""claude-opus-5""claude-sonnet-5""claude-sonnet-4-6""claude-sonnet-4-5""claude-haiku-4-5""GPT-6.1-Sol""GPT-6-Sol""GPT-6-Luna""GPT-5.6-Sol""GPT-5.6-Terra""GPT-5.6-Luna""GPT-5.5""GPT-5.4""GPT-5.2""GPT-5.1""GPT-5""GPT-5-mini""GPT-4.1""GPT-4.1-mini""GPT-4o""o3""Gemini-3.5-Flash""Gemini-3.1-Flash-Lite""Gemini-2.5-Pro""Gemini-2.5-Flash""Gemini-2.5-Flash-Lite""DeepSeek-V4-Pro""DeepSeek-V4-Flash"
"eu.claude-sonnet-4-6""eu.claude-sonnet-4-5""eu.claude-haiku-4-5""eu.gpt-5.5""eu.gpt-5.4""eu.gpt-5.1""eu.gpt-5""eu.gpt-4.1""eu.gpt-4.1-mini""eu.gpt-4o""eu.gpt-4o-mini""eu.o3"
It should not be provided if the metric is “self hosted” (has no
judge_prompt) since it does not require a model for evaluation. Optional when source is classifier and your deployment has one selectable classifier model (such as jev-1.13.0): leave it out, and the API uses that model. With several selectable classifier models, the API answers 400 until you name one. A name you send must be a selectable classifier model of your deployment.A custom prompt that defines the evaluation logic for the metric. For AI Evaluation metrics, write the evaluation criteria and scoring rubric — Galtea will prepend the selected
evaluation_params automatically. For Human Evaluation metrics, this serves as the annotation rubric. If omitted, the metric is considered a deterministic “Custom Score” metric. Must not be provided when source is classifier.The evaluation method for the metric. Possible values:
"partial_prompt"— AI Evaluation: You provide the core evaluation criteria and rubric. Galtea dynamically constructs the final prompt by prepending selected evaluation parameters to your criteria."human_evaluation"— Human Evaluation: Human annotators manually review and score evaluations using the annotation criteria you define. Evaluations enter aPENDING_HUMANstatus and are completed when an annotator submits a score."self_hosted"— Self-Hosted: For deterministic metrics scored locally using the SDK’sCustomScoreEvaluationMetric. Your custom logic runs on your infrastructure, and the resulting score is uploaded to the platform."classifier"— Classifier: a classifier model answers one typed question, which requiresclassifier_question. On a deployment without the classifier, the API answers 400 andcreatereturnsNone.
Evaluation parameters to include in the judge prompt. These parameters are prepended to the judge prompt to construct the final evaluation prompt.
To check the available evaluation parameters, see the Evaluation Parameters section.
Only applicable for AI Evaluation and Human Evaluation metrics. For
classifier, leave it out: the API derives it from the {{ param }} placeholders in every text of the question. If you send it, it must list exactly those names. Only the allowed placeholders are accepted.The one question a classifier metric asks. Required when
source is classifier, and refused for every other source. Write it in English. A plain dict with snake_case or camelCase keys is also accepted; it needs a type of "yes_no" or "scale".ClassifierYesNoQuestion:instructions: 1 to 2000 characters, naming at least one evaluation parameter as a{{ name }}placeholder.good_answer:"yes"or"no". Required, with no default.criteria: optionalClassifierYesNoCriteria(yes, no), each side at most 1000 characters, to refine what yes and no mean.
ClassifierScaleQuestion:instructions: same rules as above.levels: a list ofClassifierScaleLevel(description, score), worst first. 2 to 10 levels, or 2 to 9 withdoes_not_apply. Eachdescriptionis 1 to 1000 characters. Eachscoreis from 0 to 1 and must not go down from the worst level to the best. Set a score on every level, or on none: with none, the API spaces the scores evenly.evenly_spaced_levels(descriptions)builds the levels with those scores.does_not_apply: optional text of an extra level, 1 to 1000 characters. When it is the most probable answer, the evaluation is skipped.
does_not_apply text. The SDK sends the question as given, and the API checks these rules. When the API refuses it, create prints the API’s message and returns None.The score, from 0 to 1, at or above which an evaluation of this metric passes. When omitted, the API uses
0.5. It is fixed when you create the metric: to use another one, create a revision of the metric. Read it back as metric.pass_threshold. See Pass threshold.Which sessions get the applicability check before the metric is judged. A session where the metric does not apply is skipped, with no judge call and no credits spent. Possible values, in any case:
"production": only production sessions. This is the default for a metric the check covers:partial_prompt,classifier, deterministic, and the DeepEval-based metrics of Galtea."development": only development sessions, which are the sessions that are not production."all": every session."none": no check. This is the default for aself_hosted,human_evaluationor G-Eval (deprecated) metric.
"none" for a self_hosted, human_evaluation or G-Eval (deprecated) metric, and create returns None. A metric that scores traces never gets the check: it defaults to "none" and refuses any other value. For a classifier metric, the fields it scores are the placeholders of its question, so a traces placeholder turns the check off. The check runs only where your deployment enables it. The scope is fixed when you create the metric: to use another one, create a revision of the metric. Read it back as metric.applicability_scope.The lowest probability, from 0 to 1, that the metric applies to a session at which the metric is judged. A session below it is skipped. When omitted, the API uses
0.5, and metric.applicability_threshold reads None. The API refuses it for a self_hosted, human_evaluation or G-Eval (deprecated) metric, and for a metric that scores traces. It is fixed when you create the metric: to use another one, create a revision of the metric. See Applicability check.A list of user group IDs to associate with this metric. Only applicable when
source is human_evaluation. Each group must belong to your organization.- Only users in the linked groups can annotate evaluations for this metric.
- A metric with no linked group reaches nobody: its pending evaluations show up for no one, and its metric page shows a warning.
The generation settings the judge runs with, instead of the platform defaults. Only these four keys are accepted, and any you omit keep the provider’s own default:
temperaturetop_pmax_output_tokens: at least 256, so the judge’s structured answer fits. If the judge’s answer is still cut off at this limit, the evaluation is skipped with a message that names it.reasoning_effort— a level the model accepts, such aslow,mediumorhigh. The set differs per model, so read the settings your evaluator model publishes before choosing one.
A value the model rejects as out of range, such as a It sends one small completion to the judge model with your settings and consumes no credits.
The
temperature above the provider’s own
ceiling, is reported per evaluation: that evaluation ends as SKIPPED carrying the provider’s own
message, and is never scored at a substituted value.To find a rejected value before you create the metric, check it with the
galtea CLI. The SDK has no evaluator model surface, so the CLI is the only way
to reach this check:outcome field answers accepted, refused (with the provider’s own message in error)
or unreachable. accepted tells you only that the provider took the values, never that they
changed how the judge answers.Each CLI version ships with the commands of the API at its release, so this command appears only
in a CLI version built from an API release that carries it. Update the CLI to get it.