Skip to main content
You are reading the docs of Galtea 5.10. Read this page in the latest version.

Returns

Returns a Metric object for the given parameters, or None if an error occurs.

Examples

Parameters

string
required
The name of the metric.
string
Deprecated. This parameter is ignored and will be removed in a future release.
string
The name of the model used to evaluate the metric. Required for metrics using judge_prompt.Available models:
  • "claude-opus-5-5"
  • "claude-sonnet-5-5"
  • "claude-opus-5"
  • "claude-sonnet-5"
  • "claude-sonnet-4-6"
  • "claude-sonnet-4-5"
  • "claude-haiku-4-5"
  • "GPT-6.1-Sol"
  • "GPT-6-Sol"
  • "GPT-6-Luna"
  • "GPT-5.6-Sol"
  • "GPT-5.6-Terra"
  • "GPT-5.6-Luna"
  • "GPT-5.5"
  • "GPT-5.4"
  • "GPT-5.2"
  • "GPT-5.1"
  • "GPT-5"
  • "GPT-5-mini"
  • "GPT-4.1"
  • "GPT-4.1-mini"
  • "GPT-4o"
  • "o3"
  • "Gemini-3.5-Flash"
  • "Gemini-3.1-Flash-Lite"
  • "Gemini-2.5-Pro"
  • "Gemini-2.5-Flash"
  • "Gemini-2.5-Flash-Lite"
  • "DeepSeek-V4-Pro"
  • "DeepSeek-V4-Flash"
EU data zone. These are the same models, served from European infrastructure. Use them when your data must stay in the EU:
  • "eu.claude-sonnet-4-6"
  • "eu.claude-sonnet-4-5"
  • "eu.claude-haiku-4-5"
  • "eu.gpt-5.5"
  • "eu.gpt-5.4"
  • "eu.gpt-5.1"
  • "eu.gpt-5"
  • "eu.gpt-4.1"
  • "eu.gpt-4.1-mini"
  • "eu.gpt-4o"
  • "eu.gpt-4o-mini"
  • "eu.o3"
Model names are case-insensitive.
It should not be provided if the metric is “self hosted” (has no judge_prompt) since it does not require a model for evaluation. Optional when source is classifier: leave it out, and the API picks the classifier model (such as jev-1.13.0). A name you send must be a classifier model of your deployment.
string
A custom prompt that defines the evaluation logic for the metric. For AI Evaluation metrics, write the evaluation criteria and scoring rubric — Galtea will prepend the selected evaluation_params automatically. For Human Evaluation metrics, this serves as the annotation rubric. If omitted, the metric is considered a deterministic “Custom Score” metric. Must not be provided when source is classifier.
string
The evaluation method for the metric. Possible values:
  • "partial_prompt" — AI Evaluation: You provide the core evaluation criteria and rubric. Galtea dynamically constructs the final prompt by prepending selected evaluation parameters to your criteria.
  • "human_evaluation" — Human Evaluation: Human annotators manually review and score evaluations using the annotation criteria you define. Evaluations enter a PENDING_HUMAN status and are completed when an annotator submits a score.
  • "self_hosted" — Self-Hosted: For deterministic metrics scored locally using the SDK’s CustomScoreEvaluationMetric. Your custom logic runs on your infrastructure, and the resulting score is uploaded to the platform.
  • "classifier" — Classifier (Jev): TypeSafe’s Jev answers one typed question, which requires classifier_question. On a deployment without the classifier, the API answers 400 and create returns None.
Other string values are sent to the API unchanged, so values added in newer API versions can be used; the API rejects invalid ones.
list[string]
Evaluation parameters to include in the judge prompt. These parameters are prepended to the judge prompt to construct the final evaluation prompt. To check the available evaluation parameters, see the Evaluation Parameters section.
Only applicable for AI Evaluation and Human Evaluation metrics. For classifier, leave it out: the API derives it from the {{ param }} placeholders in every text of the question. If you send it, it must list exactly those names. Only the allowed placeholders are accepted.
ClassifierYesNoQuestion | ClassifierScaleQuestion
The one question a classifier metric asks. Required when source is classifier, and refused for every other source. Write it in English. A plain dict with snake_case or camelCase keys is also accepted; it needs a type of "yes_no" or "scale".ClassifierYesNoQuestion:
  • instructions: 1 to 2000 characters, naming at least one evaluation parameter as a {{ name }} placeholder.
  • good_answer: "yes" or "no". Required, with no default.
  • criteria: optional ClassifierYesNoCriteria(yes, no), each side at most 1000 characters, to refine what yes and no mean.
ClassifierScaleQuestion:
  • instructions: same rules as above.
  • levels: a list of ClassifierScaleLevel(description, score), worst first. 2 to 10 levels, or 2 to 9 with does_not_apply. Each description is 1 to 1000 characters. Each score is from 0 to 1 and must not go down from the worst level to the best. Set a score on every level, or on none: with none, the API spaces the scores evenly. evenly_spaced_levels(descriptions) builds the levels with those scores.
  • does_not_apply: optional text of an extra level, 1 to 1000 characters. When it is the most probable answer, the evaluation is skipped.
Placeholders may also appear in the yes/no criteria, the level descriptions and the does_not_apply text. The SDK sends the question as given, and the API checks these rules. When the API refuses it, create prints the API’s message and returns None.
float
The score, from 0 to 1, at or above which an evaluation of this metric passes. When omitted, the API uses 0.5. It is fixed when you create the metric: to use another one, create a revision of the metric. Read it back as metric.pass_threshold. See Pass threshold.
list[string]
A list of user group IDs to associate with this metric. Only applicable when source is human_evaluation. Each group must belong to your organization.
  • Only users in the linked groups can annotate evaluations for this metric.
  • A metric with no linked group reaches nobody: its pending evaluations show up for no one, and its metric page shows a warning.
list[string]
Tags to categorize the metric.
string
A brief description of what the metric evaluates.
string
A URL pointing to more detailed documentation about the metric.
object
The generation settings the judge runs with, instead of the platform defaults. Only these four keys are accepted, and any you omit keep the provider’s own default:
  • temperature
  • top_p
  • max_output_tokens
  • reasoning_effort — a level the model accepts, such as low, medium or high. The set differs per model, so read the settings your evaluator model publishes before choosing one.
Only applicable to AI Evaluation metrics, which are the only ones with an LLM judge. The settings cannot be changed later: create a revision of the metric to use different ones.
Check which settings your model accepts before you set them. Each model accepts a different subset. A setting the model does not accept is discarded when the judge runs, and the evaluation still succeeds, so nothing tells you it had no effect.On the CLI, list the models with galtea evaluator-models list. Each one reports generationCapabilities.configurableSettings, the settings it accepts. A model whose generationCapabilities is null is one we could not read, so no setting is guaranteed to apply.
A value the model rejects as out of range, such as a temperature above the provider’s own ceiling, is reported per evaluation: that evaluation ends as SKIPPED carrying the provider’s own message, and is never scored at a substituted value.To find a rejected value before you create the metric, check it with the galtea CLI. The SDK has no evaluator model surface, so the CLI is the only way to reach this check:
It sends one small completion to the judge model with your settings and consumes no credits. The outcome field answers accepted, refused (with the provider’s own message in error) or unreachable. accepted tells you only that the provider took the values, never that they changed how the judge answers.The CLI builds its command tree from the API specification, so this command appears only after the API release that carries it. Run galtea sync to pick it up.