Skip to main content
You are reading the docs of Galtea 5.15. Read this page in the latest version.

Returns

Returns a tuple containing:
  1. A Trace object
  2. A list of Evaluation objects, one for each metric evaluated

Usage

This method combines creating a trace with its evaluation in a single convenient call. It’s the recommended approach for single-turn evaluations, replacing the deprecated galtea.evaluations.create_single_turn() method.

Basic Example

Evaluate by passing Specification IDs so the API resolves their linked metrics automatically (recommended), or by listing metric names explicitly.

With Pre-computed Scores

With Custom Score Calculation

Parameters

string
The session ID to log the trace to.
Instead of a session_id, you can reference an existing session by its own conversation id (see the section below). Provide exactly one of the two.
If you already created the session via the sessions API with your own custom_id, reference it here with session_custom_id plus a version_id or product_id to log and evaluate a turn in one call. If no matching session exists, the call returns a 400 error — create it first via the sessions API.
string
Your own conversation id, as an alternative to session_id. Galtea finds an existing session with this id (per version); the session is not created here. Requires version_id or product_id.
string
The version the session is anchored to when using session_custom_id. If both anchors are given, version_id wins and product_id is ignored.
string
Product anchor used when no version_id is given with session_custom_id: Galtea finds the session with this custom id under the product (across its versions).
If no session matches the session_custom_id and anchor, the call returns a 400 error. Create the session first via the sessions API.
string | object
The generated output/response from the AI model. A dict is stored as sent. Galtea reads each element of its content list on its own: a Galtea file is stored without its upload signature and must be an accepted type, with at most 20 files and 20 MB. Every other element, such as a file URL on your own host, is stored as sent. A Galtea file of another organization fails the call with a 400. A metric scores a Galtea file only when its judge model reads that kind of file. See Content Parts.Required unless output_file_paths carries the whole answer. Give one of the two, or the call raises a ValueError.
list[str]
Files the turn answers with, for a product whose reply is a document. Each entry is a local path the SDK uploads, or a Galtea file reference from an earlier upload, which is attached without uploading it again. Any other URL raises a ValueError before anything is uploaded.One output accepts at most 20 files, 20 MB in total, of the same types an input accepts. See Upload Output File for the list and for reusing one upload across turns.A custom scoring function still receives the text of the output, never the files.
A metric scores a file only when its judge model reads that kind of file. Only a Galtea file, one you uploaded to Galtea, is sent to a judge. A file on your own host is stored as sent and never reaches a judge, so it never skips an evaluation: the judge reads the text beside it. The kinds a judge reads depend on its model, and the model picker of the metric form shows them for each model. A judge is sent PNG, JPEG, WebP, GIF and PDF files only. The other accepted types are stored, and a Galtea input file of those types still reaches your endpoint, but no judge reads them. The rule is the same for the input and the output, and a metric that reads conversation_turns reads both sides of every turn.Before an evaluation runs, Galtea checks every Galtea file on each side the metric reads. One file the judge would not read skips the evaluation, whatever text sits beside it. A skipped evaluation costs no credits, and its reason names the file, its turn and what refused it:
  • The judge model cannot read that kind of file.
  • The model catalog does not say whether the judge model reads it.
  • The metric names no judge model.
  • The metric is DeepEval-based: those judges are sent text only.
  • The metric is deterministic: it compares text without a model.
  • The metric is a classifier metric: it is sent text only.
  • The file is a type no judge reads, such as DOCX, CSV or TIFF, or an audio file: a judge reads audio only through its transcript.
When the evaluator then reads a Galtea file to send it, it skips the evaluation if the file no longer exists in storage, is empty, or is larger than the 20 MB a judge can receive. The reason names the file.Audio input with no transcript is a separate case. A metric that reads the input is skipped when no turn of the session has input text or a Galtea input file, and a turn’s audio carries no transcript. Text or a Galtea file on another turn is enough, and the reason, Evaluators read an audio input through its transcript, and no turn carries one., names no file.Human evaluation and self-hosted metrics are never skipped for a file or for audio. On deployments that do not send files to the judge, the evaluator also skips a metric whose judge would read the file, and the reason names the file.To have the content scored, pick a metric whose judge model reads that kind of file, or log the text your pipeline extracted as the turn input or output, without the file. See Troubleshooting skipped evaluations.
list[str | CustomScoreEvaluationMetric | dict]
A list of metrics to evaluate against. Supports multiple formats:
  • Strings: Metric names (e.g., ["accuracy", "relevance"])
  • CustomScoreEvaluationMetric: Objects with dynamic score calculation. Must be initialized with either ‘name’ or ‘id’ parameter.
  • MetricInput dicts: Format with optional id, name, and score.
    • If score is a float: Pre-calculated score (requires ‘id’ or ‘name’ in the dict).
    • If score is a CustomScoreEvaluationMetric: Dynamic score calculation.
Optional when specification_ids is provided (in which case metrics are resolved from the specifications).
list[str]
A list of Specification IDs. When provided, the evaluation uses the metrics linked to these specifications.Can be combined with metrics — the API merges and deduplicates by metric ID.
This parameter allows you to evaluate against specific product specifications without manually listing all their associated metrics. At least one of metrics or specification_ids must be provided.
string | object
The input text/prompt. If not provided, will be inferred from the test case linked to the session. A dict is stored as sent. Galtea reads each element of its content list on its own: a Galtea file is stored without its upload signature and must be an accepted type, with at most 20 files and 20 MB. Every other element, such as a file URL on your own host, is stored as sent. A Galtea file of another organization fails the call with a 400. A metric scores a Galtea file only when its judge model reads that kind of file. See Content Parts.
list[str]
Files the turn received, for a product that reads documents. Each entry is a local path the SDK uploads, or a Galtea file reference from an earlier upload, which is attached without uploading it again. Any other URL raises a ValueError before anything is uploaded.On a dataset session with no input, the files are added to the test case input. A production session accepts files with no input: the input is then the files alone. A production session with neither raises a ValueError. With session_custom_id and only a product_id, also pass input.One input accepts at most 20 files, 20 MB in total, of the same types a test case input accepts. See Upload Output File for the list.Input files follow the same rule as output_file_paths above.
string
Context retrieved by a RAG system, if applicable.
float
Latency in milliseconds from model invocation to response.
dict[str, int]
Token usage information from the model call. Supported keys: input_tokens, output_tokens, cache_read_input_tokens.
dict[str, float]
Cost breakdown for the model call. Supported keys: cost_per_input_token, cost_per_output_token, cost_per_cache_read_input_token.
string
Version of Galtea’s conversation simulator used to generate the input.
string
The Run the evaluations join. Defaults to the run of the enclosing galtea.runs.start() block, when there is one. An explicit value wins over the block.