Returns
Returns a tuple containing:- A Trace object
- A list of Evaluation objects, one for each metric evaluated
Usage
This method combines creating a trace with its evaluation in a single convenient call. It’s the recommended approach for single-turn evaluations, replacing the deprecatedgaltea.evaluations.create_single_turn() method.
Basic Example
Evaluate by passing Specification IDs so the API resolves their linked metrics automatically (recommended), or by listing metric names explicitly.- Using specification IDs
- Using metrics
With Pre-computed Scores
With Custom Score Calculation
Parameters
The session ID to log the trace to.
Instead of a
session_id, you can reference an existing session by its own conversation id (see the section below). Provide exactly one of the two.Log to an existing session by your own conversation ID
Log to an existing session by your own conversation ID
If you already created the session via the sessions API with your own
custom_id, reference it here with session_custom_id plus a version_id or product_id to log and evaluate a turn in one call. If no matching session exists, the call returns a 400 error — create it first via the sessions API.Your own conversation id, as an alternative to
session_id. Galtea finds an existing session with this id (per version); the session is not created here. Requires version_id or product_id.The version the session is anchored to when using
session_custom_id. If both anchors are given, version_id wins and product_id is ignored.Product anchor used when no
version_id is given with session_custom_id: Galtea finds the session with this custom id under the product (across its versions).If no session matches the
session_custom_id and anchor, the call returns a 400 error. Create the session first via the sessions API.The generated output/response from the AI model. A dict is stored as sent. Galtea reads each element of its
content list on its own: a Galtea file is stored without its upload signature and must be an accepted type, with at most 20 files and 20 MB. Every other element, such as a file URL on your own host, is stored as sent. A Galtea file of another organization fails the call with a 400. A metric scores a Galtea file only when its judge model reads that kind of file. See Content Parts.Required unless output_file_paths carries the whole answer. Give one of the two, or the call raises a ValueError.Files the turn answers with, for a product whose reply is a document. Each entry is a local path the SDK uploads, or a Galtea file reference from an earlier upload, which is attached without uploading it again. Any other URL raises a
ValueError before anything is uploaded.One output accepts at most 20 files, 20 MB in total, of the same types an input accepts. See Upload Output File for the list and for reusing one upload across turns.A custom scoring function still receives the text of the output, never the files.A metric scores a file only when its judge model reads that kind of file. Only a Galtea file, one you uploaded to Galtea or one that Galtea stored from inline media, is sent to a judge. A file on your own host is stored as sent and is never sent to a judge as a file, so it never skips an evaluation: the judge reads its entry as JSON text, in its place in the turn. The kinds a judge reads depend on its model, and the model picker of the metric form shows them for each model. A judge is sent PNG, JPEG, WebP, GIF and PDF files only. The other accepted types are stored, and a Galtea input file of those types still reaches your endpoint, but no judge reads them. The rule is the same for the input and the output, and a metric that reads
conversation_turns reads both sides of every turn. A metric that reads traces also reads the Galtea files of every span, and a metric that reads retrieval_context reads the Galtea files in the output of RETRIEVER spans.Galtea numbers the files of a session. Turn files come first, in turn order, input before output. The files of every span come next, in span start order. A file that sits in several places of a session takes one number, at its first place, and the judge receives it once. The judge reads each file in its place in the text as [File N], and the session page and the span page show the same File N beside each Galtea file.Before an evaluation runs, Galtea checks every Galtea file on each side the metric reads. One file the judge would not read skips the evaluation, whatever text sits beside it. A skipped evaluation costs no credits, and its reason names the file with its number, its turn and what refused it:- The judge model cannot read that kind of file.
- The model catalog does not say whether the judge model reads it.
- The metric names no judge model.
- The metric is DeepEval-based: those judges are sent text only.
- The metric is deterministic: it compares text without a model.
- The metric is a classifier metric: it is sent text only.
- The file is a type no judge reads, such as DOCX, CSV or TIFF, or an audio file: a judge reads audio only through its transcript.
Evaluators read an audio input through its transcript, and no turn carries one., names no file.Human evaluation and self-hosted metrics are never skipped for a file or for audio. On deployments that do not send files to the judge, the evaluator also skips a metric whose judge would read the file, and the reason names the file.To have the content scored, pick a metric whose judge model reads that kind of file, or log the text your pipeline extracted as the turn input or output, without the file. See Troubleshooting skipped evaluations.A list of metrics to evaluate against. Supports multiple formats:
- Strings: Metric names (e.g.,
["accuracy", "relevance"]) - CustomScoreEvaluationMetric: Objects with dynamic score calculation. Must be initialized with either ‘name’ or ‘id’ parameter.
- MetricInput dicts: Format with optional id, name, and score.
- If
scoreis afloat: Pre-calculated score (requires ‘id’ or ‘name’ in the dict). - If
scoreis aCustomScoreEvaluationMetric: Dynamic score calculation.
- If
specification_ids is provided (in which case metrics are resolved from the specifications).A list of Specification IDs. When provided, the evaluation uses the metrics linked to these specifications.Can be combined with
metrics — the API merges and deduplicates by metric ID.This parameter allows you to evaluate against specific product specifications without manually listing all their associated metrics. At least one of
metrics or specification_ids must be provided.The input text/prompt. If not provided, will be inferred from the test case linked to the session. A dict is stored as sent. Galtea reads each element of its
content list on its own: a Galtea file is stored without its upload signature and must be an accepted type, with at most 20 files and 20 MB. Every other element, such as a file URL on your own host, is stored as sent. A Galtea file of another organization fails the call with a 400. A metric scores a Galtea file only when its judge model reads that kind of file. See Content Parts.Files the turn received, for a product that reads documents. Each entry is a local path the SDK uploads, or a Galtea file reference from an earlier upload, which is attached without uploading it again. Any other URL raises a
ValueError before anything is uploaded.On a dataset session with no input, the files are added to the test case input. A production session accepts files with no input: the input is then the files alone. A production session with neither raises a ValueError. With session_custom_id and only a product_id, also pass input.One input accepts at most 20 files, 20 MB in total, of the same types a test case input accepts. See Upload Output File for the list.Input files follow the same rule as output_file_paths above.Token usage information from the model call.
Supported keys:
input_tokens, output_tokens, cache_read_input_tokens.Cost breakdown for the model call.
Supported keys:
cost_per_input_token, cost_per_output_token, cost_per_cache_read_input_token.The Run the evaluations join. Defaults to the run of the enclosing
galtea.runs.start() block, when there is one. An explicit value wins over the block.Related
- Create Evaluation - Evaluate an existing session or trace
- Create Trace - Create a trace without immediate evaluation
- Upload Output File - Upload one file and reuse it across turns