Skip to main content
You are reading the docs of Galtea 5.17. Read this page in the latest version.
Some products read documents: a scanned contract, an invoice, a report. You attach the document to the test case, and Galtea stores it. Galtea does not answer from the document for you. Reading it is your product’s job. This page walks the round trip your own pipeline drives: attach the documents, list the test cases, download each attached file, answer with your own pipeline, and score the answer. Galtea can also deliver the documents to your endpoint and run the inference for you. Name the files in the endpoint connection template with {{ input_files[0].url }} or {{ input_files[0].base64 }}, then run the dataset the usual way. Use the pipeline below when your product is not reachable over HTTP.
A metric scores a file only when its judge model reads that kind of file. Only a Galtea file, one you uploaded to Galtea or one that Galtea stored from inline media, is sent to a judge. A file on your own host is stored as sent and is never sent to a judge as a file, so it never skips an evaluation: the judge reads its entry as JSON text, in its place in the turn. The kinds a judge reads depend on its model, and the model picker of the metric form shows them for each model. A judge is sent PNG, JPEG, WebP, GIF and PDF files only. The other accepted types are stored, and a Galtea input file of those types still reaches your endpoint, but no judge reads them. The rule is the same for the input and the output, and a metric that reads conversation_turns reads both sides of every turn. A metric that reads traces also reads the Galtea files of every span, and a metric that reads retrieval_context reads the Galtea files in the output of RETRIEVER spans.Galtea numbers the files of a session. Turn files come first, in turn order, input before output. The files of every span come next, in span start order. A file that sits in several places of a session takes one number, at its first place, and the judge receives it once. The judge reads each file in its place in the text as [File N], and the session page and the span page show the same File N beside each Galtea file.Before an evaluation runs, Galtea checks every Galtea file on each side the metric reads. One file the judge would not read skips the evaluation, whatever text sits beside it. A skipped evaluation costs no credits, and its reason names the file with its number, its turn and what refused it:
  • The judge model cannot read that kind of file.
  • The model catalog does not say whether the judge model reads it.
  • The metric names no judge model.
  • The metric is DeepEval-based: those judges are sent text only.
  • The metric is deterministic: it compares text without a model.
  • The metric is a classifier metric: it is sent text only.
  • The file is a type no judge reads, such as DOCX, CSV or TIFF, or an audio file: a judge reads audio only through its transcript.
When the evaluator then reads a Galtea file to send it, it skips the evaluation if the file no longer exists in storage, is empty, or is larger than the 20 MB a judge can receive. The reason names the file.Audio input with no transcript is a separate case. A metric that reads the input is skipped when no turn of the session has input text or a Galtea input file, and a turn’s audio carries no transcript. Text or a Galtea file on another turn is enough, and the reason, Evaluators read an audio input through its transcript, and no turn carries one., names no file.Human evaluation and self-hosted metrics are never skipped for a file or for audio. On deployments that do not send files to the judge, the evaluator also skips a metric whose judge would read the file, and the reason names the file.To have the content scored, pick a metric whose judge model reads that kind of file, or log the text your pipeline extracted as the turn input or output, without the file. See Troubleshooting skipped evaluations.
A file test case runs as a single turn, so the rule above applies to every metric that reads the input. The manual create_and_evaluate(..., input=...) loop below can log your pipeline’s extracted text as the turn input, without the file, and the metric scores that text. evaluations.run() cannot override the input, so the turn keeps the file and the rule above decides whether the metric scores it or skips. Otherwise score the output only, with a metric such as JSON Field Match.

Workflow

1

Attach the documents

Add an input_file_paths column to your dataset CSV. The SDK uploads each file and attaches it to that row’s input.
2

Read the test cases back

galtea.test_cases.list() returns the text in input and the attachments in input_files.
3

Download each attached file

galtea.storage.download() saves a file locally and returns the path it wrote.
4

Answer and score

Run your own pipeline over the local files, then log the answer and evaluate it.

1. Attach the documents

Name the local files in an input_file_paths column. Separate several paths with ; or |. A row may attach files and leave input empty, which is how you write a test case that is only a document:
Each expected_output here is a JSON object, because the metric below compares it field by field. CSV escapes a quote by doubling it, which is why every " inside those cells appears as "".
The SDK uploads each file and rewrites the row before it sends anything, so the column never reaches the platform and your local paths stay private. A file named by several rows is uploaded once.
To attach a document to one test case instead of a whole dataset, pass input_file_paths to test_cases.create().

Check some fields by rule

An expected field can hold a field rule when the exact value is not what you test. For a document that holds an ID number, a name, an address and a birth date, write the expected_output like this:
Each backslash is written twice because the value is JSON. In a CSV cell, each " is also doubled. This example is not part of the tutorial dataset.

2. Write your pipeline

This is the step Galtea cannot do. It takes the question and the local file paths, and returns whatever your product would answer:

3. Read, download, answer, score

Four details in that loop are worth knowing:
  • test_case.input is None for a document-only test case. The document is the whole input, so there is no text to read. Use test_case.input_data when you need the full structured input, including the file parts.
  • include_legacy=False returns only the current revision of each test case. Leave it at its default and an edited test case comes back once per revision, so you download the same document again for each one.
  • The file is saved under the name you uploaded it with. Storage keys are random ids, so without filename you would get 9f3c1a.pdf on disk. download() reads the name from the AttachedFile.
  • A failed download raises. The message names the file and the cause, never the presigned link. Wrap the loop in try/except if one unreadable document should not stop the rest.

4. The same round trip through evaluations.run()

When the dataset is linked to a specification, one call replaces the loop above. Give evaluations.run() an agent function whose first parameter is annotated AgentInput, and the attached files arrive as input_data.input_files, the same AttachedFile objects test_case.input_files returns:
Only this signature receives the files. A (str) -> str agent receives the question alone, so the SDK warns once when a test case also attaches files and fails the trace for a document-only test case. See Reading attached files.
galtea.storage.download() also takes a plain URI, so it fetches any file your organization uploaded, not only test case attachments. See Download File.

Next Steps

File inputs

The limits, the accepted file types, and how editing a document test case works.

Storage Service

Upload and download files directly, for any purpose.

Run Dataset-Based Evaluations

The same loop for test cases that are plain text.

JSON Field Match

Score a structured answer field by field, without reading the input.