{{ input_files[0].url }} or {{ input_files[0].base64 }}, then run the dataset the usual way. Use the pipeline below when your product is not reachable over HTTP.
A metric scores a file only when its judge model reads that kind of file. Only a Galtea file, one you uploaded to Galtea or one that Galtea stored from inline media, is sent to a judge. A file on your own host is stored as sent and is never sent to a judge as a file, so it never skips an evaluation: the judge reads its entry as JSON text, in its place in the turn. The kinds a judge reads depend on its model, and the model picker of the metric form shows them for each model. A judge is sent PNG, JPEG, WebP, GIF and PDF files only. The other accepted types are stored, and a Galtea input file of those types still reaches your endpoint, but no judge reads them. The rule is the same for the input and the output, and a metric that reads
conversation_turns reads both sides of every turn. A metric that reads traces also reads the Galtea files of every span, and a metric that reads retrieval_context reads the Galtea files in the output of RETRIEVER spans.Galtea numbers the files of a session. Turn files come first, in turn order, input before output. The files of every span come next, in span start order. A file that sits in several places of a session takes one number, at its first place, and the judge receives it once. The judge reads each file in its place in the text as [File N], and the session page and the span page show the same File N beside each Galtea file.Before an evaluation runs, Galtea checks every Galtea file on each side the metric reads. One file the judge would not read skips the evaluation, whatever text sits beside it. A skipped evaluation costs no credits, and its reason names the file with its number, its turn and what refused it:- The judge model cannot read that kind of file.
- The model catalog does not say whether the judge model reads it.
- The metric names no judge model.
- The metric is DeepEval-based: those judges are sent text only.
- The metric is deterministic: it compares text without a model.
- The metric is a classifier metric: it is sent text only.
- The file is a type no judge reads, such as DOCX, CSV or TIFF, or an audio file: a judge reads audio only through its transcript.
Evaluators read an audio input through its transcript, and no turn carries one., names no file.Human evaluation and self-hosted metrics are never skipped for a file or for audio. On deployments that do not send files to the judge, the evaluator also skips a metric whose judge would read the file, and the reason names the file.To have the content scored, pick a metric whose judge model reads that kind of file, or log the text your pipeline extracted as the turn input or output, without the file. See Troubleshooting skipped evaluations.create_and_evaluate(..., input=...) loop below can log your pipeline’s extracted text as the turn input, without the file, and the metric scores that text. evaluations.run() cannot override the input, so the turn keeps the file and the rule above decides whether the metric scores it or skips. Otherwise score the output only, with a metric such as JSON Field Match.
Workflow
1
Attach the documents
Add an
input_file_paths column to your dataset CSV. The SDK uploads each file and attaches it to that row’s input.2
Read the test cases back
galtea.test_cases.list() returns the text in input and the attachments in input_files.3
Download each attached file
galtea.storage.download() saves a file locally and returns the path it wrote.4
Answer and score
Run your own pipeline over the local files, then log the answer and evaluate it.
1. Attach the documents
Name the local files in aninput_file_paths column. Separate several paths with ; or |. A row may attach files and leave input empty, which is how you write a test case that is only a document:
expected_output here is a JSON object, because the metric below compares it field by field. CSV escapes a quote by doubling it, which is why every " inside those cells appears as "".
Check some fields by rule
An expected field can hold a field rule when the exact value is not what you test. For a document that holds an ID number, a name, an address and a birth date, write theexpected_output like this:
" is also doubled. This example is not part of the tutorial dataset.
2. Write your pipeline
This is the step Galtea cannot do. It takes the question and the local file paths, and returns whatever your product would answer:3. Read, download, answer, score
test_case.inputisNonefor a document-only test case. The document is the whole input, so there is no text to read. Usetest_case.input_datawhen you need the full structured input, including the file parts.include_legacy=Falsereturns only the current revision of each test case. Leave it at its default and an edited test case comes back once per revision, so you download the same document again for each one.- The file is saved under the name you uploaded it with. Storage keys are random ids, so without
filenameyou would get9f3c1a.pdfon disk.download()reads the name from theAttachedFile. - A failed download raises. The message names the file and the cause, never the presigned link. Wrap the loop in
try/exceptif one unreadable document should not stop the rest.
4. The same round trip through evaluations.run()
When the dataset is linked to a specification, one call replaces the loop above. Give evaluations.run() an agent function whose first parameter is annotated AgentInput, and the attached files arrive as input_data.input_files, the same AttachedFile objects test_case.input_files returns:
(str) -> str agent receives the question alone, so the SDK warns once when a test case also attaches files and fails the trace for a document-only test case. See Reading attached files.
galtea.storage.download() also takes a plain URI, so it fetches any file your organization uploaded, not only test case attachments. See Download File.Next Steps
File inputs
The limits, the accepted file types, and how editing a document test case works.
Storage Service
Upload and download files directly, for any purpose.
Run Dataset-Based Evaluations
The same loop for test cases that are plain text.
JSON Field Match
Score a structured answer field by field, without reading the input.