Test Case
A single challenge for evaluating product performance
What is a Test Case?
A test case in Galtea is a challenge designed to evaluate the performance of a product. It represents a specific input and context sent to an AI model alongside an expected output that can be used to assess the product’s capabilities.
Using Test Cases in Evaluations
Test cases are used with evaluation tasks to assess the performance of specific versions of your product.
Create an Evaluation
Learn how to use tests in evaluations
SDK Integration
Test Case Service SDK
SDK methods for managing test cases
Test Case Properties
When creating a test case in Galtea, you’ll need to provide the following information:
The ID of the test you want to create the test case for.
The input data used for inference on the LLM product’s version. Example: “How much does the iPhone 16 cost?”
The expected output for the evaluation task, extracted from the test file. Example: “The iPhone 16 costs $999.”
Additional context provided to the product’s version upon inference alongside the input. Example: “[previous messages of the conversation]”
It is test-case-specific and provides context to the model. Do not mistake for the system prompt/few-shot examples; those should be defined in the product’s version unless they change on an interaction basis.
The tag for the test case. This is a label that can help you categorize or identify the test case.
The original source text used to create the test case. This field is particularly useful for tracing the origin of the information used in the test case.
Typically, this is only filled if the test case is created by Galtea’s Test Case Generator, but you can also use it to provide the source of the test case if you created it manually.