Skip to main content

​
Returns

Returns a Test object for the given parameters.

​
Examples

  • Quality Test
  • Red Teaming
  • Scenario Test
  • Quality Custom Test from CSV
Create a quality test to evaluate the quality and correctness of outputs.
test = galtea.tests.create(
    name="example-test-tutorial",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    ground_truth_file_path="path/to/knowledge_file.pdf",
    language='english',
    variants=["paraphrased", "typos", "incorrect"],
    max_test_cases=100
)

​
Parameters

​
name
string
required
The name of the test.
​
type
string
required
The type of test. Possible values:
  • QUALITY: Tests that evaluate the quality and correctness of outputs
  • RED_TEAMING: Tests that evaluate security, safety, and bias aspects
  • SCENARIOS: Tests that use conversation simulation to evaluate multi-turn dialogue interactions
​
product_id
string
required
The ID of the product you want to evaluate.
​
ground_truth_file_path
string
Path to a local file containing the knowledge base. This file is uploaded to Galtea, which then generates test cases based on its content.
Supported formats include .pdf, .txt, .json, .html, .md, and .zip. Upload a .zip archive to generate test cases from multiple documents at once.
​
few_shot_examples
string
Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases. Example:
Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of Germany?
A: The capital of Germany is Berlin.
This field only applies if tests are generated by Galtea and are of type QUALITY.
​
custom_user_focus
string
Narrow down the scope of generated scenarios by describing a specific type of user, context, or situation. This helps ensure test cases align with your most relevant goals and flows. Example: “A medical professional specialized in dementia with more than 15 years on the field.”
This field only applies if tests are generated by Galtea and are of type SCENARIOS.
​
language
string
The language for generating synthetic test cases if ground_truth_file_path is provided. Defaults to the language detected in the ground truth file. If provided, it should be written in English and be a valid language name from the ISO 639 standard (e.g., “english”, “spanish”, “french”). More information can be found at https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes.
​
variants
list[string]
A list of strings that specifies how to generate test cases related to its objectives. The meaning of this parameter depends on the type of the test:
  • For RED_TEAMING tests: Specifies the Threat category to generate test cases for (e.g., ["data_leakage"]). See the full list in the Red Teaming Threats documentation.
​
strategies
list[string]
A list of strings that specifies how to generate test cases related to its style.
  • For RED_TEAMING tests: Strategies are techniques for modifying or obfuscating prompts generated for each threat (as specified in variants for Red Teaming tests). See the full list of red teaming strategies.
  • For SCENARIOS tests: Strategies define the conversation style. Currently written and spoken are supported, which influence the tone and formality of the generated dialogues. If not specified, the default strategy is written.
​
max_test_cases
int
An optional integer to limit the maximum number of test cases generated by Galtea when ground_truth_file_path is used.
​
test_file_path
string
Path to a local CSV file containing predefined test cases. This file is uploaded to Galtea.File Format by Test Type:
  • QUALITY/RED_TEAMING: Standard format with input, expected_output, tag, source columns
  • SCENARIOS: Conversation simulator format with goal, user_persona, initial_prompt, stopping_criterias, max_iterations, scenario columns
See the Conversation Simulator Tutorial for detailed SCENARIOS CSV format examples.
⌘I