​
Returns

Returns a Test object for the given parameters.

​
Example

test = galtea.tests.create(
    name="example-test-tutorial",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    ground_truth_file_path="path/to/knowledge_file.pdf",
    language='english',
    variants=["paraphrased", "typos", "incorrect"],
    max_test_cases=100
)

See a complete example of creating custom tests in our Create a Custom Test example.

​
Parameters

​
name
string
required

The name of the test.

​
type
string
required

The type of test. Possible values:

  • QUALITY: Tests that evaluate the quality and correctness of outputs
  • RED_TEAMING: Tests that evaluate security, safety, and bias aspects
  • SCENARIOS: Tests that use conversation simulation to evaluate multi-turn dialogue interactions
​
product_id
string
required

The ID of the product you want to evaluate.

​
ground_truth_file_path
string

Path to a local file (e.g., PDF, TXT, JSON, HTML, Markdown) containing the knowledge base. This file is uploaded to Galtea, which then generates test cases based on its content.

​
few_shot_examples
string

Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases. Example:

Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of Germany?
A: The capital of Germany is Berlin.
This field only applies if tests are generated by Galtea and are of type QUALITY.
​
language
string

The language for generating synthetic test cases if ground_truth_file_path is provided. Defaults to the language detected in the ground truth file. If provided, it should be written in English and be a valid language name from the ISO 639 standard (e.g., “english”, “spanish”, “french”). More information can be found at https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes.

​
variants
list[string]

List of variant types to generate for each test case from the ground_truth_file_path. For details on variants for Quality tests, see Quality Test Evolutions. Example: ["paraphrased", "typos", "incorrect", "cognitively_diverse", "linguistic_diverse"]. For Red Teaming tests, the variants parameter can be used to specify which threat categories to generate test cases for.

​
strategies
list[string]

A list of red teaming strategies used when generating test cases for Red Teaming tests. Strategies are techniques for modifying or obfuscating prompts generated for each threat (as specified in variants for Red Teaming tests). This parameter applies only when type is RED_TEAMING. If specified, this list must include the ‘original’ strategy. Example: ["original", "base64", "leetspeak", "zero_width_insertion"].

​
max_test_cases
int

An optional integer to limit the maximum number of test cases generated by Galtea when ground_truth_file_path is used.

​
test_file_path
string

Path to a local CSV file containing predefined test cases. This file is uploaded to Galtea.

File Format by Test Type:

  • QUALITY/RED_TEAMING: Standard format with input, expected_output, tag, source columns
  • SCENARIOS: Conversation simulator format with goal, user_persona, initial_prompt, stopping_criterias, max_iterations, scenario columns

See the Conversation Simulator Tutorial for detailed SCENARIOS CSV format examples.