The Test Service in the Galtea SDK allows you to manage tests for evaluating your products. This Service is exposed by the galtea.tests object and we will further explore its API down below.

Remember that we will be using the galtea object. More information here.

​
Create Test

This method allows you to create a test for your product.

test = galtea.tests.create(
    name="example-test-tutorial",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    ground_truth_file_path="path/to/knowledge_file.pdf",
    language='english'
)

See a complete example of creating custom tests in our Create a Custom Test example.

​
name
string
required

The name of the test.

​
type
string
required

The type of test (e.g., QUALITY, RED_TEAMING).

​
product_id
string
required

The ID of the product you want to evaluate.

You must provide either test_file_path or ground_truth_file_path, but not both.

​
ground_truth_file_path
string

The path to the knowledge file (pdf format) in which We will find the information the source of truth for the model answers. We have to wait for the platform to generate a synthetic dataset based on the ground_truth provided, so we need to wait for the test to be ready before we can use it.

The ground truth is also referenced as the knowledge base.

​
language
string

The language that will be used for the generation of the synthetic data. Defaults to the language used in the knowledge base file.

Available languages are: “English”, “Spanish”, “Portugese”, “Italian”, “French”, “German”, “Korean”, “Japanese” and “Chinese”.

This field only makes sense if the test is generated by Galtea.

​
variants
list[string]

variants represent modifications of the original queries derived from the knowledge base file. These variants are generated by our system to test different scenarios and question formats. Example: ["ambiguous", "typos", "cognitively_diverse"]

Available variants:

  • paraphrased
  • expanded_question
  • specific_focus_question
  • ambiguous
  • incorrect
  • incomplete
  • typos
  • slang
  • abbreviations
  • unconventional_phrasing
  • combined_topics
  • novel_phrasing
  • hypothetical_scenarios
  • informal
  • linguistic_diverse
  • typographic_error
  • cognitively_diverse

This field is applicable only when the test is generated by Galtea.

​
test_file_path
string

The path to the custom test file. This file should contain the test cases you want to evaluate. If given, the platform won’t generate a synthetic dataset based on the ground_truth provided, so we can use the test immediately.

​
Listing Tests

This method allows you to list all tests associated with a specific product.

tests = galtea.tests.list(product_id="YOUR_PRODUCT_ID")
​
product_id
string
required

The ID of the product for which you want to list tests.

​
offset
int

The number of tests to skip before starting to collect the result set.

​
limit
int

The maximum number of tests to return.

​
Retrieving Test

This method allows you to retrieve a specific test by its ID.

test = galtea.tests.get(test_id="YOUR_TEST_ID")
​
test_id
string
required

The ID of the test you want to retrieve.

​
Retrieving Test By Name

This method allows you to retrieve a specific test by its name.

test = galtea.tests.get_by_name(product_id="YOUR_PRODUCT_ID", name="YOUR_TEST_ID")
​
product_id
string
required

The ID of the product for which you want to retrieve the test.

​
name
string
required

The name of the test you want to retrieve.

​
type
string

The type of test (e.g., QUALITY, RED_TEAMING).

​
Deleting Test

This method allows you to delete a specific test by its ID.

galtea.tests.delete(test_id="YOUR_TEST_ID")
​
test_id
string
required

The ID of the test you want to delete.