Skip to main content

​
Returns

Returns a list of Evaluation objects, one for each metric provided.

​
Usage

This method is versatile and can be used for two main scenarios:
  1. Test-Based Evaluation: When you provide a test_case_id, Galtea evaluates your product’s performance against a predefined challenge.
  2. Production Monitoring: When you set is_production=True and provide an input, Galtea logs and evaluates real user interactions.

​
Example: Test-Based Evaluation with Standard and Custom Metrics

from galtea import Galtea, CustomScoreEvaluationMetric

# Define your custom metric
class MyAccuracy(CustomScoreEvaluationMetric):
    def __init__(self):
        super().__init__(name="my-accuracy")
    def measure(self, *args, actual_output: str | None = None, **kwargs) -> float:
        """
        Returns 1.0 if 'paris' is in actual_output (case-insensitive), else 0.0.
        All other args/kwargs are accepted but ignored.
        """
        if actual_output is None:
            return 0.0
        return 1.0 if "paris" in actual_output.lower() else 0.0

custom_score_accuracy = MyAccuracy()

# If the metric does not exist yet in the platform, you can create it like this:
galtea.metrics.create(name=custom_score_accuracy.name, description='checks for the presence of the keyword "paris" in the output')

# Create evaluations using a mix of standard and custom metrics
evaluations = galtea.evaluations.create_single_turn(
    version_id="YOUR_VERSION_ID",
    test_case_id="YOUR_TEST_CASE_ID",
    actual_output="The capital of France is Paris.",
    metrics=[
        "Role Adherence",            # Standard metric
        custom_score_accuracy   # Your custom metric object
    ]
)

​
Example: Production Monitoring

evaluations = galtea.evaluations.create_single_turn(
    version_id="YOUR_VERSION_ID",
    is_production=True,
    input="A real user's query from production",
    actual_output="The model's live response.",
    metrics=["Contextual Relevancy", "Non-Toxic"],
    retrieval_context="Context retrieved by RAG for this query."
)

​
Parameters

​
version_id
string
required
The ID of the version you want to evaluate.
​
metrics
List[string | CustomScoreEvaluationMetric]
required
A list of metrics to use for evaluation. You can provide:
  • Standard metrics as strings (e.g., “Role Adherence”).
  • Custom, locally-scored metrics as objects inheriting from CustomScoreEvaluationMetric.
​
actual_output
string
required
The actual output produced by the product.
​
test_case_id
string
The ID of the test case to be evaluated. Required for non-production evaluations.
​
input
string
The input text/prompt. Required for production evaluations where no test_case_id is provided.
​
is_production
boolean
Set to True to indicate the evaluation is from a production environment. Defaults to False.
​
retrieval_context
string
The context retrieved by your RAG system that was used to generate the actual_output.
​
latency
float
Time in milliseconds from the request to the LLM until the response was received.
​
usage_info
dict[str, int]
Information about token usage during the model call. Possible keys include:
  • input_tokens: Number of input tokens sent to the model.
  • output_tokens: Number of output tokens generated by the model.
  • cache_read_input_tokens: Number of input tokens read from the cache.
​
cost_info
dict[str, float]
Information about the cost per token during the model call. Possible keys include:
  • cost_per_input_token: Cost per input token sent to the model.
  • cost_per_output_token: Cost per output token generated by the model.
  • cost_per_cache_read_input_token: Cost per input token read from the cache.
​
conversation_simulator_version
string
The version of Galtea’s conversation simulator used to generate the user message (input). This should only be provided if the input was generated using the conversation_simulator_service.
⌘I