Skip to main content

​
What are Metrics?

Metrics in Galtea define the specific criteria and methods used to evaluate the performance of your product. They determine how outputs are scored during evaluations, ensuring consistent and meaningful assessment.
Metrics are organization-wide and can be reused across multiple products.
You can create, view and manage your metrics on the Galtea dashboard or programmatically using the Galtea SDK.

​
Conceptual categories

At Galtea, we use two types of metrics to evaluate large language model (LLM) outputs: deterministic and non-deterministic.
  • Deterministic metrics are rule-based and computed using strict logic, such as SQL queries or structural checks. These include things like answer format validation, presence of required fields, or exact string matches. Their results are consistent and reproducible. Common examples of deterministic checks include:
    • Answer format validation (e.g., ensuring the output is a valid JSON or follows a specific template)
    • Presence of required fields (e.g., checking if all necessary information is included in the response)
    • Exact string matches (e.g., verifying if a specific keyword or phrase is present)
    • Numerical range checks (e.g., confirming a value falls within an acceptable range)
    • Boolean condition checks (e.g., ensuring a specific condition evaluates to true or false as expected)
    Currently the deterministic metrics accessible directly from the platform are BLEU, ROUGE, METEOR, Text Similarity, Text Match, IOU and Spatial Match and URL Validation. Additionally you can define your own custom deterministic metrics. In order to do that, you can implement a class that inherits from the CustomScoreEvaluationMetric base class and pass an instance of it to the evaluation creation method. See more in our example.
    The platform cannot automatically evaluate deterministic metrics, as it lacks the necessary information. However, by using the CustomScoreEvaluationMetric class in the SDK, you can execute your custom logic locally and have the scores seamlessly uploaded to the platform for visualization.
  • Non-deterministic metrics are powered by the LLM-as-a-judge methodology. The tested and deployed Galtea judges are human-aligned and optimized for the type of evaluation they are designed for. These metrics assess aspects like factual accuracy, misuse resilience, and correct task completion. Additionally you can create your own metrics using a template from the platform, assessing any aspect of your AI product you desire.
Galtea metrics are versatile and can be applied to any type of output, including strings, numbers, and boolean values.
  • For deterministic metrics: Evaluating numerical or boolean outputs is straightforward. For example, you can check if a returned numerical value is within a valid range or if a boolean output matches a specific condition.
  • For non-deterministic metrics: While typically used for open-ended text, they can also assess the reasoning or justification behind a numerical or boolean value when needed. For instance, verifying whether a model’s numeric prediction aligns with the provided context or input data.

​
List of metrics available in the Galtea Platform

The following table provides a summary of the default metrics available in the Galtea platform. You can also create custom metrics tailored to your specific needs.
MetricCategoryDescription
Factual AccuracyRAGMeasures the quality of your RAG pipeline’s generator by evaluating whether the actual_output factually aligns with the expected_output.
Resilience To NoiseRAGEvaluates whether the generated output is resilient to noisy input, such as typos, OCR/ASR errors, and distracting content.
Answer RelevancyRAGMeasures the quality of your RAG pipeline’s generator by evaluating how relevant the actual_output of your LLM application is compared to the provided input.
FaithfulnessRAGMeasures the quality of your RAG pipeline’s generator by evaluating whether the actual_output factually aligns with the contents of your retrieval_context.
Contextual PrecisionRAGMeasures your RAG pipeline’s retriever by evaluating whether nodes in your retrieval_context that are relevant to the given input are ranked higher than irrelevant ones.
Contextual RecallRAGMeasures the quality of your RAG pipeline’s retriever by evaluating the extent of which the retrieval_context aligns with the expected_output.
Contextual RelevancyRAGMeasures the quality of your RAG pipeline’s retriever by evaluating the overall relevance of the information presented in your retrieval_context for a given input.
BLEUDeterministicMeasures how many n-grams in the actual output overlap with those in a set of expected output.
ROUGEDeterministicEvaluates automatic summarization by measuring the longest common subsequence that preserves the word order between actual output and expected output summaries.
METEORDeterministicEvaluates translation, summarization, and paraphrasing by aligning words using exact matches, stems, or synonyms.
Text SimilarityDeterministicQuantifies the overall textual resemblance between a generated summary and a reference summary by using character-level fuzzy matching.
Text MatchDeterministicDetermines whether generated text matches a reference with high character-level similarity using fuzzy matching, returning a binary outcome based on a threshold.
IOUDeterministicMeasures the spatial overlap between a predicted bounding box and one or more reference boxes to quantify alignment in object detection and layout tasks.
Spatial MatchDeterministicPerforms a binary evaluation of the spatial alignment between a predicted bounding box and reference boxes using the best Intersection over Union (IoU) score to return a pass/fail signal.
URL ValidationDeterministicPerforms a binary evaluation to check if all the URLs present in the model response are valid and safe.
Role AdherenceConversationalDetermines whether your LLM chatbot is able to adhere to its given role throughout a conversation.
Conversation CompletenessConversationalDetermines whether your LLM chatbot is able to complete an end-to-end conversation by satisfying user needs throughout a conversation.
Conversation RelevancyConversationalDetermines whether your LLM chatbot is able to consistently generate relevant responses throughout a conversation.
Knowledge RetentionConversationalAssesses the ability of your LLM chatbot to retain factual information presented throughout a conversation.
User SatisfactionConversationalEvaluates how satisfied the user was with the chatbot interaction, focusing on efficiency and user sentiment.
User Objective AccomplishedConversationalEvaluates whether the chatbot successfully and correctly fulfilled the user’s stated objective, optionally verifying against an expected_output.
Non-ToxicRed TeamingDetermines whether the responses of your LLM based product responds are free of toxic language.
UnbiasedRed TeamingDetermine whether your LLM output is free of gender, racial, or political bias.
Misuse ResilienceRed TeamingEvaluates whether the generated output is resilient to misuse and remains aligned with the product description.
Data LeakageRed TeamingEvaluates whether the LLM returns content that may include sensitive information.
Jailbreak ResilienceRed TeamingA previous version of the Jailbreak Resilience v2 Metric.
Jailbreak Resilience v2Red TeamingEvaluates the ability of an LLM-based product to resist attempts at breaking or manipulating its intended behavior.

​
Metric Properties

​
Name
Text
required
The name of the metric. Example: “Factual Accuracy”
​
Description
Text
A brief description of what the metric evaluates.
​
Documentation URL
URL
A URL pointing to more detailed documentation about the metric.
​
Evaluator Model
Text
The name of the model used to evaluate the metric. This model will be used to assess the quality of the outputs based on the metric’s criteria. Example: “GPT-4.1”.
It does not apply to deterministic metrics since it does not require a model for evaluation.
​
Tags
Text List
The tags for the metric so it can be easily identified and categorized. Example: [“RAG”, “Conversational”]
​
Validation method
Enum
required
The method used to evaluate the metric. This can be either:
  • LLM as a Judge
  • None
A custom prompt that defines the evaluation logic for an LLM-as-a-judge metric. You can use placeholders like {input}, {actual_output}, {expected_output}, etc., which will be populated at evaluation time.
​
Evaluation Parameters
List[Enum]
required
This must not be provided if the validation method is set to “Custom”.
A list of relevant parameters for this metric’s evaluation criteria or steps. These parameters should be explicitly mentioned in your evaluation criteria or steps to ensure they’re taken into account during assessment.
ParameterDescriptionAvailability
inputThe prompt or query sent to the model. (Always required in the list).All metrics
actual_outputThe actual output generated by the model.All metrics
expected_outputThe ideal answer for the given input.All metrics
contextAdditional background information provided to the model alongside the input.All metrics
retrieval_contextThe context retrieved by your RAG system before sending the user query to your LLM.All metrics
product_descriptionThe description of the product.Yes
product_capabilitiesThe capabilities of the product.Yes
product_inabilitiesThe product’s known inabilities or restrictions.Yes
product_security_boundariesThe security boundaries of the product.Yes
input (lowercase) must always be included in this list.
You can directly reference these parameters in your criteria or evaluation steps. For example: “Evaluate if the Actual Output contains factually correct information that aligns with verified sources in the Retrieval Context.”
To ensure accurate evaluation results, include only parameters that you’ve explicitly referenced in your criteria or evaluation steps.

​
Creating Custom Metrics via Judge Prompt

The non-deterministic metrics, powered by Large Language Models that act as Judges, utilize models that have demonstrated the best performance in our internal benchmarks and testing. We are committed to continuously evolving and improving these evaluator models to ensure the highest quality assessments over time.
Custom Judge metrics use custom judge prompts to evaluate LLM outputs based on your specific requirements. The primary way to create these metrics is by providing a judge_prompt that defines the evaluation logic.

​
How Judge Prompts Work

Judge prompts are templates that tell the evaluator model how to assess your outputs. You can include placeholders for various parameters that will be automatically filled in during evaluation:
  • {input} - The original user input or prompt
  • {actual_output} - The response generated by your model
  • {expected_output} - The ideal or reference response
  • {context} - Additional background information
  • {retrieval_context} - RAG-retrieved context
  • {product_description} - Your product’s description
  • {product_capabilities} - What your product can do
  • {product_inabilities} - What your product cannot or should not do
  • {product_security_boundaries} - Security restrictions
  • {user_persona} - Information about the user interacting with the agent
  • {goal} - The user’s goal
  • {scenario} - The scenario in which the user is operating
  • {stopping_criterias} - List of criteria that define when the conversation should end
  • {conversation_turns} - All turns in the conversation

​
Evaluation Process

Two-step Galtea judge process:
  1. Assessment: The evaluator model analyzes the inputs according to your judge prompt
  2. Score Computation: The model assigns a score on a 1–5 scale, which is then normalized to 0–1 via token-probability weighting

​
Example Judge Prompt

judge_prompt = """
You are an expert evaluator. Evaluate the factual accuracy of the given response by comparing it to the reference answer and considering how well it addresses the user's input.

**User Input:** {input}
**Reference Answer:** {expected_output}
**Response to Evaluate:** {actual_output}

Scoring Guidelines:
- Score 0: The response contains factual errors, omits essential information needed for the input, or includes unsupported content.
- Score 0.5: The response is partially accurate. It is generally relevant but includes minor inaccuracies, missing key details, or some unsupported claims.
- Score 1: The response fully aligns with the reference answer with no errors, omissions, or unsupported additions.
"""
When designing judge prompts, be specific about your scoring criteria and reference the evaluation parameters explicitly. This ensures consistent and reliable evaluations.

​
SDK Integration

The Galtea SDK allows you to create, view, and manage metrics programmatically.

Metrics Service SDK

Manage metrics using the Python SDK

Evaluation

The assessment of an evaluation using a specific metric’s criteria
⌘I