Skip to main content
The User Objective Accomplished metric is one of several non-deterministic Metrics Galtea uses to evaluate whether a conversation led to the user’s intended goal being fulfilled. Unlike satisfaction-based measures, this metric centers on objective correctness—whether the agent actually met the user’s stated objective. This metric is particularly useful for use cases where accuracy and goal fulfillment matter more than tone or fluency, such as customer support resolutions, fact-based Q&A, or task execution scenarios.

​
Evaluation Parameters

To compute the user_objective_accomplished metric, the following parameters are required:
  • input: The user messages sent to the chatbot.
  • actual_output: The chatbot’s corresponding responses.
  • goal: The stated objective or intent of the user.

​
How Is It Calculated?

The user_objective_accomplished score is derived using an LLM-as-a-judge approach with strict correctness criteria and a chain-of-thought style evaluation:
  1. Goal Identification: Determine the user’s stated objective.
  2. Agent Response Evaluation: Analyze how the agent attempted to fulfill the goal across the conversation and examine the final actual_output.
  3. Correctness Check: Judge whether the final actual_output correctly, completely, and directly fulfills the user’s stated goal. Identify any factual errors, omissions, or misunderstandings that prevent accomplishment.
Based on this process, the LLM assigns a binary score:
  • 1 (Accomplished): The agent successfully and correctly fulfilled the user’s objective.
  • 0 (Not Accomplished): The agent failed to fulfill the user’s objective.