Evaluation Parameters
To compute thefactual_accuracy metric, the following parameters are required:
input: The original user query or question that prompted the response.expected_output: The reference or ground truth answer that the model’s output should be compared against. A field rule in a JSON expected output reaches the judge as plain JSON text.actual_output: The response generated by your LLM application.
How Is It Calculated?
Factual Accuracy is computed using an LLM-as-a-judge process that evaluates the response on three key dimensions: accuracy, completeness, and relevance, in the following sense:- Accuracy: whether the information is correct and consistent with what is expected.
- Completeness: whether all relevant points from the input are properly addressed.
- Relevance: whether the content stays focused on the topic without adding unnecessary or unsupported material.
- Score 1: Response aligns with the reference answer with no errors, omissions, or unsupported additions
- Score 0: Response contains factual errors, omits essential information needed for the input, or includes unsupported content
This metric is inspired by best practices in the open source community and is implemented natively in the Galtea platform.