Skip to main content
The JSON Field Match metric is one of the Deterministic Metric options in Galtea. It performs a field-level comparison between two JSON objects, giving a more granular and interpretable score than text-based matching when the expected output is structured JSON.

​
Evaluation Parameters

To compute the JSON Field Match metric, the following parameters are required:
  • actual_output: The JSON output generated by the model (string or object). Supports lenient parsing — JSON wrapped in markdown code fences or embedded in surrounding text is automatically extracted.
  • expected_output: The reference JSON object to compare against. Must be a valid JSON object.

​
How Is It Calculated?

The metric compares top-level fields between the expected and actual JSON outputs:
  1. Parse Inputs expected_output is parsed as a strict JSON object. actual_output is parsed leniently — the metric will attempt to extract a JSON object from markdown code fences (e.g., ```json ... ```) or surrounding text before parsing. If either cannot be resolved to a valid JSON object, the evaluation raises an error.
  2. Compare Fields For each top-level key in the expected output, check whether the same key exists in the actual output with an equal value. Comparison uses strict equality — type mismatches (e.g., string "30" vs number 30) count as non-matching. Nested objects are compared by deep equality.
  3. Compute Score The score is calculated as: 
    score = 1 - (non_matching_fields / total_fields)
    Where total_fields is the number of top-level keys in the expected output, and non_matching_fields is the count of keys that are missing or have different values in the actual output.

​
Interpretation of Scores

  • 1.0 — All expected fields are present in the actual output with matching values. Also returned when the expected output is an empty JSON object {}.
  • 0.5 — Half of the expected fields match.
  • 0.0 — No fields match.
Extra keys in the actual output that are not in the expected output are ignored. Missing keys in the actual output count as non-matching.

​
Suggested Test Case Types

Use JSON Field Match when evaluating:
  • Structured extraction tasks where the model outputs a JSON document with specific fields.
  • Form-filling or slot-filling agents where each field must match independently.
  • API response validation where the expected output is a known JSON structure.
  • Golden dataset evaluations where partial credit is useful (e.g., 3 out of 4 fields correct).