What the classifier is
The classifier is a judge for an AI Evaluation metric. A classifier model answers one typed question about the conversation and returns a probability for each answer. It writes no reasoning.- It is fast: an answer usually takes less than a second.
- It costs 1 credit per evaluation, whatever the classifier model.
"classifier".
Choose the classifier model
A classifier model is a decision model that the LLM gateway of your deployment serves. It is either a TypeSafe Jev version, such asjev-1.13.0, or another decision model, such as microsoft-decision-1. Each deployment decides which classifier models it offers. A Jev version is offered only where the gateway of the deployment serves it.
- In the dashboard, the metric form and the Convert to classifier page show a Classifier Model picker when your deployment offers several classifier models, and you must pick one. With one model, there is no picker, and the metric uses that model. Test on a sample runs with the same model.
- In the SDK and the API, send the model as
evaluator_model_name(evaluatorModelNamein the API). With one selectable classifier model, you can leave it out and the API uses that model. With several, the API answers 400 until you name one. A name that is not a selectable classifier model also gets a 400.GET /evaluatorModelswithkind=CLASSIFIERandselectable=truelists them. - A metric keeps its model. To change it, create a revision with another classifier model.
When to use it
- Classifier: a clear rule that a reply must follow or must never break, or a graded quality whose levels you can describe as concrete situations.
- LLM judge: the check needs reasoning, a written explanation of the score, or files.
- Deterministic metric: exact text or JSON matching. Use one of Galtea’s metrics, such as Text Similarity or JSON Field Match.
- Regex, number or date checks on a JSON field: use the
$regex,$numand$datefield rules of JSON Field Match. - Counting, arithmetic, dates or time limits: use an LLM judge metric or a Self-Hosted metric.
Question types
A classifier metric asks exactly one question. The question names the data it reads with placeholders, such as{{ actual_output }} or {{ input }}. Insert them from the Evaluation Parameters list above the text.
Allowed placeholders
A question can use only these names. Any other name is refused, including the deprecated parameters.product_descriptioninputexpected_outputcontextexpected_toolsgoalactual_outputretrieval_contexttracestools_used
Yes/No
A question with a yes or no answer. You must set the Good answer: Yes or No. Example: “Does{{ actual_output }} ask the user for their full card number?”, with Good answer No.
Under Advanced, Refine what yes and no mean lets you describe a boundary case. Use it only when the question alone is not clear.
Scale
A question with two or more levels, worst first. Each level has a description and a score from 0 to 1. The scores are evenly spaced by default (for three levels: 0, 0.5 and 1). You can edit them, but they must not go down from the worst level to the best. A Scale question can also have an optional Does not apply level. Use it when the question covers a situation that happens only in some conversations. When Does not apply is the most probable answer, the evaluation is skipped and gets no score. Example: “Does{{ actual_output }} offer to transfer the user to a human agent?”, with these levels:
A Scale question has 2 to 10 levels, or 2 to 9 levels when it has a Does not apply level.
How the score is calculated
The score is the expected value of the classifier’s answer, not only its most probable answer.- Yes/No: the probability of the good answer. When the classifier gives “yes” a probability of 0.99 and the good answer is No, the score is 0.01.
- Scale: the level scores, each weighted by its probability, divided by the total probability of the scored levels. So the Does not apply probability does not lower the score. The classifier answers 0.1 for “Does not offer a transfer” (score 0) and 0.9 for “Offers a transfer” (score 1), so the score is 0.9.
Yes/No or a 2-level Scale
The same rule can give different numbers in the two forms. A Yes/No score is the probability of the good answer, so a borderline reply shows doubt. In a test of five card-number checks on Jev, both forms gave the same answer, but Yes/No scored 0.99 or 0.01 and the 2-level Scale scored exactly 0 or 1. So a 2-level Scale can hide doubt that Yes/No shows.- Prefer Yes/No when the rule applies in every conversation. The score then tells you how sure the classifier is.
- Prefer a Scale when the rule applies only sometimes, because only a Scale has a Does not apply level. Also use it for a graded quality with three or more levels.
- With a 2-level Scale, read the confidence next to the score to see doubt that the score hides.
The classifier answered Yes with probability 0.99. The good answer is No. or Expected score 0.90. Most probable level: "Offers a transfer" (0.90). The metric applies no threshold to the score. An evaluation passes when its reported score reaches the metric’s pass threshold. The evaluation page and score_breakdown in the SDK show the probabilities (see Evaluation).
One question per metric
A metric asks one question, so its result tells you about one thing. When your description has several conditions:- Every condition always applies: you can list them all in one Yes/No question (“Does
{{ actual_output }}do A and B?”). The result cannot show which condition failed. - Some conditions apply only sometimes: create one metric per condition. A question that merges them needs conditional wording, which the classifier answers less reliably.
Tips for good questions
Ask about the property itself, and set Good answer
Ask about the property itself, and set Good answer
The classifier answers positive questions more reliably than negated ones.Avoid: “Is
{{ actual_output }} free of card number requests?”Do: “Does {{ actual_output }} ask the user for their full card number?” with Good answer = No.Use concrete words
Use concrete words
The classifier answers better when the question names what to look for.Avoid: “Does
{{ actual_output }} share another customer’s personal data?”Do: “Does {{ actual_output }} share personal data, such as an address, a balance or account details, about a person other than the user?”Describe each Scale level as a concrete situation too, not as a degree such as “moderately good”.Use a Scale with Does not apply for a situation that happens only sometimes
Use a Scale with Does not apply for a situation that happens only sometimes
A Yes/No question counts in every conversation. Never write it with a condition (“When X, does it Y?”): a conversation without X then gets a score that means nothing. This is the one case where a 2-level Scale is the right choice, even though its score shows less doubt than Yes/No (see How the score is calculated).Avoid: “When the user asks for a human agent, does
{{ actual_output }} offer a transfer?”Do: a Scale question “Does {{ actual_output }} offer to transfer the user to a human agent?” with the levels “Does not offer a transfer” and “Offers a transfer”, and the Does not apply level “Does not apply: {{ input }} does not ask for a human agent”.Name the field in the Does not apply text
Name the field in the Does not apply text
The classifier receives only the fields that your question names, in any of its texts. A placeholder in the Does not apply text sends that field too. Do not name more fields than the question needs: extra data lowers the classifier’s accuracy.Avoid: “Does not apply: the user did not ask for a human.”Do: “Does not apply:
{{ input }} does not ask for a human agent.”Write the question in English
Write the question in English
The classifier answers English questions best, also for conversations in Spanish or Catalan. Keep the question, the levels and the Does not apply text in English.
Do not ask the classifier to count, do arithmetic, compare dates or match exact text
Do not ask the classifier to count, do arithmetic, compare dates or match exact text
The classifier cannot do these checks reliably. Use an LLM judge metric or a Self-Hosted metric for them, and a deterministic metric for exact matching. For a pattern, a number or a date in a JSON field, use the field rules of JSON Field Match.
Test the question before you save it
Test the question before you save it
Test on a sample runs the draft question on a session or on values you type. Nothing is saved and no credits are used.
Complete with AI
Complete with AI builds the question from the metric’s name, description and tags. For one condition that always applies, it writes a Yes/No question, or a Scale question for a graded quality. For one condition that applies only sometimes, it writes a Scale question with a Does not apply level.- When the description has several conditions that always apply, it writes one Yes/No question that lists them all. A hint tells you to create one metric per condition if you want to score each one on its own.
- When some of the conditions do not always apply, it writes no question. A hint tells you to create one metric per condition.
- It lists any part of the description it could not turn into a question, such as counting or dates, with a suggested alternative.
Convert an LLM judge metric
Convert to classifier turns an LLM judge metric into classifier metrics. It is a button on the metric’s page. The AI reads the judge prompt and writes one question for each condition in it. Each question becomes its own Classifier metric. The button shows only for an LLM judge metric that has a judge prompt, where classifier metrics are on and you can create metrics: a Member needsmetric.create in a user group list. The source can be a metric of your organization or a Galtea metric. The new metrics always belong to your organization. It is off while an optimization of the metric runs or waits for review, and for a metric with too many revisions to check for one.
- Generate questions. The AI splits the judge prompt. This takes 10 to 40 seconds. Nothing is saved and no credits are charged.
- Check the proposals. Pick how many metrics to create, and the classifier model they all use. Edit a question or leave it out.
- Create. Galtea creates all the metrics you kept in one step. If one is refused, for example because its name is taken, none is created.
- One metric per condition (Recommended): one question for each condition, so a failed result shows which condition broke. Each metric is its own evaluation, at 1 credit each. Each metric is named
<source name>: <label>. A specification’s score pools the scored evaluations of its metrics, so one failed condition lowers it only by that metric’s share. - One metric for all conditions: one Yes/No question that lists every condition. It is available only when all of those conditions always apply. The result does not show which condition failed. The metric is named
<source name> (Classifier).
<source name> (Classifier).
Each new metric is linked to the specifications of your organization that the source metric is linked to. A user limited to the products their user groups grant gets and links only the specifications of those products. The source stays linked too, so each condition is checked twice. After you compare the scores, unlink the source from those specifications. Each new metric also takes the source’s tags.
A part of the judge prompt that the classifier cannot check, such as counting, dates or exact matching, is listed under Not converted. Only the source metric still checks it.
The source metric does not change, and its past results stay. Before you stop using it, run the new metrics next to it on a few sessions and compare the scores. The classifier gives no written reason, and its scores can differ from the LLM judge.
The converted metrics are new metrics. A monitor that uses the source metric keeps using it and does not get the new metrics.
A new revision keeps the evaluation type of its metric, so it cannot switch a metric between an LLM judge and the classifier. To try the classifier, use Convert to classifier: it keeps the LLM judge metric and adds classifier metrics next to it.
Limits
- Text only: the classifier does not read images, audio, PDFs or other files. A turn with a Galtea file on a side the metric reads skips the evaluation, and no credits are charged. A file on your own host is not sent as a file: the classifier reads its entry as JSON text, in its place in the turn. See Troubleshooting skipped evaluations.
- Input size: the classifier model accepts about 32,000 tokens of conversation plus the question. A longer input is skipped, not scored, and no credits are charged for a skipped evaluation.
- Long input: when the classifier model reads more than about 2,000 tokens and no reviewer has scored the evaluation, the result is marked Long input and left out of pass rates and score averages. The evaluation keeps its score.
- Skipped evaluations: analytics show how many evaluations were skipped, next to the passed and not passed counts.
- Classifier model: the metric stores its classifier model, such as
jev-1.13.0. To change it, create a revision with another classifier model. Every result records the model that answered inscore_breakdown.model. A gateway decision model answers with its base group name, so a metric oneu.microsoft-decision-1recordsmicrosoft-decision-1. - Prompt injection: the conversation itself can try to steer the classifier. Clear questions reduce the risk but do not remove it.
Data processing
To score a classifier metric, Galtea sends the question and the fields it names to the provider of the metric’s classifier model. Files are never sent.- A
jev-*model: the data goes to TypeSafe, which processes it in the United States. See TypeSafe legal. - Any other decision model: the data goes to the provider that serves that model on your deployment.
DECISION_MODEL: a Jev version by default, or another decision model. For each check, Galtea sends the input and actual_output of each turn, the other fields the metric scores (such as retrieval context, tool calls or the expected output), and the metric’s name, description and criteria. Traces are never sent. The criteria are the judge prompt, or the question for a classifier metric.
Create a classifier metric with the SDK
evaluation_params from the placeholders in every text of the question: here actual_output for the first metric, and actual_output and input for the second. The snippet leaves out evaluator_model_name, which works when your deployment has one selectable classifier model. With several, pass one as evaluator_model_name. The API checks every rule of the question, including the placeholder names, and refuses a question that breaks one. See Create a metric.