Mistral Workflows Calls Take Their Tokens From the Event
A Mistral Workflows LLM call that the Mistral SDK does not trace, such as a call through your own HTTP gateway, can now report its tokens. Addgen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens to its llm_call_details event. Galtea stores these counts on the span and prices the call from its model. See Monitor a Mistral Workflows agent.LLM Spans Find Their Model Price in More Cases
An LLM span now finds its model when its name has a date or lacks the provider prefix of the catalog:gpt-realtime-mini-2025-10-06 is priced as gpt-realtime-mini, and mistral-small-2603 as mistral/mistral-small-2603. A version’s model fact uses the same rule, so a fact such as mistral-small-2603 or azure/gpt-4o now links a model where it linked none (azure/gpt-4o links gpt-4o when the catalog has no azure/gpt-4o).The response model now wins over the requested model: a span with response openai/gpt-4o and request gpt-4o-mini links gpt-4o, where it linked gpt-4o-mini. Spans already stored keep their link. See Model for the full order.EU Applicability Check Uses the Decision Model’s EU Group
Self-hosted operators: the evaluator settingDECISION_MODEL_EU is removed. For an EU-only organization, the applicability check now asks the EU deployment of DECISION_MODEL, mapped by DECISION_MODEL_EU_NAMES like a classifier metric’s model. If you set DECISION_MODEL_EU, delete it and add your decision model to DECISION_MODEL_EU_NAMES. Until then, the check makes no call for an EU-only organization and the metric is judged.Mistral Workflows Traces Show Each LLM Call
Galtea now reads OpenTelemetry traces from agents built on Mistral Workflows. Your code records each LLM call as anllm_call_details event. Galtea now shows that event as a generation span with its input, output and model, as it did on Abraxas. This holds for any sender. Any llm_call_details event with input_messages or output_content now shows as a generation span. Before, it showed as an empty span unless it carried an abraxas.* key. Monitor a Mistral Workflows agent lists the settings that control turn text, tokens and redaction.Gemini 2.5 Judges Retire on 4 December 2026
Google has retired Gemini 2.5 Pro, Gemini 2.5 Flash and Gemini 2.5 Flash-Lite, so you can no longer pick them as the judge of a new metric. Metrics that already use them keep evaluating until 4 December 2026, and then their evaluations are skipped. To keep a metric running, create a revision of it with another model, for example Gemini 3.5 Flash or Gemini 3.1 Flash-Lite.Generators Translate With an LLM
Red teaming and gold standard generation translate with an LLM by default instead of Google Translate. Red teaming tests in another language now get their encoding instructions and prefixes in that language, also on installs without a Google key, and translations keep email addresses, URLs, codes and product names as written.Analytics Bar Chart Labels Stay on One Line
The value labels at the end of the bars in the Analytics bar charts, such as the Specifications, Metrics and inference charts, no longer wrap onto several lines or overlap on narrow cards. Each label is drawn on one line and keeps the same fade as its bar.Failed tests show their status
A test whose generation failed now shows a “Failed” status pill on its details page, so you can tell at a glance that it did not finish.One Name per Evaluator Model
Each judge and classifier model now has one name in every environment: theeu. names, such as eu.gpt-4.1 and eu.microsoft-decision-1, are gone, and a metric that used one now names the plain model. An organization with the EU-only data region still reaches only the model’s EU deployments under the plain name. Each evaluation now reports the region it ran in, as dataRegion in the API.Metric Page Separates the Metric From Its Revision
The Details tab of a metric now shows a Metric card with what every revision shares, and a Revision card with the judge settings of the revision you pick. Edit metric changes the name, description and tags for every revision, and the links of the active revision. A new revision takes an optional name of its own, and Convert to classifier says that it creates new metrics.See and Rename Metric Revisions in the Dashboard
Every screen that shows one metric revision now shows its number, such as R3, or R3: stricter rubric when you gave it a name. The metric page says from R2 for the revision it was made from, and Rename revision in its menu names or clears a revision’s name. When you revise a metric, the name field starts empty and a name you type becomes the revision’s own; the API fields are on the metric page.Make an Older Metric Revision Active Again
You can now make an older revision of a metric the active one withPOST /v2/revisions/{id}/activate, or galtea revisions activate <revisionId> in the CLI. The current active revision becomes Legacy, its specifications link the activated revision instead, and every evaluation, name and history stays. The endpoint is alpha and may still change; it answers 400 for a revision that cannot become active, such as one whose judge model is retired.Annotations Count for Every Revision of a Metric
A human annotation now belongs to its conversation, or one turn of it, and to the metric. Every revision of the metric shows it, so an evaluation of a new revision shows the existing annotation. Fixing or removing an annotation changes it for every revision. Annotating no longer needs a claim, so nothing locks the conversation. Human evaluation metrics are not part of this: their score stays on each evaluation, and they still need a claim. Annotations given before this change carry over. Where one conversation had different scores, the latest is kept. The evaluation endpoints keep their response field names. On a metric a judge scores, their human evaluator fields now describe the annotator. Sorting the evaluation list by these fields follows the annotation too. The new alpha/annotations endpoints read and write annotations directly. See Human Annotations.Clearer Errors When a Model Has No EU Deployment
For an EU-only organization, a call to a model with no EU deployment now fails at once, with an error that says so, instead of after about 15 seconds of retries. When the model has EU deployments that are busy, Galtea still retries for a short time, and the error then says that no EU deployment is available right now. See Data Region.EU Jobs No Longer Use Google Translate
For an EU-only organization, no text of a red teaming or gold standard generation job is sent to Google Translate. A red teaming job keeps the instruction of its base64 and hex encoding prompts in English instead of translating it, and a gold standard job translates the expected output and detects the document language with the EU LLM. See Data Region.Build Date in Your Local Time
The Platform version card in Settings now shows the build date in your local time zone. Hover the date to see the exact UTC time.SDK Reference Shows What Your SDK Version Can Call
Each deprecated SDK parameter or method in the SDK reference now shows the date it is removed and what to use instead. A new method or parameter shows the SDK version it needs, so a reader on an older SDK knows to upgrade before using it.Metrics Without an EU Judge Show as Unavailable
In an organization with the EU-only data region, a metric whose judge model has no EU deployment now shows as Unavailable, with the reason, in the dashboard and inunavailableReason of the API. Its evaluations are skipped before any call leaves the EU. The metric page offers New Revision, or Use as Template for a Galtea metric, to pick another judge, and the judge picker lists only judges with an EU deployment. A classifier metric follows the same rule through its decision model.The Trend Range Explains Its Dates
In Is it getting better or worse? on the product Summary, the range button now shows your pick. A preset shows in days, and presets no longer than the page period are disabled, with the reason on hover. The calendar offers no start after the page period start, and an info icon explains extra days drawn.Classifier Metrics Stay in the EU
For an EU-only organization, the evaluation of a classifier metric, its “Test on a sample” preview and the applicability check now run on an EU deployment of the decision model. A classifier metric whose decision model has no EU deployment ends SKIPPED, and its preview fails, with no call to another region. See Data Region.galtea docs Shows the Docs of Your Platform
The new galtea docs command prints the docs address for your platform version, the llms.txt index, and the platform version. It works without a login: then it prints https://docs.galtea.ai and a notice on stderr. Use it to point a coding agent at the right docs.Read and Change a Metric and Its Revisions Separately (Alpha)
The new/v2/metrics endpoints read and change a metric by its metric ID, with the ID of its active revision, and the new /v2/revisions endpoints list, read, create, rename and export the revisions of a metric. In the CLI, galtea metrics-v2 and galtea revisions run them, for example galtea metrics-v2 get <metricId> and galtea revisions list --metric-id <metricId>. A new revision can start from any revision of the metric, not only the active one. galtea revisions update gives a revision its own name, which must be unique within its metric, and galtea revisions create takes the same optional --name for a new revision, answering 409 when another revision of the metric holds it. When this release ships, a revision name that repeats within one metric stays on the active revision, or on the newest one when none is active, and each other revision takes (R<n>), or no name when that is taken too, so monitor alert emails, Markdown and webhooks print the new text. Both are alpha and may still change; the /metrics endpoints and galtea metrics keep working as before, with five changes. Excluding a revision that is already excluded keeps the first date and user. An ID of another organization answers 404, like an ID that does not exist, where some routes answered 403. A lookup by name checks metric names first, then the own names of active revisions, and answers 409 when two of them share the name. In the Python SDK, metrics.get_by_name raises the new AmbiguousMetricNameException for that 409; pass the metric ID instead. A /metrics response also carries the revision’s number (R1, R2, R3) and revisionName (its own name, or null), while name stays the revision’s own name when it has one, else the metric’s. And a create on POST /metrics with parentMetricId stores no name when the name is “R” followed by digits or another live revision of the metric holds it, and still answers 201, so an alert on that revision prints no name of its own.Docs for Each Galtea Version
The docs site now keeps the docs of every supported Galtea version from 5.10 on. Pick your version in the version menu. A page of an older version shows a banner that links to the same page in the latest version.Metrics score text and data files
On deployments that send files to the judge, an LLM-as-a-judge metric and a classifier metric now score a Galtea file of text, such as TXT, CSV, Markdown, JSON, XML or HTML, instead of skipping the evaluation. Every LLM judge model reads the file’s text, under its name and number. DeepEval-based metrics still skip a turn that carries a file. See how judges read files.Analytics coverage follows the coverage permission
A user whose group does not listanalytics.readCoverage on a product no longer receives the test-case coverage of its Analytics Details through GET /analytics/coverage, and the dashboard hides it. galtea analytics get-coverage refuses such a user, and the Analytics export leaves coverage out and keeps the scores. GET /analytics still returns the coverage to anyone who can see Analytics. galtea analytics get-version-summary reads the average score per version and the count of excluded results. See Roles and permissions.An LLM Call Counts Once
When a framework span wraps the provider span of an LLM call and both report usage, a turn and the Where does the cost come from? card of the product Analytics now count the call once, from the span below. Tokens and cost show the real spend, not twice the real spend. Turns stored before this change keep their stored totals. The card counts their tokens once from their stored spans. The extra cost that such a turn stored shows under the Version’s model. See Token usage and cost.LLM Calls Price Cached Input at the Cache-Read Rate
An LLM span now reads the cached input count that OpenTelemetry instrumentations report (gen_ai.usage.cache_read.input_tokens) and prices those tokens at the model’s cache-read price, so span and turn costs match the provider bill. A span whose reported model matches no Model now links to the model it requested, or to the name without its provider prefix (openai/gpt-4o matches gpt-4o). cache_read_input_tokens sent through the SDK or the API is now read as part of input_tokens, so a turn’s total counts it once and the cached part takes the cache-read price; a count above input_tokens is still counted apart.A link from one details page to another opens it fresh
A link from one evaluation, trace, user, subscription, model, monitor alert or connection to another of the same kind now opens that page from a fresh start. Before, an open panel or a toggle of the page you left could carry over to the new one.The first product says how many specifications it could not create
When you create your first product and some of its generated specifications fail to save, the confirmation now says how many could not be created. You can add them later from the product’s Specifications tab.Every table row with a details page offers Show Details
The row menu of every dashboard table whose rows open a details page now has a Show Details item, so you can reach that page from the keyboard as well as with a click. The runs list, the monitor alert history, the list of metrics a classifier conversion created and, for every role, the products tab of a user group now have this menu too.Lists by ID No Longer Fail When a Filter Excludes a Row
A list call that namesids now answers with the rows that match every filter, instead of a “not found” error for the rows a filter or the page size leaves out. For example, galtea evaluations list --ids <id1>,<id2> --statuses PENDING answers an empty list once both evaluations finish. An ID that does not exist still answers “not found”. In the dashboard, a user group with more members than the page size now shows its members.Readers See Cost and Latency
The Reader preset now includes Cost and latency (evaluation.readInference). Members of a user group that follows Reader now see latency, tokens and cost in Analytics, the versions list, monitors and the evaluation lists.To keep these values hidden from a group, give it its own permission list without Cost and latency. See Roles and permissions.The Scores Chart Loads Fast on Large Products
The scores-over-time chart in the dashboard no longer times out on products with many versions or many production sessions. The chart opens in seconds on the default “Last 2 years” range, and the series, values and counts are the same as before.Analytics inference values follow the inference permission
A user whose group does not listevaluation.readInference on a product no longer sees the latency, cost and token values in its Analytics Details. galtea analytics get-inference refuses such a user, and the Analytics export leaves those values out and keeps the scores. See Roles and permissions.See Where the Cost Comes From
The product Analytics Summary now has a Where does the cost come from? card. It splits the estimated cost of the period by model, by version, or in Production by end user, and the rows add up to the total cost. Each row shows the LLM spans, tokens, cost and share of cost, andgaltea analytics get-cost-breakdown returns the same data. See Token usage and cost.The changelog groups updates by release
The changelog now shows one block per release, newest first. Each block names the release version and the date the release was cut, so you can see what a given version holds. Updates from before this change show only their date.galtea --verbose No Longer Prints Your API Key
The CLI’s --verbose (-v) output now shows <redacted> in place of your API key and the values of the headers you set in GALTEA_EXTRA_HEADERS. The rest of the request and the response still print in full, so you can paste verbose output into a bug report or a CI log.Filter expressions are refused in output mappings and trace source rules
An endpoint connection’s output mapping and a trace source rule now refuse a JSONPath filter such as[?(...)] when you save them, and Galtea never runs a filter that an older saved path still holds: that path extracts nothing. Use a plain path such as $.choices[0].message.content instead. See Output Mapping.Judges Read Each File in Its Place
A judge now reads a turn as you logged it: text, files and your own elements in their order. Each Galtea file reads as[File N] where it sits, and your own elements, such as an image_url part or a file on your host, read as their JSON text. Galtea numbers the files of a session once, so a skip reason, the session page and the span page show the same File N beside a file. See Files and Evaluation.Stored Files Come Back as Files
A file stored as a Galtea file token in a trace now comes back as the file when you read it: setrestoreFiles=true on GET /inferenceResults, or restore_files=True in traces.list from SDK 5.17.0. A run also sends past turns to your endpoint with their files back, a provider byte field as raw base64, and an evaluation judge sees [File N] in place of each file. See Content Parts.Long words wrap inside chat bubbles
On the Traces tab, a long word or a one-line JSON block with no spaces now wraps inside its chat bubble. It no longer runs past the edge of the card.Every Revision of a Metric Shares Its Name
All revisions of a metric now share one name, description, tags, documentation URL and evaluation type. Renaming any revision renames the metric. Deleting a metric by its ID deletes all its revisions. Deleting a revision deletes only that revision, and the metric too when no live revision is left; the active revision can be deleted on its own only when it is the last one. A new revision can no longer change the evaluation type. An analytics exclusion still applies to one revision: the other revisions of the metric keep counting.Through the API,DELETE /metrics/{id} and DELETE /metrics/batch with the ID of a metric’s active revision now return 400 while the metric has other live revisions. Delete the other revisions first, or delete the metric by its ID.See the end user of each session and trace
Sessions and traces that Galtea ingests from OpenTelemetry now record their end user: the customer of your agent. Each trace stores the end user of its own spans, and the session keeps the first one. Sessions linked to a test case record it too. Galtea reads it from the standard keys ofuser.id, OpenLLMetry and Langfuse with no setup, and your organization or product can name custom span attributes. The sessions table of the dashboard has an End user filter and an End user column you can turn on, and galtea.sessions.list(end_user=...) filters in the SDK. See How Galtea routes a trace.Long File Names Stay in the File Row
A long file name in a file row no longer runs over the row’s controls. The name now ends with an ellipsis, and the full name shows when you hover over it. This applies to the files of a turn, the test case attachment list, the test connection result and the span page.Spans check their Galtea files like traces
A Galtea file in the input or output of a span is now checked as in a trace. A file of another organization fails a single span write with a400. In a batch, or in OpenTelemetry ingest, only that span is refused and the other spans are stored. See Content Parts.Saved Credentials Stay With Their Address
Galtea now binds the stored token, password, and secret headers of an endpoint connection to the origin they were saved for. While a connection holds a saved secret, a test of it with a URL of another origin is refused. When you save a new origin, Galtea clears the stored secrets, and you send the new ones in the same request. See Endpoint Connection Configuration.Simulated callers read IDs out in groups
In a voice test, the simulated caller now says an order number, phone number or ID in short groups, as a person does, instead of text-to-speech reading it as millions. Amounts, lists of amounts and numbers the caller already grouped stay as written.Evaluation Comparison Follows the JSON Field Match Rules
On an evaluation scored by a JSON Field Match metric, the output comparison table now uses that metric’s matching rules. A field checked by a field rule shows as a “Rule” row and no longer counts as a difference, and the strict variants mark a field found only in the actual output as failed. The metric API response now includesdefinition, the evaluator implementation that the metric runs.Simulated callers speak in short turns
In a voice test, the simulated caller now says one short line per turn and no longer pauses mid-line, so your agent does not cut in on a monologue or a long silence. Every call over a phone or WebRTC connection uses the spoken strategy, whatever the test case sets.galtea sync No Longer Shows in the CLI Help
The deprecated galtea sync command no longer appears in galtea --help, like every other deprecated command. It still runs, does nothing, and prints how to update the CLI until its removal on 2028-01-17. Take it out of your scripts.Each LLM Call Is Priced at Its Own Model
The span API and the SDK now return each LLM span’s token counts, per-token prices and cost, priced at the Model that answered the call. A turn’s tokens and cost are the sums of its spans, so a turn that calls two models costs each call at its own price, even when no Model is linked to the Version. A span that names no model is priced at the Version’s Model. See Token usage and cost.Spans Show the Model They Ran On
Each LLM span you send to the OpenTelemetry endpoint (/otel/traces), and each generation in a Langfuse file import, now shows the model that answered the call. Galtea reads it from gen_ai.response.model or gen_ai.request.model, and from MLflow and Mistral spans. When one of your Models, or a global Model, has that name, the span links to it, and the span page opens that Model. The span API answers the same as modelName and modelId. See Model.User Groups Now Also Limit Metrics, Models and Comments
A user group’s permission list now also covers metrics, models and comments, for example a group of metric authors. These entries sit under Shared across products in the Permissions tab, and its Reaches column shows what each See entry shows. A Member, Reader or Human evaluator in no user group no longer works with metrics, models or comments: add them to a group to give these actions back. Galtea metrics now stay visible to anyone whose group can see metrics, whatever products the group covers, so a team can pick a Galtea metric that its products do not use yet.A Metric Always Has One Active Revision
When you start a new revision from an older revision of a metric, the metric’s active revision now becomesLegacy. Before, the metric kept two active revisions, and it was not clear which judge was in use. At release, we also fixed existing metrics that had several active revisions: each keeps one active revision, usually the newest, and the others become Legacy. Their evaluations are kept. See Revisions and Analytics.OpenLLMetry Spans Show Their Input, Output, Type, Tokens and Session
Apps instrumented with Traceloop’s OpenLLMetry now send complete spans with no Galtea attribute. The@workflow, @agent, @task and @tool spans show their input, output and type, so a turn’s input is the message your workflow received, not an LLM prompt. LLM-call spans from older OpenLLMetry releases now show their messages and token counts, and turns that share a session ID set with Traceloop.set_association_properties group into one session. See Send OpenTelemetry traces to Galtea.See Each Product’s Color Beyond the Products Table
The color and icon tile that a product has in the Products table now also marks it in the sidebar product selector, at the top of its pages, and in the breadcrumb of pages under it. The open product is easier to recognize at a glance, and the same product has the same color everywhere, in light and dark mode.See the Cost per Session Over Time
The product Analytics Summary now shows the estimated cost of one session by day, week or month, under the failure trend and on the same range. A band shows the usual range of session cost, from the 10th to the 90th percentile, and the tooltip gives the cheapest and the most expensive session, so you can match a cost jump to a date. The same data is available withgaltea analytics get-session-cost-over-time.Users of a Frozen Organization Keep Their Role
When Galtea freezes an organization, the users it keeps now keep their role instead of becoming Reader. The organization is read-only for them while it is frozen: they can see everything their role allows, but they cannot create, edit or delete anything except their own account settings. When the organization is unfrozen, they can write again with the same role.User Groups Can Follow a Preset, and the User Forms Show Group Access
A user group now either follows a preset or holds its own list. A group that follows Editor, Human evaluator or Reader changes with it when Galtea changes what the preset grants. Pick Custom, or Customize under a preset, to keep today’s permissions as the group’s own list and edit it. Set this in the group’s create and edit forms, or with Edit permissions in its Permissions tab. Through the API, sendpermissionPreset to POST /userGroups or PATCH /userGroups/{id}. See Presets.The invite and edit user forms now offer the Owner and Member roles, not Reader or Human evaluator. For read-only access, invite a Member into a group that follows Reader. Each group in the form shows its products, users and permissions, and a summary shows the permissions the user gets on each product. When your organization has no group yet, a dialog after the save offers to create the first one with the new Member. The Users tab now shows each user’s role and their groups with their permissions, and flags a Member with no group. Pick a group there to see its products, users and permissions. On a metric’s page, the User Groups card now says that a user must be in one of those groups to evaluate the metric.Reader and Human Evaluator Roles Retire
From this release, a new or edited user can no longer get the READER or HUMAN_EVALUATOR role. Give them the MEMBER role and add them to a user group with the Reader or Human evaluator preset instead. Galtea moves existing users with one of these roles to the MEMBER role in a one-off run, and they join a copy of each of their groups, so their access stays the same. Until that run, they keep their role.Inline file bytes stay out of judge prompts
An image or a compressed PDF logged inline as adata: URI no longer reaches an LLM-as-a-judge as base64: the judge sees [inline bytes dropped: <type>, <size> bytes] in its place. A text file logged the same way, such as a CSV or a JSON file, now reaches the judge as its text, which costs fewer tokens than base64. Metrics that compare values exactly, such as JSON Field Match, still read the stored value. To have an image or a PDF scored, upload it to Galtea as a file. See File inputs.Model Pickers Show Which Files a Model Reads
Each judge model in the metric form, in AI metric generation and in the classifier model picker now carries a tag: Reads text, images and PDFs, Reads text and images, Reads text and PDFs, Text only, or File support unknown when the model catalog does not say. A note under the picker states what the picked model does with a turn that carries a file, and the metric page shows the tag beside its model. Every classifier model shows Text only, because a classifier metric skips any turn with a file uploaded to Galtea on a side it reads. See Troubleshooting skipped evaluations.Skipped Evaluations Name the File and What Refused It
When the API skips an evaluation before it runs, the reason now says which file, on which turn, and what refused it: the judge model, the model catalog, a file type no judge reads, a metric with no judge model, or a metric that reads text only. DeepEval and classifier metrics now end Skipped when any turn carries a file on a side they read, even if another turn has text. An LLM-as-a-judge metric whose judge model reads images or PDFs gets the file on deployments that send files to the judge, and a human-evaluation metric is no longer skipped for a file. The dashboard file row no longer carries a judge tag, and the evaluation page warns only about audio with no transcript. See Troubleshooting skipped evaluations.Your Own Files Stay as Sent Beside Galtea Files
Galtea now reads each element of acontent list on its own, in a trace input, a trace output and a test case input. Your own file URLs, image_url parts and other parts are stored exactly as sent, even beside a file uploaded to Galtea. Galtea recognizes its own files by their storage key on any host. An edit that keeps a Galtea file unchanged no longer fails when the file was deleted, and a trace keeps a missing Galtea file, stored without its signature or its sizeBytes. The SDK now refuses a URL that is not a Galtea file in input_file_paths or output_file_paths before it uploads anything. See Content Parts.Resend a Pending Invitation
The users list and the user page now mark a member who has not accepted their invitation with an Invitation pending status; hover it to see when they were invited. An owner or admin can send them a new invitation email from the row menu with Resend invitation, with the same role and groups, instead of deleting the member and inviting them again.Charts Draw Each Series in a Soft Fill and a Strong Line
Charts that compare versions or metrics now draw each series in two tones of its color: a soft fill for bars and bands, and a stronger tone for lines and outlines. A series keeps its place in the palette; in the dark theme its lines now draw in the lighter tone and its bars in a softer one. This covers the version comparison bars and their legend dots, the scores over time chart and the monitor score chart.Every Deprecation Names Its Removal Date
Every deprecated SDK call, API field and CLI command now names its removal date, and none is removed before 17 January 2028. SDK warnings sayRemoval: 2028-01-17. instead of Removal: not scheduled., and the CLI warnings name the date. In the API reference, the deprecated specificationId field of datasets carries it as x-sunset, with specificationIds as its replacement. Deprecated metrics keep their own schedule.Search Any Table by Name Right Away
Tables with a name filter now show a search field next to the Filters button. Type a name and the table filters at once, without opening the Filters menu first. The field and the Name row in the Filters menu always show the same value, and the search stays in the link like the other filters.Products Stand Out in the Products Table
Each product now has its own color and icon tile, the same every time you see it. The organization’s products table lists the products that received sessions in the period first, each with a small chart of its sessions over the period; hover a point to see the dates and the count. The products with no session follow behind a row that counts them and opens or folds them, and each one offers a link to set up its evaluation to users who can create or edit specifications. Charts that compare versions or metrics also use a new set of colors that are easier to tell apart.Pinned products still come first, in the sort you pick. The API and the CLI (galtea analytics list-products) now also lead with the products that received a session, return each product’s sessions per slice of the period with the dates of each slice, and send the count of those leading products in the X-Active-Total header.Invitation Links Stay Valid for 14 Days
When you invite someone to your organization, the invitation link in the email now stays valid for 14 days instead of 12 hours, so they have time to accept it.See Whether a Product Gets Better or Worse
The product Analytics Summary now shows the share of evaluated sessions with a critical failure and with minor issues by day, week or month, with a band that shows how sure each point is: the band is thin when most received sessions were evaluated and wide when few were. A second period selector widens the chart, and the page period stays tinted, so you can compare it with what came before. The same data is available withgaltea analytics get-session-outcomes-over-time.SDK Creates Say When the Record Was Stored
When the platform stores a new record but cannot return it, an SDKcreate now raises WrittenResourceUnreadableException instead of returning None. The record exists, so do not create it again: read it with the service’s get or list method. See Errors.Install an Older CLI With Any Package Manager
APT and DNF now keep every CLI release from 4.24.0, as PyPI already did, and Homebrew keeps the newest patch of each minor version. You can install the CLI that matches an older Galtea platform with the tool you already use. When a CLI is too new for its platform, it now prints the install command for every tool. See Install an older version.LLM Judges Skip Turns That Carry a File
On deployments that do not send files to the judge, an LLM-as-a-judge metric, custom judge metrics included, now ends Skipped when a turn it reads carries an attached file, and the reason names the file. Before, the judge scored the turn’s text alone, so the score could describe a document the judge never saw. DeepEval-based and classifier metrics still score the text. See Troubleshooting skipped evaluations.See How a Product Is Going and What Fails Most
A product’s Analytics has a new Summary view next to Details, which still opens by default and shows the charts as before. Once you pick Summary, Analytics keeps opening on it. Summary shows the product’s sessions and estimated cost for one period, and lists the specifications that failed most, critical failures first. Each specification shows its share of failed sessions, and whether it got worse or better than in the previous period. Its name opens the specification, and a click on its row opens the sessions that failed it on the specification’s Sessions tab, where a new Evaluations filter keeps only the failed ones. The same list is available withgaltea analytics list-failing-specifications.See and Open Voice and File Outputs the Same Way
The agent’s answer now gets the same file and voice marks as the input. The evaluations table’s Output column, the sessions table’s Last Output column and the evaluation conversation summary show a file or voice icon beside the answer’s text, and name the file when the answer is only a file, instead of an empty cell. The full-text dialog lists the answer’s files and plays its audio.See All Your Products at a Glance
The top of the organization’s products page now sums up the products the table shows: how many sessions came in, how many were evaluated, and how many of those had no issues, minor issues or a critical failure. It also shows the estimated AI model cost per session and in total, and, when any got worse, lists the products whose share of critical failures went up the most, each linking to its analytics. The boxes follow the period, the view mode and the product and tag filters. The same numbers are available withgaltea analytics get-sessions-overview, get-cost-overview and list-changes.Text Content Parts Count as Text
When a turn’s input or output holds{"type": "text", "text": ...} content parts, the judge, monitors, past_turns and the dashboard now read their text, next to user_message or assistant_message and any other fields. Before, that text could be lost, or the judge got the whole value as JSON. In the SDK, .input, .actual_output and the message traces.generate sends now include this text, and an audio-only value gives its transcript instead of None. An output’s other fields are now scored beside assistant_message. Every CSV export of an input or output now writes the stored value as it is, so you can import the file again. See the Input field of a trace.Only owners manage billing
Only an organization owner can now change the plan, open the billing portal or buy extra credits. Members still see the plan. See Roles and permissions.See Which Product Needs Attention
The organization’s products page is now a table that shows, for each product, how many sessions came in, how many were evaluated, how many failed a critical specification, and whether that rate got worse than in the period before. It also shows the estimated cost per session. Pick the period at the top right: by default it covers the last 30 whole days, today included. Sort by most columns, pick the products or tags to show from searchable lists, and pin the products you follow so they stay first. Click a row to open the product’s analytics. The same numbers are available withgaltea analytics list-products.Retirement Emails Name the Evaluator Model
The retirement warning now arrives as “Action needed: an evaluator model used by your metrics is retiring”, and the notice after the date speaks of evaluator models too. Each email suggests other models of the same kind you can move to. To list the models of one kind, passkind to GET /evaluatorModels (LLM_JUDGE by default); each model now also returns its kind.Limit What a User Group May Do on Its Products
You can now give a user group a permission list: the product actions its users may do on the products the group links, for example only reading analytics. A group with no list keeps every action its members’ role allows, and the role still caps every list. Set a list withuser_groups.update, PATCH /userGroups/{id}, or the new Permissions tab of the group page.In the tab, an owner or admin starts from the Reader or Human evaluator preset, ticks or unticks each action, and saves. The grid uses the dashboard’s names, so the Trace row covers traces and the Span row covers spans. Ticking an action also ticks the reads it needs. Reset to default, or the Member preset, gives the group back every action its members’ role allows.Mark a Specification as Critical
You can now mark a Capability or Policy specification as critical, when its failure is serious for your business. Use the Critical switch in the dashboard,is_critical in specifications.create(), or galtea specifications update for an existing one. Critical specifications show a red Critical pill, and the Critical filter lists only them or only the others. specifications.list() filters the same way. When the AI writes a specification for you, it also suggests the Critical flag, and you can change it before you save.Each CLI Version Has a Fixed Set of Commands
The CLI now carries the commands of its own release, instead of downloading them from the API. An API update can no longer change your commands or flags between two runs. To get new commands, update the CLI.galtea sync now does nothing and will be removed, and GALTEA_CACHE_TTL has no effect, so take both out of your scripts. See Getting new commands.The SDK Checks That It Works With Your Platform
When you create theGaltea client, the SDK now checks that its version works with the Galtea platform it calls. If it does not, it stops and tells you which version to install. This matters most on a self-hosted platform: a newer SDK than the platform supports now stops at start-up, so install the version the message names. The CLI shows the same information with galtea platform-compatibility. See Upgrading the SDK.See and Sort Sessions by Number of Turns
The sessions table has a new Turns column with the number of turns in each session, deleted turns left out. Click its header to sort the filtered list. Sorting works on up to 50,000 sessions: for more, narrow the list with the Created At or Status filter first. In the API,GET /sessions returns turnCount and can sort by it.The Simulated Caller Stops When Your Voice Agent Cuts In
On a phone connection, and now also on a WebRTC connection, the simulated caller stops talking within about one second when your agent makes a sound over it. A short sound such as “mm-hm” is enough. The turn keeps only the audio that was played, and its transcript is cut to match it as closely as possible.Large Evaluation Runs Wait Out Rate Limits
When an evaluation run goes over a judge model’s per-minute quota, Galtea now waits for the quota to free up, for up to 5 minutes, instead of failing the evaluation. Large runs end with fewer failed evaluations. If the provider asks for a longer wait, the evaluation still fails at once.Unlink Rows by Unchecking Them or From the Row Menu
In the link dialogs of a specification and of a user group, rows that are already linked now show as checked. Uncheck a row and click Save links to unlink it. Unlinking never deletes the row itself.- Row menu: a specification’s Datasets and Metrics tabs offer Unlink from specification on each row, and a group’s Products tab offers Unlink from group. In a group’s Products tab, the row item asks you to confirm first, because the group’s users lose access to the product.
- No Delete in linked lists: the row menu of a specification’s Datasets and Metrics tabs and of a group’s Metrics tab no longer offers Delete, so one misclick cannot delete a dataset or a metric.
Voice Turns With Extra Fields Are Scored on What Was Said
When a voice turn’s input has your own fields next to the audio, for example alanguage field, the judge now reads the spoken transcript first, then those fields. Before, it could score the turn on your fields alone. When no turn of a session has text, a metric that reads the input is now skipped instead of scored. See the Input field of a trace.The Dashboard Picks Up User Group Changes Without a Reload
When an admin changes your user groups, the dashboard now applies your new access when you come back to its browser tab, at most once every 30 seconds. New products appear, and a page that was denied opens. If you lose access to the product you have open, you land on the organization’s Products tab.Import LangSmith Dataset Exports as Sessions
Import traces now accepts a LangSmith dataset export (JSONL), as well as a Langfuse export. Each conversation becomes one closed session, and each example one turn. In the dashboard, choose the file type in the new Format field before you import. In the API,POST /sessions/import takes an optional format, and detects the format from the file when you leave it out. Import sessions from a trace export explains how examples are grouped.The Best Version Is Shown in Bold in Comparison Tables
In the version comparison tables of the specification and metric analytics, the best version of each row is now in bold, instead of carrying a grey pill. Every row keeps the same height. A row with only one scored version shows no best version.Members See Only the Products of Their User Groups
A Member now sees and works only in the products that their user groups grant, like a Reader and a Human Evaluator already did. Nobody loses access with this change: every existing Member was added to a group that grants all products. Owners and Admins are not affected.- A new Member with no group sees no product until an Owner or Admin adds them to a group. The invite and edit user forms warn before they leave a Member with no group.
- Only a Member in a group that grants all products can create a product.
- Metrics stay shared: a Member still reads and edits every metric of the organization.
Evaluations Cut Off by the Token Limit Are Skipped, Not Failed
When a metric’smax_output_tokens is too low for the judge to finish its answer, the evaluation is now Skipped, with a reason that names the limit. Before, it Failed as a platform error. To fix it, create a revision of the metric with a higher limit and evaluate again. A metric now needs a max_output_tokens of at least 256, to be created or optimized: see metrics.create().The CLI Works Behind a Gateway That Serves Galtea Under a Path
The CLI now works when your API gateway serves Galtea under a path, such ashttps://gw.example.com/org/galtea, and your key comes from GALTEA_API_KEY. Before, the CLI showed no API commands in this setup. Pass the full URL, path included, to --host, as Behind an API gateway shows.Creating From a User Group Tab Links the New Entity
A user, metric or product you create from inside a user group’s own tab now joins that group automatically, instead of landing unlinked.- Users tab: the invite form opens with the group pre-selected, in the Advanced options.
- Metrics tab: a blank metric starts as Human Evaluation with the group selected. The form clears the group when you switch to any other metric type, and puts the tab’s group back when you return to Human Evaluation — a different group you picked before switching away is not kept. Starting from a template or revision keeps that source’s own groups and adds this one.
- Products tab: the name field notes which group the new product will join.
userGroup.update), and it is dropped the moment you navigate to a plain creation page instead of the group tab. The SDK’s metrics.create also takes an optional list of group ids to link on creation; there is still no products.create in the SDK. Creating a metric from the AI generation tab of the metric creation page does not link a group yet.JSON Field Match Checks a Field by Rule
A JSON Field Match test case can now give a field a rule instead of an exact expected value, by writing an object with a$ key: $norm (same text once accents, case, spaces and every other character that is not a letter or digit are dropped), $regex (an RE2 pattern, case-insensitive by default), $num (a decimal comparison within a tol tolerance) or $date (reads common date formats, with an order of DMY, MDY or any). The same four rules work the same way across the strict, normalized and normalized-strict variants.A rule field shows as a neutral “Rule” row in the evaluation details, rather than a red mismatch, and it isn’t counted as a difference. A test case with a broken rule (an invalid pattern, an unknown option, or a pattern too large to run) skips the evaluation instead of failing it, with no retry.This release also changes what happens when the expected or actual output itself can’t be read, for every JSON Field Match evaluation, rule or not: an expected output that isn’t valid JSON, or output nested too deeply, now skips the evaluation instead of failing it as a platform error. If you track pass rates against past FAILED counts for this metric, expect some of those to move to skipped going forward.See and Open Voice and File Inputs the Same Way
The sessions table’s First Input column, and the evaluations table’s session-wide Input column, now show the caller’s first turn instead of the agent’s opening greeting, scanning up to the first three turns of the session. In the evaluations table, an input with nothing to show — including a single-turn evaluation whose input is an empty envelope — now leaves the cell blank, matching the Output column, instead of showing a dash. The same empty input exports as an empty CSV cell.Audio inputs get the same voice icon files already had, shown even next to a transcript, and alone when there is none. A cell holding only audio or only a file, in the sessions, evaluations or test case tables, now carries the same hover button a long text cell has, so one click opens the player or the file instead of the cell being a dead end. Long file names are truncated with an ellipsis instead of spilling into the next column.Build a Monitor Step by Step, with Its Cost in View
The create and edit forms of a monitor now follow the order a session travels through it: Filter sessions first, then Sampling and cost. The preview walks the same funnel, from the closed sessions in the read window, to the ones that pass your filters, to the sampled share, ending in≈ N credits per month with capped at N when your credit cap is lower, and the N sessions × N credits each behind it.- Pick sessions by how many messages they hold. A monitor takes up to 10 conditions, joined with AND, each comparing
messages(user and agent) oruser_messagesusinglt,lte,eq,gteorgtagainst a value from 0 to 200. In the SDK, passmessage_conditionstomonitors.create()andmonitors.update(). It replacesmin_user_messages: a stored0becomes no condition, and a storedNbecomesuser_messages gte N. A new monitor starts atmessages gt 1, so a session holding a single agent turn with no reply is skipped. - Conditions no session can meet are refused.
messages gt 5together withmessages lte 3is rejected on create, update and preview, and the dashboard disables Enable monitor and Preview while the rows contradict each other. - The form lists the specifications and metrics you chose, and the catalog opens in a dialog from Add specifications or metrics. The one-time Add from Specification metric copy is gone.
- Save as paused stores a new monitor without starting it. Coverage is forward-only: a monitor never scores sessions that predate it, and resuming it does not go back over the sessions that arrived while it was paused.
- Turning the credit cap on prefills 15% of your organization’s monthly credit plan, instead of a flat 5000. The 5000 remains the prefill when that plan is unknown.
Analytics Shows What Has Not Been Evaluated
A specification, metric or version with no score now reads Not evaluated on a dashed, empty track, so it no longer looks like a zero. Both the Development and Production views now list every specification that links at least one metric, and every metric, that your filters still allow, rather than only the ones that happened to be scored. The specification picker of the Metrics section offers the unscored specifications too.In the version comparison tables, a version with no score for a listed row gets its own Not evaluated cell instead of being dropped from the row. The tables still list only rows that at least one version scored; the bar view is where an entirely unscored specification or metric appears.The selected version’s detail column adds Coverage: how many of its test cases were evaluated, as a count and a percentage, split into Evaluated, Not evaluated and Total, and measured against the most-evaluated version where there is one. A Coverage by version table below the charts gives a row per compared version it has coverage for. Both appear only for roles whose permissions include analytics coverage, so a reader or human evaluator does not see them.Every Score Names the Evaluations Behind It
Each score in the Scores panel, on the specification and metric bar charts, and in the version comparison tables now showspassed X/Y beside it, so you can tell a strong result from one backed by a handful of evaluations. A score built from fewer than 10 evaluations is muted, with a tooltip saying it is weak evidence. The Average Score carries no count, since it counts each metric once however many evaluations it holds.The count is one per evaluation, not per test case. A specification with 5 metrics on 2 test cases therefore counts 10 evaluations and is not muted, though it covers only 2 test cases. Read it with the coverage figures above.Manage a User Group from Its Own Tabs
The Users and Metrics tabs of a user group now link and unlink, instead of only creating. New User becomes a menu with Link existing user and Create new user. The Metrics tab gains Link existing metric, which offers Human Evaluation metrics only, and replaces Delete with Unlink from group, which confirms first and leaves the metric itself in place. A metric that already holds evaluations, and a Human Evaluation metric that belongs to no other group, cannot be unlinked. On the Products tab, Link products becomes a Register Product menu, and its dialog shows the products the group already has as checked and not selectable, with an Already linked tooltip. The group list adds a Products column that reads the number of linked products, or All.Send Custom Headers Through an API Gateway
When Galtea sits behind an API gateway that expects headers of its own, both the SDK and the CLI can now send them.In the SDK, passheaders to Galtea(), or set GALTEA_EXTRA_HEADERS to a JSON object, as described in Custom API URL and headers. The argument replaces the environment variable rather than merging with it. They travel with every API call and with span exports. authorization, user-agent, content-type, content-length and host are dropped with a warning, a malformed name or value raises an error while the client is built rather than on the first call, and no header follows a redirect to another origin. File uploads and downloads go to signed storage URLs, which do not carry them.The CLI reads the same GALTEA_EXTRA_HEADERS, and keeps headers held on its default profile for the host you logged into, merging the two per key with the environment variable winning a clash. See Behind an API gateway. Login sends them too, and a gateway that refuses the call before the API sees your key is now reported as the gateway’s own rejection, apart from a rejected API key.Documentation and Examples
A new guide covers the Claude Code tracing plugin: installation, its options, what each conversation sends, redaction, when a session closes and reopens, and how to stop logging. An interactive conversation arrives as a production session and each turn as a trace, with model calls, tool calls and subagents as spans, so a monitor can score a sample of those sessions as they close. One-shot runs, such asclaude -p and the Agent SDK’s query(), are not logged.Each session also lands on a version named after the model of its first answer, which gives its turns a cost. That cost is an estimate at list prices: every model call of a turn, subagents included, is priced at the main model’s rates, and cache writes are left out. A model name Galtea does not know gets no price, and a Galtea that refuses the version, such as an older self-hosted one, puts the session on the product’s latest version with no cost at all.Platform Improvements
- A monitor edit saves while it holds a metric that is no longer active. Only a metric family you add needs an active metric, so renaming a monitor or changing its sampling no longer fails over a family that is retiring, disabled or deleted. The form and the monitor’s own reads name each family and its state — Retiring, which still scores, Disabled or Deleted — instead of showing a raw id. A monitor whose families have all been deleted still reads as Active and scores nothing.
- The Development view of Analytics opens on the latest versions over all time. On a product with more than five versions that have results, it compares five: the four newest, plus the version with the highest total average score, or the fifth-newest when the best is already among them. A removable Latest versions + best pill marks it; remove the pill to compare every version. The Production view keeps its own defaults.
- The Scores comparison radar card is gone from Analytics, and the Coverage card takes its column.
- The Analytics Scores panel paginates, 10 rows to a page with the page kept in the URL, instead of scrolling inside the panel. Changing the sort, grouping, filters or view returns you to the first page.
- The comparison tables keep their rows when you highlight a version, instead of blanking to loading rows.
- The Analytics metrics filter tells same-named metrics apart. A repeated name gains a suffix:
Galteafor a Galtea metric, with its status when Galtea metrics share the name, or the status and creation date for one of your own. Labels that still match add the last six characters of the metric id. - The Ask Val launcher lifts clear of a button at the end of a page instead of covering it, by up to half the window height. A control it covers mid-page still needs a scroll.
- A deprecated SDK call names its removal date in the warning it raises, so you can plan the upgrade. One with no date set yet says
Removal: not scheduled.Deprecations the platform owns, such as legacy metrics, carry no date.
Link a Dataset to Several Specifications
A dataset can now link to more than one Specification of the same product, as long as they share the same dataset type (Accuracy, Security, or Behavior). Before, linking a dataset to a new specification silently dropped its old link.- The dataset details page lists every specification it links to. Each one opens its own page.
- The edit form lets you add or remove links. Changing the list shows an Edited badge you can click to undo.
- Running an evaluation on several specifications that share a dataset scores it once, with the combined metrics of every specification in that run. Running it from one specification’s page still scores the dataset with only that specification’s metrics, even when the dataset links to others too.
- In the SDK, pass
specification_idstodatasets.create()anddatasets.update(). The olderspecification_idstill works but is deprecated and only remembers the oldest link. Agent runs need SDK 5.11.0 or later to honor every link; an older SDK only reads the oldest one.
Analytics Opens as a Worst-First Table
The product Analytics page now opens its Specifications, Metrics, and Inference sections as a table, so you can scan every score without reading a chart. Specifications and Metrics sort worst score first; Inference keeps its usual row order, since latency, cost, and token counts are not directly comparable. Click a row to open that specification’s or metric’s page. Switch back to Bar Chart from the view selector: your choice is remembered across a refresh or a shared link, and the default Table view does not clutter the URL.Dashboard Shows Only the Actions You Can Use
The dashboard now shows a write action, and opens a create or edit page, only when your role allows it. A reader or human evaluator no longer sees buttons they cannot use: a page they cannot open now shows a clear access-denied message, and a review mark or score they cannot change shows as plain text instead of a control. The platform assistant’s write-approval card checks the same permissions: when your role cannot do what it proposes, the card explains why and offers only Dismiss.Platform Improvements
- Invited people get the Galtea invitation email. Before, Keycloak’s own “Update your account” email went out instead of “Welcome to Galtea! You have been invited to join the Galtea platform.” The sign-in link does not change.
- The monitor form now shows a specification’s metric families correctly even when a specification links a family only through an older metric revision. Before, such a family showed as empty or disabled in the form, though the monitor already scored it correctly.
- A provider’s content block now ends an evaluation with a clear reason, telling you to pick a different evaluator model, instead of a generic failure. In the Val assistant, a block partway through an answer now ends the reply with a clear error instead of cutting it off with no explanation.
- Creating or updating a dataset now returns the full dataset, not just its id, so you do not need a follow-up read. Organization, product, and dataset writes also accept
Prefer: return=minimalwhen you only need the id, or no body, back.
Link One Dataset to Several Specifications
A dataset can now link to several Specifications of its product. An evaluation that names several of them runs the dataset once and scores it with the metrics of all of them.- In the SDK, pass
specification_idstodatasets.create()anddatasets.update().specification_idis deprecated and holds the oldest link. - Agent runs need SDK 5.11.0 or later. An older SDK reads only the oldest link of a dataset, so
evaluations.run()with anagentskips a dataset linked to several specifications for every specification but its oldest one.
Alert Rules Can Watch a Whole Specification
A monitor’s alert rule can now watch one Specification the monitor binds, not only one metric family.- Pick a Specification or a metric. The rule form’s Watches field has Specifications and Metrics tabs. It opens on Specifications when the monitor binds one with a scored metric; you cannot switch tabs once the rule exists. Add one rule per target to watch several.
- A session fails the rule when any metric the Specification brings fails it: a score at or below the bad metric score, or, for a “Sessions get no score” rule, an evaluation that settles with no score. A session that fails two of the Specification’s metrics still counts once. See Rules on a Specification.
- The rule follows the Specification. Link a new metric to it and the next check watches that metric too, with no rule edit.
- If the Specification later brings no scored metric, the rule stays and opens no new alert. You can still edit it; the form warns and names the fix. An alert already open keeps recovering on the metrics it measured when it opened.
- The rule list gets a Watches column, and the rule page names its Specification or metric in its condition. The alert and webhook payloads name the Specification and the metrics that caused it.
See the Specification and Criteria Behind a Score
An evaluation’s Verdict card now shows which Specifications the score answers and what the judge graded it on, so a reader who did not write the metric can judge the score on the same card.- A Specification line above the metric names the Specifications of the evaluation’s product that link the metric, each linked to its page. One shows its description; several show the count and names.
- A Criteria panel, closed by default, opens on the full metric description, then the classifier question with its good answer, or the judge prompt, rendered as Markdown.
- The panel header names the revision that produced the score: its creation date, its metric id, and a status pill (Legacy, Disabled) when that revision is no longer active.
Platform Improvements
- Specification lists load their dataset counts past 20 rows. A dashboard filter sent array ids as indexed query parameters (
ids[0]=a&ids[1]=b), which the API’s query-string parser silently turned into an object above 20 ids and rejected with a 400; the specifications list showed “Missing dataset” on every row once a page passed that count. Array filters now go out as repeated keys, the form the API publishes and the SDK already sends. - Picker dialogs no longer hide their create button. The Endpoint Connection, Phone Connection, WebRTC Connection and Model pickers, and the Specification and classifier Session pickers, now give their toolbar a title row, so the dialog’s close button no longer sits on top of the button beside it.
One Filters Menu for Every Table and for Analytics
Every table that has filters, the trace Spans list, the monitor alert history and the Analytics page now open them from one Filters button.- One row per filter. Each value shows as a pill that you remove with its ×. A badge on the button counts the active filters.
- Remove all filters clears the filters, text searches included, and puts a filter that has a default back on that default. On alert history the default time range is Last 7 days, which always shows as a pill.
- The same 9 time ranges everywhere. Table date filters such as Created At, alert history and Analytics list Last hour, 6 hours, 12 hours, 24 hours, 7 days, 30 days, 90 days, Last year and Last 2 years beside the calendar. A range moves with the clock: “Last 7 days” in a saved link means the 7 days before you open it. Dates picked in the calendar stay fixed.
- Analytics keeps its filters in the URL. The time range, datasets, versions, metrics and languages survive a reload, and a shared link opens with them. Group by stays outside the menu.
Set the Score That Counts as a Pass for Each Metric
Every metric now has a pass threshold: the score, from 0 to 1, at or above which an evaluation passes. Metrics without one keep 0.5, so existing results do not change.- Set it at creation. In the dashboard, pick the evaluation type, then set Pass threshold (%) under Advanced Options; a new metric starts at 50. In the SDK, pass
pass_thresholdtometrics.create(). The API takespassThreshold. It is fixed once the metric exists: create a revision to change it, and the revision starts with the threshold of its source. - Read each verdict. An evaluation now carries
passedand the threshold it was judged at:metric_pass_thresholdin the SDK,metricPassThresholdin the API.passedis null when the evaluation has no usable score. Score pills in the dashboard are green at or above the threshold and red below it. - Filter by verdict. The evaluation lists have a Result filter and a hidden Result column, which the CSV export includes. In the SDK,
evaluations.list(passed=True)keeps the evaluations that passed. The API takespassed=true|falseonGET /evaluations. - Optimization follows the threshold. The optimizer judges at the metric’s pass threshold, and an optimized revision keeps it.
Review Human Evaluations One After Another
The Human Evaluations page is now a hub with three tabs: Evaluate, Pending and Submitted. You need the human evaluation or human annotation permission, and a user group, to see it. See Annotate in the Dashboard.- Score without a dialog. The Evaluate tab opens your next pending evaluation and reserves it for you. Click Save & Next to submit and move on, or Skip to give it back to the group.
- Pick a verdict. The score starts on Incorrect, Partially correct or Correct, stored as 0, 0.5 and 1. Switch to Percentage or Fraction for any other human evaluation score.
- Keep the guidelines in view. On a window at least 64rem wide and 37.5rem tall, the guidelines, score, reason and buttons stay on the right while the conversation scrolls on the left.
- Attach evidence. Select a phrase in the conversation, click the evidence icon, and write a note. The Evidence (N) button next to Reason lists them for edit or removal. Evidence is saved as text at the end of the reason, so
human_reasonin the SDK returns it too. Only the Evaluate tab offers evidence, not the Manual Evaluation dialog. - Limits. Your reservation ends when you leave the tab or close the browser. Otherwise Galtea releases it between 2 and 2.5 hours after you took it. Through the API, releasing an evaluation that you already submitted now answers 400 and keeps you as its author.
Alert When a Monitor’s Sessions Get No Score
A monitor alert rule can now warn you when evaluations stop producing scores.- Pick “Sessions get no score” under Alert when in the monitor’s Alerts tab. The API type is
NO_SCORE_RATE. Set Threshold (%): a percent above 0 and at most 100, with a default of 20. - A session counts when its latest evaluation failed or was skipped and has no score, a classifier “Does not apply” answer included. Pending and cancelled evaluations do not count, and a human score counts as a score. On a metric where “Does not apply” is normal, set the threshold above that normal rate.
- The rule needs at least 5 evaluated sessions in its window before it opens or closes an alert. Until then, an active rule shows a Waiting status and a note with the session count. Through the API, a
NO_SCORE_RATErule carries the same counts inreadiness, which is null when Galtea cannot read the count. - The alert closes at the threshold minus its square root, rounded down: a 10% rule closes at 6%.
- Limit: a failed evaluation counts in the window in which it was created, not when it failed, and an evaluation that never finishes does not count.
POST /monitors/{monitorId}/alert-rules now answers 201 with only { id }; read the rule with a GET. Update, pause and resume now answer 204 with no body. See Rule types.Send Monitor Alerts to a Webhook or a Slack Channel
A monitor alert rule can now post to a webhook URL, next to or instead of its email recipients.- Add a Webhook URL to a rule in the monitor’s Alerts tab, or send
webhookUrlthrough the API and the CLI. A rule holds one URL, and now needs at least one email address or a webhook URL. - Galtea sends one signed
POSTwhen an alert opens and one when it recovers. The JSON body carries the same fields as the alert email, with up to 25 session links. When a delivery times out or gets a 408, 429 or 5xx answer, Galtea tries again, up to 6 attempts in about 5 minutes. Any other error stops the delivery at once. - A Slack incoming-webhook URL shows a readable message, with links to the alert and, when it opens, to up to 10 sessions. Mattermost and Rocket.Chat show the same message from the
textproperty. - The URL is write-only. The dashboard and the API show only its host, and its port when it is not the default. An address in a private network is refused.
- Check the signature. Each rule has a secret. In the dashboard, tick Show the signing secret after saving when you save the URL; on a rule that already has one, the box reads Show a new signing secret after saving and replaces the secret. Through the API or the CLI,
POST /monitors/{monitorId}/alert-rules/{alertRuleId}/regenerate-webhook-secretorgaltea monitors regenerate-alert-rule-webhook-secretreturns a new secret. See Webhooks.
Monitor a Product by Its Specifications
A monitor can now score the metrics that your Specifications link, not only metrics you pick one by one.- Bind Specifications. In the monitor form, select them on the Specifications tab. In the API, send
specificationIdsonPOST /monitorsorPATCH /monitors/{id}. In the SDK, passspecification_idstomonitors.createormonitors.update.metric_group_idsis now optional. - The monitor scores both sources. It scores the families you picked plus the families of each bound Specification. Read the full set in
scoredMetricGroupIds;metricGroupIdskeeps only your picks. - It follows the Specification. When you link or unlink a metric on a bound Specification, the monitor scores the new set from its next scan.
- Updates replace the list.
specificationIdson an update replaces the bound Specifications. When an update removes a family from the monitor, its alert rules are deleted. - Policy and Capability only. A monitor refuses an Inability or a Specification of another product, and a bound Specification cannot become an Inability.
- Some metrics are skipped with no error: Self-Hosted metrics, metrics that cannot score production sessions, and Classifier metrics while they are off.
- The empty-scope check runs only on save. If a bound Specification later loses its metrics, new sessions are not scored, and an alert rule on that family opens no alert.
- The form shows current metric revisions only, so it can show fewer families than the monitor scores.
Route OpenTelemetry Traces to a Product and Version
A collector that sends only standard resource attributes, such asservice.name and service.version, now reaches the right product and version. See How Galtea Routes a Trace.- Find the product by its key. The organization lists 1 to 10 resource attributes that identify a product. A new organization uses
service.name. Each product gives its own value for each attribute (otelProductKeyValues), and it can use its own key instead. A trace must carry every attribute of the key, and matching is exact. When several products match, the key with the most attributes wins, and a tie routes nowhere. - Find the version by its attributes. The organization lists up to 32 resource attributes, and a product can replace that list. Galtea finds or creates the version whose facts are the listed attributes that the trace carries. The list is empty until you set it.
- Set it on the new OpenTelemetry card of the Update Organization page and in Advanced Options of the product edit form, or with
PATCH /organizations/{id}andPATCH /products/{id}(galtea organizations updateandgaltea products updatein the CLI).
PATCH /products/{id}andPATCH /organizations/{id}now refuse unknown fields with an error, where the product route ignored them before.- Two products cannot hold the same key values, and the organization cannot change its key attributes while a product that follows the key holds values. Both answer
409. - A trace that matches no product is rejected, with the reason in the response’s
partialSuccess. - Routing happens once, at ingestion. A later settings change moves no stored trace.
Attach Files to a Trace Input
A trace you log yourself can now carry the documents your agent received. Galtea stores and keeps them like the files of a test case input.- Pass
input_file_pathstotraces.create,traces.updateortraces.create_and_evaluate. Each entry is a local path, which the SDK uploads, or a Galtea storage reference such astrace.input_files[0].uri. inputis optional when you pass files, except intraces.create_and_evaluatewithsession_custom_idand only aproduct_id.- Galtea keeps the files. On S3 storage, the cleanup of temporary uploads no longer removes them, and the stored reference does not expire.
- The test case rules apply. The SDK allows at most 20 files and 20 MB in total. When an input holds a Galtea file, the API also checks the limits, the file types a test case input accepts, and that every file belongs to your organization, and refuses a trace that breaks a rule.
- Evaluators cannot read input files yet. A metric that reads the input is skipped when no turn of the session has input text without a file. To have a turn scored, log the text your pipeline extracted as the input, without the file.
traces.updatereplaces the whole input.- The SDK refuses a URL outside Galtea storage. A direct API call stores such a URL as sent, with no check, unless the same input also holds a Galtea file: then the API refuses it.
- An OpenTelemetry turn whose input breaks a rule is dropped.
Platform Improvements
- The Python SDK needs Python 3.10 or newer. The
langfuse-langchainextra now installs LangChain 1.x, which removesAgentExecutor: build LangChain agents withcreate_agent. See Installation. - Evaluations blocked by the judge’s content filter end Skipped. A blocked evaluation now gets no score and no credits charged, instead of a charged 0. Averages, pass rates and
BAD_SESSION_COUNTalerts leave it out; aNO_SCORE_RATEalert counts it. To score it, use a metric with another evaluator model. An empty judge reply after all retries now ends Failed, and so does a block on some built-in metrics. - Agent spans land on their turn in Direct Inference. Galtea now sends
traceparentandbaggageheaders on every turn call (not on initialization, finalization or test-connection calls), so a span your agent exports before it answers attaches to its turn. Header values now accept{{ trace_id }},{{ galtea_session_id }},{{ session_id }},{{ test_case_id }},{{ dataset_id }}and{{ language_code }}, and{{ auth_token }}underBEARERorAPI_KEYauth. Action needed: a connection that sets its owntraceparentheader is refused, so remove that header the next time you edit the connection. See W3C Trace Context Propagation. - Specifications at a glance. The metrics table and the metric picker have a Specifications column with the specifications each metric is linked to. You see only specifications in products your user groups can read.
- Specifications open where the results are. A specification page opened without a chosen tab now opens on Evaluations when it has evaluations in the current view mode, and on Overview otherwise.
- Ask Val stays out of the way. The Ask Val launcher in the dashboard shrinks to its icon 4 seconds after it appears, so it no longer covers buttons at the bottom right.
Create an Empty Dataset and Add Test Cases by Hand
You can now create a dataset with no test cases and write each test case yourself. Nothing is generated, so no credits are spent.- In the dashboard, pick “Add test cases manually” as the dataset origin. It is available for every dataset type, with or without a linked specification. The new dataset opens on its Test Cases tab, where New Test Case adds each test case.
- Test cases can carry file attachments. On a Behavior test case, a file runs on a single turn only, so a multi-turn simulation of it fails. See Accuracy datasets, Security & Safety datasets and Behavior datasets.
- Through the API, send
empty: trueonPOST /tests.emptymust be a boolean. You cannot combine it with a file (uri,groundTruthUri,dataCatalogUri), a source dataset (sourceTestId) or a generation field such asmaxTestCasesorstrategies. The error names each field it refuses.POST /tests/batchrefusesempty: true, so create each empty dataset on its own. The SDK has noemptyoption yet.
- An evaluation of a dataset with no test cases fails when you start it, with “No test cases found to process”. Add at least one test case first.
- A specification counts a linked empty dataset as done, so once it has a metric it offers Run Evaluation, which then fails the same way.
- The “From an existing Accuracy Dataset” origin lists empty Accuracy datasets. Picking one fails, and no credits are spent.
Give Readers and Human Evaluators Product Access Through User Groups
A Reader or a Human Evaluator sees only the products that their user groups grant. You can now set up that access in the dashboard.Manage a group’s products. Open a user group: a new Products tab lists the products the group grants.- Link products opens a searchable picker of your organization’s products.
- Unlink removes the selected products. It warns you first that the group’s users lose access, unless another of their groups grants it. The products themselves are not deleted.
- The create and edit forms of a group now have an All products, current and future toggle and a product picker. The edit form shows the same warning before a save that removes access.
- While All products is on, the tab shows only a note. The group keeps its linked products, and they apply again when you turn the toggle off.
A Focused View for Readers
Readers and Human Evaluators now get a view focused on the products that their user groups grant.- Land on Analytics. With exactly one product, they land on that product’s Analytics after sign-in, and when they go home from an error page. With several products, they land on the product list, as before.
- Only the metrics their products use. The metrics list shows only the metrics that a granted product evaluates, links to a live specification, or watches with a live monitor. A metric that only other products use cannot be opened. Users in a group with All products still see every metric.
- No inference metrics or judge cost for Readers. Evaluation lists leave out the credits and judge token columns, also in the column menu and the CSV export. Session and version pages hide their inference metrics, and the Trace card hides its metrics tab. Human Evaluators keep them. Span latency and the Analytics version summary stay visible.
Platform Improvements
- Evaluate imported and production sessions against the specifications you choose. When you evaluate existing sessions by specifications instead of by a list of metrics, sessions that have no specification of their own, such as imported traces and production sessions, are now scored too. Each one gets the metrics of the chosen specifications of its own product, and each metric is used once. The “Import Traces” onboarding flow now evaluates the sessions you import, and Evaluate sample on the production insights card scores your production sessions when it evaluates by specifications.
- A session whose own specification is not in your list is still skipped. A session is also skipped when none of the chosen specifications of its product has linked metrics. The response message counts the skipped sessions for each reason.
- When nothing was evaluated, the production insights card now shows that message as an error.
- When you run a Run again, it scores these sessions with the same metrics as the original Run. Most Runs created before this release with chosen specifications skip these sessions again when you run them again.
- Run details and the Run again dialog now count every metric of a Run scoped per specification.
- Link a version to its model from its facts. A version fact named
modelormodel_name(case,_and-are ignored), such asmodel: azure/gpt-4o, now links the version to that model, so sessions on the version report token cost from the model’s prices when the inference result sends no cost of its own. This works when you create a version without amodel_id, when get-or-create or OpenTelemetry traces create one, and when you update the facts of a version that has no model.- The value must match a model name exactly, provider prefix included. Only case is ignored. Your organization’s models and the global models count, and your own model wins over a global one with the same name.
- A fact never replaces a model the version already has. Existing versions are not linked until you update their facts. When two fact values each match a different model, no model is linked, with no error.
PATCH /versions/:idnow refusesproductId,number,parentVersionIdand unknown fields, andversions.updatein the SDK no longer takesparent_version_id. A revision’s parent is set when you create it, as before.
- Read long names in tables and form fields. When a table cuts a linked name short, such as a metric, test or version name in the evaluations tables, hover the name or move keyboard focus to it to see the full name in a tooltip. Dialog picker fields and dropdown fields now keep a long value on one line, end it with an ellipsis, and show the full value when you hover it. These field tooltips open on hover only.
- The dataset form no longer shows a false “At least one type is required” error when you create a dataset from a specification or change its linked specification.
Limit Readers and Human Evaluators to Chosen Products
A Reader or Human Evaluator now sees only the products their user groups grant. Owners, Admins and Members are unchanged.- Existing readers and human evaluators keep every product. The upgrade puts them in an “all products” group.
- A Reader or Human Evaluator you invite from now on sees no product until you add them to a group. To give full access, add them to the “all products” group.
- Choose a group’s products with the SDK (
link_products), the CLI (link-products) or the API. The dashboard has no screen for this yet. - A human-evaluation metric with no linked group no longer reaches Human Evaluators.
- Analytics: Coverage, Scores comparison and Inference are hidden.
- Product sidebar: Versions, Datasets, Tutorial, Product, Connections and Monitors are hidden.
- Readers only: Human Evaluations and some evaluation page cards are hidden.
Get Emailed When a Monitor Scores Badly
A monitor can now raise an alert when too many sessions score badly, and email people about it.- Add alert rules in the monitor’s Alerts tab, or through the API and the CLI (
galtea monitors create-alert-rule). The SDK has no alert methods. - A rule holds a metric, a window of 15 minutes, 1 hour, 6 hours or 24 hours, a bad metric score, a threshold and 1 to 20 recipient emails. The dashboard prefills 0.5 and 10. The API and CLI require both values. The recipients need no Galtea account.
- Every 2 minutes Galtea counts the bad sessions. A session is bad when its score is at or below the bad metric score. It counts once, by its latest score on that metric inside the window. Only evaluations made by that monitor count.
- The alert opens when the count reaches the threshold, and recovers when it falls to the threshold minus its square root, rounded down. With a threshold of 10, that is 6. A window with no scored sessions leaves the alert as it is.
- Recipients get an email when an alert opens and another when it recovers, with links to up to 25 sessions. A deployment with no email configured sends none.
- Deleting or pausing a rule closes its alert and sends no email. So does changing its metric, bad score or window, which also checks the rule again at once. Editing only the threshold keeps the alert open.
- Follow alerts in the dashboard. A rule shows OK, Open or Paused. A monitor page offers alert history with filters, and each alert has a page listing the evaluations behind it.
- Pause and resume a monitor with
monitors.pause()andmonitors.resume(),galtea monitors pause, or the dashboard. A paused monitor scores no new sessions. Its open alerts stay open once the window holds no scored sessions. You can pause a single rule too. - A monitor can no longer take a self-hosted metric. Creating a monitor with one, or adding one, is refused. Existing monitors keep theirs, and those metrics are skipped.
- Removing a metric from a monitor deletes the alert rules on that metric.
Review an Optimized Judge Prompt Before It Replaces Your Metric
Metric optimization no longer replaces your metric when it finishes. It now produces a candidate revision for you to review. See Optimization review.- The candidate waits as “Ready for review” with the changes, the optimized prompt and the original prompt in tabs. Apply optimization makes it the active revision. It keeps the original’s current name, description and tags, and copies its specification and user-group links. Keep original marks it “Not applied” and changes nothing.
- A declined candidate cannot be applied later. A metric holds one waiting candidate at a time, and applying or declining needs permission to update the metric.
- An unchanged prompt ends as “No improvement”.
- While an optimization runs, monitors score only the original metric.
- Your annotations are split before the run. With at least 30 annotated evaluations, about one third of them (at least 10) are held out, chosen by session. The optimizer never searches on them. The confirmation shows the split. The review shows how well each prompt agrees with your annotations on both sets.
- Below 30 annotated evaluations there is no held-out set, unless an earlier attempt on the same metric kept some. The review says when it has no validation result.
- Only an organization-owned AI Evaluation metric that is the active revision can be optimized. The SDK has no optimize method. Use the dashboard, the CLI or the API.
Know Before a Judge Model Retires
When the gateway reports the date a provider retires a judge model, Galtea now emails your organization about it.- Within 30 days of the date, Owners, Admins and the creators of the affected metrics receive “Action needed: a judge model used by your metrics is retiring”. It lists up to 50 metric names. Your organization gets one warning for each model and date.
- After the date, once the model has been gone for 6 hours, a second email says your metrics stopped scoring. Their evaluations are skipped while the model is unavailable.
- A metric on a legacy model keeps scoring until the model is disabled.
- The SDK reference lists new judge models, among them
claude-opus-5-5,claude-sonnet-5-5,GPT-6-SolandGPT-5.6-Sol. The models you can pick depend on your deployment.
For Delivered and Self-Hosted Deployments
These changes are off or optional unless your deployment turns them on.- Transfer files through the API. With
STORAGE_PROXY_TRANSFER_ENABLED=true, download links point to your API (valid for up to 1 hour) and uploads stream through it (15 minutes, 200 MiB per file by default). The flag also needsSTORAGE_TRANSFER_TOKEN_SECRETandAPI_URL. The Python SDK follows these links. It is off by default. - Check that every service can reach its dependencies. Each Python AI service now reports on
GET /self-check. The API gathers them in oneGET /self-checkandgaltea self-check. An Admin, or a member of an Enterprise organization, can read it. Only an Admin sees the resolved values and the failure detail. An optional Helm Job (selfCheckJob, off by default) runs the check after a delivery, retries for a few minutes and prints a table to its log. The check makes billed model calls. - A half-set Azure storage configuration now raises an error when a service first uses storage, and the self-check shows it. Before, the service wrote to local disk.
- Sign in with Enterprise SSO. When your deployment runs Keycloak sign-in, entering a team slug leads to your organization’s identity provider. Every logout lands on the sign-in screen. New SAML providers validate response signatures, and a save is refused when no signing certificate is available. Existing SAML providers keep their old setting, so ask your Galtea operator to turn Validate Signatures on for them in Keycloak.
Platform Improvements
- Deactivate a user without deleting them. An Owner or Admin can deactivate and reactivate a user from the users table or the user page. The user cannot sign in and their API keys stop working. Their role and data stay. Galtea refuses it when it would leave an organization with members and no active Owner or Admin.
- See pass counts on analytics score bars. Hover a bar for the Passed and Not passed counts. Passed means a reported score of 0.5 or more. Merged bars now weight by evaluation count. Red now ends at 50% (it was 60%) and green starts at 85%.
- Secret header values on endpoint connections are encrypted and shown as
••••••••. Only a short list of common headers, such asContent-TypeandAccept, and values made of placeholders such asBearer {{ bearer_token }}stay readable. A value saved earlier is masked, but it stays unencrypted until you save the headers again. - Retries claim each evaluation once. An evaluation left pending for 24 hours now fails and can be retried. A self-hosted metric is never retried.
- Newly generated accuracy metrics read the test case’s expected output when their judge prompt asks for the correct answer. Metrics you already have are unchanged.
- Scenarios for a specification about language now exercise it. A rule such as “never answer in Portuguese” now meets users who write in Portuguese, even on a Spanish test. Other specifications keep the test’s language.
- A run where no session completed says how many test cases failed, for example “3 of 3 test cases failed; no session completed”.
- Last Successful on the analytics page now counts evaluations of metrics excluded from analytics and skips deleted sessions.
- Faster pages on large products: analytics scores and the metrics list for a product.
Every Launch Now Leaves a Run
A run is the record of one launch. Galtea creates one every time you launch evaluations, from the dashboard, the CLI, the SDK or the REST API. A queue job disappears; a run stays, so you can still read last week’s launch.- Find runs under Runs in the product sidebar, behind More unless Show all tabs is on. A run’s page has four tabs: Overview, Launches, Sessions and Evaluations. Outside the Production view, session and evaluation tables gained a Run column and a Runs filter.
- A run whose test cases produced no session ends as
FAILED, with its reason. Sessions and no evaluations is still a success: a generation with no metric endsCOMPLETEDand carries no score. - Run again repeats a run and answers with a new one, leaving the original alone. The button is hidden, not disabled, when a run cannot repeat: two launches of one kind, no evaluation naming a session, or over 1000 sessions. For a run your own code drove, it re-scores the existing sessions rather than producing new ones. Only a re-launch accepts a different
versionId. - Cancel a queued launch without stopping the rest of the run. A launch that finishes inside the request that started it cannot be cancelled. Whatever it already created is kept.
- Each run carries a number, shown as
#42, starting at 1 in each product. A deleted run keeps its number, so gaps are normal. - Group several launches in one run. Open a run, pass its id to each launch, and close it when they are done.
fromMetricsandfromTestCasestake norunIdand open their own runs, and a run you opened yourself records no version. - Python SDK:
galtea.runsoffersstart(),list(),get(),get_by_custom_id(),close()anddelete().start()returns a context manager, so sessions and evaluations created inside awithblock join the run. The write methods also takerun_iddirectly, andevaluations.list()filters by it. - CLI:
galtea runs list,get,create,close,relaunch,cancel-launch,get-metric-summaryanddelete.
runId for and the SDK leaves out of a with block; a monitor evaluation; a retry, which keeps the run it started in; work you start on another thread inside the block; and a session get_or_create finds rather than creates. A with block also cannot close a run holding a queued launch. It raises instead and asks you to call runs.close() once the launch ends. Deleting a run leaves its sessions in place.Start a Test Case Mid-Conversation
ASCENARIOS test case can now start from a conversation that already happened. The simulator continues from that point instead of opening the dialogue itself. Use it to test what your agent does on turn five, not only on turn one.- Two ways to give the conversation.
conversationHistorywrites the turns out.seedSessionIdcopies them from an existing session, with its spans, so it carries the retrieval context and tool calls the written form cannot. Sending both is refused. - Live Turns to Run counts only the new turns. A 4-turn conversation with Live Turns to Run set to 1 ends with 5 turns. The field is
max_iterationsin the SDK and in a CSV file. - Set
userPersona,goalandscenario, or the run fails on its first live turn. Nothing checks them when you save the test case. Aninputcarrying files fails the same way. - Not on voice yet. A phone or WebRTC run ignores the conversation and starts from turn one.
simulate(agent_goes_first=True)is refused on a seeded test case, after your agent’s opening turn is recorded. The session keeps that turn and produces no score.
Configure the Judge That Scores Your Metric
An AI Evaluation metric can now name the generation settings its judge runs with, and prompt-based evaluations report what the judge spent.- Prompt judges no longer run at
temperature: 0by default. Galtea used to apply it implicitly, and that is removed. Existing metrics were backfilled with an explicit0, so they are unchanged. A new metric created without a temperature now follows the provider’s own default and is less deterministic. Settemperatureyourself to keep the old behavior. - Four settings are accepted:
temperature,top_p,max_output_tokensandreasoning_effort. Set them when you create the metric, in the Model settings section under the evaluator model picker or throughjudge_generation_settingsin the SDK. Any other key is refused. - Only
FULL_PROMPTandPARTIAL_PROMPTmetrics use them. A DeepEval metric fixes its own temperature and ignores the settings, so many built-in Galtea metrics cannot be tuned this way and report no tokens. - The form offers only what your model accepts, and a model can accept none of the four. A setting your model does not accept is dropped when the judge runs and the evaluation still succeeds, so nothing tells you it had no effect. A value your provider rejects is different: that evaluation ends
SKIPPEDcarrying the provider’s own message. List what a model accepts withgaltea evaluator-models list. reasoning_effortis free text. The levels differ per model:minimal,noneandmaxexist besidelow,mediumandhigh.Max output tokensis capped by your model’s ceiling in the dashboard only. The API accepts any whole number of 1 or more, so an SDK or CLI caller can store a value the model will not honour.- Settings cannot be changed later. Create a revision of the metric to use different ones. The dashboard carries them into the revision, keeping those the new model also accepts. Through the SDK, a revision that omits them gets none.
- Prompt-based evaluations now record the judge’s token usage. The evaluation page shows it in a card under the Verdict, and the evaluation list offers Judge Tokens, Input Tokens, Output Tokens, Reasoning Tokens and Token Distribution columns. The Python SDK returns the counts on
Evaluation. Reasoning tokens sit inside the output count, so never add all three together.
Choose What Counts in Your Analytics
You can now exclude a metric revision from analytics, and read the Scores card in the order that answers your question.- Exclude a revision from analytics. Its results leave the score, coverage and insight numbers, and keep their own score and reason. Evaluation totals, latency, cost and token figures still count them. Before, analytics pooled every revision, so a stricter replacement diluted the current score.
- The exclusion is organization-wide, not per product. One action changes every product holding results from that revision.
- Excluded results are then hidden by default, in the evaluation list, in the metric list and in the SDK. Turn on Show excluded from analytics, or pass
include_excluded_from_analytics=True. The metric list’s Analytics column is hidden until you reveal it. - A legacy revision still counts. Creating a revision marks the previous one
Legacy, and only an exclusion stops it counting. After you save a revision, the dashboard says how many results and products the earlier one affects, and keeps them by default. Excluding is a separate request, so it can fail on its own and leave the revision saved and still counted. - The decision is reversible from the metric’s actions menu, and it changes no evaluation. You cannot exclude a Galtea platform metric, and a read-only account cannot exclude anything.
- A product’s Analytics page says how many results sit outside the reported numbers, and links to them. There is no way to read analytics with them added back.
- Python SDK:
metrics.set_analytics_exclusion()andmetrics.get_analytics_exclusion_impact(). - The Analytics Scores card draws bars instead of rings, lowest score first. A picker beside the heading switches to
Highest firstorA to Z, and your choice rides the URL. It appears once the card holds two bars.
Compare the Expected Output Against the Actual One
The evaluation details page now shows the expected output beside the actual one, and the expected tools beside the tools that were called.- A JSON payload reads as a per-field table, one row per path, marked
Match,Similar,Differs,Type mismatch,Missing,UnexpectedorLength differs. Matching rows hide behind a toggle that names how many it holds back. A large payload stops at 500 rows. - Fields, Pretty and Raw. A payload with fields offers all three. A short label or plain prose offers Pretty and Raw only, and the tools card carries no view controls at all.
- Each payload carries its own view controls, so you can read one output as Raw while the page stays Pretty. That override is not saved: a reload returns every payload to the stored setting.
- Raw is the only view of the stored characters. Fields and Pretty read the scored value, which unwraps
assistant_messageand content parts.
PROGRAMMATIC or HUMAN_DIRECT. Everywhere else, including a conversational product, the Diff switch in the card header reaches it in one click.Read Every Attached File in Place
A file attached to a turn now carries Full size, Expand and Download wherever the turn itself is shown, so checking an extracted field against its source stays on one screen.- Expand opens the file under its row, and Full size opens it in a lightbox. Download is still there for anything stored.
- A PDF renders inline, next to the output and the verdict, without leaving for a download or another tab.
- An image gets the same controls, for the formats a browser renders:
.png,.jpg,.jpeg,.gif,.webpand.bmp. - A single file opens on its own when the whole conversation is one turn and that file is one the platform can preview.
.docx, a .csv or a TIFF, keeps the download alone. And a file listed in a table cell carries no controls, so open the session or the evaluation to reach it. Judges never read an attached file, whatever it shows.Every Version Now Has a Number
Each version carries a number, unique inside its product and starting at 1.v7 is the label Galtea derives from it.- A version with no name shows its canonical label. The dashboard shows
v7, orv7: support releasewhen the version carries a custom name. Galtea no longer invents a name for the first version it creates for you. vfollowed by digits is a reserved name shape.v28,v01andv0are all refused, unless the name is that version’s own label. A name such asv1.0stays valid.- A name is no longer unique inside a product. A version named
v5before numbering and the product’s fifth version can now stand side by side, soversions.get_by_namemay raiseAmbiguousVersionNameException. - Numbers are not contiguous. Deleting a version leaves a permanent gap.
- Python SDK:
Version.numberis on the response model, andversions.create(product_id=...)now works with no name.numberis optional, and the label falls back to the bare name when it is absent.
Reach Every Product Section From the Sidebar
Datasets, Sessions, Evaluations, Runs and Tutorial appeared only when Show all tabs was on, so most readers never saw them listed.A More entry now sits after the other work entries and opens those five in a menu. Each item is a real link, so you can middle-click it into a new tab. In the Production view, Datasets is absent and More holds four. Turn Show all tabs on and the five render inline, with the More row gone. The setting lives in your browser, so it does not follow you to another device. Two sections also left the setting in this release: Connections, now development-only, and Monitors.OpenTelemetry: Mark Test Traffic, and Name the Real Turn Input
Three changes for products instrumented with plain OpenTelemetry and no Galtea SDK.- Mark a trace as test traffic with the
galtea.test_case.idattribute, on the resource or on a span, where a span attribute wins. The session it opens is then non-production and linked to that test case, so a monitor never scores it. Before, every ingested session was production, so development traffic reached your production monitors. Three things to know: a value that is not a non-empty string is ignored in silence and you get a production session, exactly as if you sent nothing; a marked session staysPENDINGuntil you widen the product’s auto-close scope or close it yourself; and only OTLP ingestion reads the attribute, so file import andPOST /traces/batchignore it. - Name where the turn input lives with
otelInputSourceRule, per product or per version, when your spans carry no recognised input attribute. Galtea used to fall back to the LLM span’s messages array, which is your system prompt plus a copy of the user message, so the judge scored the reply against a prompt your user never wrote.otelOutputSourceRuledoes the same for the output. Two limits: when the rule finds nothing, the turn input is empty, because the old fallbacks no longer apply; and a version cannot opt out of a product rule. - A batch missing
galtea.version.idnow logs one warn line per sender, so you can find the sender environment to fix. The export response and the dropped-span counter name the reason but not the sender.
Platform Improvements
- Invite someone who already has an account: inviting a registered address now moves that person across, when they are alone in their organization, that organization has a free plan or none, the invite names no user group, and neither side uses an enterprise IdP. They get an email naming the organization and the role, and confirm from a page within 24 hours. The move is final: their old organization stays behind unreachable, with its products, evaluations and credits, and their API keys start writing into the new one. Outside those conditions the invite fails as before.
- Choose the language of generated test cases: the Setup for evaluation dialog now asks for a language in the same step as the number of test cases, from 13 options. It defaults to Automatic, which is the previous behavior. The field is ignored while no file is attached.
- Test an endpoint connection from its actions menu: a Test Connection entry now sits in a saved HTTP endpoint connection’s actions menu, on both the list and the details page, and opens a dialog with the status code, the latency, the raw response and the extracted output fields. A blank Authentication Token keeps the stored credential, as long as you do not change the authentication type. A blank username still overwrites.
- One SDK exception for a missing entity: a read of an entity the platform does not hold now raises
EntityNotFoundException, importable asfrom galtea import EntityNotFoundException, whether you read by id or by name. Nothing forces you to migrate: a 404 raises a subclass of both that andrequests.HTTPError, so existingexcept requests.HTTPErrorcode keeps working. A lookup by name raises the plain exception, which is not anHTTPError, as before. - Clicking inside an open span no longer closes it: only the span’s header row toggles it now, by mouse.
- Keeping your work leads the setup dialog: closing the specification setup dialog while metrics are still generating now puts the primary style on the button that keeps them.
- Disconnecting a linked account cannot hang: every Disconnect button is disabled from the moment the confirmation dialog opens.
- Generated scenarios stay grounded: a scenario generated from a conditional specification is now built on a catalog record the rule governs, and Galtea picks one itself when the planning model names none. When no record matches the rule, the scenario falls back to the whole catalog and the guarantee does not apply.
Review Evaluations and Correct a Wrong Judge Score
Finding where your judge disagrees with your reviewers, and fixing it, now happens in one place and counts everywhere.- A metric page is now a hub with three tabs: Details, Revision History, which draws every revision of the metric and how they descend from each other, and Evaluations, which lists every evaluation the metric produced across every product that used it.
- See the disagreement directly. The Evaluations tab shows the judge’s score, the reviewer’s score, and the Score gap between them in points. The evaluation page gained a Verdict card that puts both side by side with their reasons, and says when they disagree on pass or fail.
- Your correction now counts everywhere: analytics averages, the score on a product card, scores over time, insight cards, coverage counters, and monitor series. Before, every total kept using the judge’s original score.
- It applies to reviews you already made, so some numbers change as soon as this release is live, without you doing anything.
- More evaluations can carry a review: one that failed, was skipped, or was cancelled now counts once a reviewer scores it. Before, only a successful one did.
Send Attached Files and Audio to Your Agent
A test case that attaches a document, an image, or audio can now run end to end without leaving Galtea. Creating a product that reads PDFs, scanned receipts, spreadsheets, photographs, or screenshots is supported too, and the product form now says so instead of warning you off.- Platform runs: your endpoint connection reads attachments through a new
input_filescollection in its template, with four properties per file:url,base64,filename, andmime_type. See the template syntax. - Python SDK: an agent called through
evaluations.run,simulator.simulate, ortraces.generatereceives each attachment as anInputFileobject, instead of a raw dictionary buried in the message metadata. Read one into memory with the newgaltea.storage.read(), or see reading attached files. - Dashboard: an input holding only files used to render an empty cell. It now names the files, and a download saves under the real filename instead of the storage key.
- Video is not supported yet, and a product that processes video recordings or live video is now named as such rather than left unclear.
{{ input.content[0].uri }} no longer returns an address, so use {{ input_files[0].url }}. The document inputs tutorial covers the full path.Delete Your Own Account
The Settings page now has a Delete Account card. Deletion is immediate and cannot be undone: you are signed out at once and every API key you issued stops working. Your organization and everything in it is kept, and your email address becomes free for another organization to invite. See roles and permissions.Platform Improvements
- Read any payload as Pretty, JSON, or Raw: every product payload now opens through one viewer with a
Pretty,JSON, andRawswitch, so you can tell whether a field held the string"950"or the number950. Copy a value from any view without switching first, one nested row at a time if you want. - Input and Output panels: a session whose product is not conversational drops the User and Assistant bubbles, and renders an Input panel above an Output panel per turn.
- Set up a specification for evaluation: a specification still missing a dataset or a metric now offers Setup for evaluation in place of a disabled Run Evaluation button, and unlocks on its own once a generating dataset finishes. An Inability specification explains that inabilities are never evaluated, instead of asking for a dataset and a metric it can never hold.
- Filter runs by who launched them: the sessions and evaluations tables gain a Launched By filter, listing the people in your organization with you first. The Python SDK takes the same filter through
user_idsonsessions.list()andevaluations.list(). - Monitor columns that never appeared: the monitor hub now shows Settled At Turn and Monitor Sampling. Both columns existed but had stayed hidden.
- Readable table headers: a long column header now shortens with an ellipsis and reveals its full text when you hover it, instead of running past the edge of the column. The sort arrow is no longer clipped either.
- Scenario cases your agent can actually fail: a scenario dataset generated from a specification now contains only direct cases. Before, roughly half the generated cases reported a violation the agent had no chance to commit. Expect these simulated conversations to run for more turns than before.
- European language varieties: picking a language such as Spanish, French, or Portuguese now stores the European variety, and a simulated conversation with a bare language code uses it too. A bare
esused to produce Latin American replies. English, Japanese, Chinese, and Korean stay as they are, and an explicit region you set always wins. See test case.
Attach Files to Test Cases
A test case input can now include an uploaded file, such as a PDF or an image, instead of only text. This lets you evaluate an AI product that reads documents, for example a contract-review agent, without paraphrasing the document yourself first.- Python SDK: pass local file paths to
input_file_pathswhen you create a test case, or add aninput_file_pathscolumn to a dataset CSV. Each file can be up to 20 MB, in common document, image, or text formats. - Dashboard: the test case create and update forms now have an Attach file button. An attached image shows a thumbnail you can open at full size.
- Getting files back out:
galtea.storageis now public. Callupload()to send a file to Galtea’s storage, anddownload()to save one back to disk, including a file read straight fromtest_case.input_files. - A dataset whose test cases already carry a file input can no longer be augmented: the AI-assisted augmentation flow only reads text, so it’s refused instead of silently generating rows about the wrong document.
More Natural Simulated Conversations
Simulated users in the conversation simulator now write more like real people, with more natural pacing and phrasing instead of stiff, scripted-sounding turns. On Galtea’s internal quality benchmark, the same rubric and model scored 11% on this before the change and 69% after.More Reliable Trace Assembly
Traces sent through OTLP (the OpenTelemetry protocol most tracing SDKs export with) now assemble correctly when a single conversation turn is split across several export batches, which is the normal case for most SDKs:- A span whose parent arrived in an earlier batch now keeps its place in the trace tree, instead of showing up as a separate root next to the turn it belongs to.
- A turn whose conversation link rides on only one of its batches is now assembled whole, instead of every batch that arrived without that link being rejected.
Platform Improvements
- Failed conversation turns retry: A conversation turn that fails to generate is now retried, instead of being left empty while the session still completes and gets evaluated normally.
- Dataset generation no longer stalls: A malformed response from the underlying LLM used to double the retry wait with no ceiling, stalling generation for over an hour. The wait is now capped at 60 seconds between attempts.
- Breadcrumbs on every form: Every create and update form now shows a breadcrumb back to the record it belongs to, so you can leave the form without using the browser’s back button.
- Clearer evaluation errors: An evaluation that can’t be scored because the actual output has the wrong shape now shows the specific reason, instead of a generic error.
- Generate Metrics button restored: The specifications list in production view shows a Generate Metrics button again, so you don’t need the row’s menu to reach it.
Organization Pages Open Faster
Opening an organization used to take around 4 seconds while eight analytics queries competed for database capacity. That work is now done far more efficiently, so the page and its product cards render noticeably faster.- The heaviest analytics queries were rewritten to stop re-reading the same rows multiple times per request.
- Two new database indexes let the database answer coverage and score queries directly, instead of walking back to the full table for every row.
- API responses are now compressed. One organization page used to transfer 2.4 MB of JSON in the clear.
- Every product card’s last-evaluation date now loads in a single request for the whole page, instead of one request per card.
Analytics Improvements
- Group by opens where the data is: Analytics used to default to grouping by Specification, which could leave the page showing an empty Scores panel even when a product had evaluated metrics. It now opens on whichever dimension, Specification or Metric, actually has scored data.
- Accurate test case counts: A product’s Total and Not Evaluated counts no longer include platform-wide test cases the product never actually ran. This previously inflated Not Evaluated for the vast majority of products platform-wide.
- “Performance by Dataset Type” removed: This section, which grouped scores by how a dataset was generated, is gone. The merged Analytics view with its filters already covers the same ground.
Trace and Span Import Improvements
- Unrecognized spans no longer vanish: If your agent labels a trace span with a type Galtea doesn’t recognize, that span (and its whole turn) used to be silently dropped. It’s now mapped to a sensible type instead, and a span dropped for a data-quality reason shows up as a warning on the turn.
- Original span names are kept: When Galtea replaces a span’s name with one derived from its attributes, for example a GenAI
execute_toolspan whose name is taken fromgen_ai.tool.name, the name your producer sent is now kept in the span’s metadata. - Broader import support: Traces imported through MLflow, the OpenTelemetry GenAI semantic conventions, or Mistral now get a real span type instead of falling back to a generic one. MLflow and the GenAI conventions are read from the span kind the producer already declares; a Mistral LLM call is recognized from its own attributes. This improves
tools_usedandretrieval_contextdetection for those integrations. See Send OpenTelemetry Traces to Galtea for the span conventions Galtea reads.
API Key Limits and Emails
- API key limit, visible everywhere: The dashboard now shows your subscription’s API key limit on every screen that already shows a plan limit, and clarifies that it’s counted per user, not per organization.
- Accurate API key emails: The email you get when an API key is created, regenerated, or revoked now states only what actually changed for you, instead of a generic warning that all your previous keys stopped working.
Platform Improvements
- Organization link goes to Products: The sidebar’s Organization entry now opens the organization hub on the Products tab, matching where the sidebar logo and post-login landing already take you, instead of opening on Users.
Version 5.0.0: A Redesigned Dashboard
This release reworks how you navigate and use the dashboard: the sidebar, specification setup, evaluation flow, and onboarding all change, and three core entities get clearer names. Here is what moved and why.New Names: Dataset, Trace, and Span
Three entities you use every day change their name, so each level of the platform now has one clear word:- Test is now Dataset: the collection of test cases you evaluate a version against.
- Trace is now Span: one step your product took while answering, like a tool call or a retrieval.
- Inference Result is now Trace: one input and the output your product returned.
Test Case keeps its name; only the collection around it becomes a dataset.The API, CLI, SDK, and dashboard all use the new names. If you call the platform through the SDK or CLI, read the migration guide before you upgrade. In the SDK, most old names keep working with a warning, and you can pin galtea<5 to delay the change. Two things need your attention: galtea.traces now returns traces instead of spans, so it changes meaning without an error, and the CLI has no aliases. Pinning does not help the CLI either, because it reads its command names from the API and refreshes them on its own. Update your CLI calls before the release.Redesigned Sidebar Navigation
The sidebar now carries most of what used to live in page tabs.- A product selector and environment switch sit at the top, so you pick your product and switch between Development and Production without leaving the sidebar.
- A product’s sections list directly in the sidebar, replacing the product hub’s top tab strip. Analytics, Specifications, and Versions are always there; Datasets, Sessions, Evaluations, and Tutorial appear once you turn on Show all tabs in your settings. The configuration sections, Product, Connections, and Monitors, sit inside a collapsible Configuration folder, marked with a caret so you know it expands rather than opens a page.
- The organization hub is now the landing page after login, with Products as its first tab and a product count next to Users and Groups. The product matching your sidebar selection is outlined in that list, so you can tell them apart at a glance.
- Organization-level entries (organization settings, Human Evaluations, Metric library) now group at the sidebar’s bottom, under your organization’s own name.
- Your account menu moved into a floating card at the sidebar’s bottom, instead of a page of its own.
Starting an Evaluation
“Run Evaluation” used to appear in twelve places, and not all of them meant the same thing. It now starts from the units you actually evaluate: a version, a specification, a session, or a single trace, each from its own page or its Evaluations tab. The Analytics page, the dataset and test case detail pages, and the version’s “Evaluate All” button no longer offer it, since the evaluation dialog already covers what those did in one extra click. Launching a run against a product’s latest configured version automatically is no longer available in the dashboard; the SDK’sevaluations.run(product_id=...) still supports it. A capability can now be an evaluation target from the dashboard too, matching what the API already allowed.Launching an evaluation now also takes you straight to the tab where it shows up, instead of leaving you where you clicked.Streamlined Specification Setup
Getting a specification ready for its first evaluation used to take several disconnected screens. Now:- One “Setup for evaluation” dialog asks how many test cases to generate, plus a knowledge base file for an accuracy specification. It creates the dataset and proposes up to three metrics, which you then review and save on the metric page.
- A new specification offers this dialog right after you save it, instead of leaving you on an empty page with no dataset and no metric. Inabilities are never evaluated, so they are not offered it.
- The Specifications table shows real dataset status: a count,
Generating,Failed, orMissing dataset. Run Evaluation stays blocked until a dataset is ready, so a generation that failed needs a retry from the same dialog. - Linking and unlinking a dataset or metric works the same way everywhere: the specification field on the dataset and metric forms now matches the picker already on the specification hub.
- A warning flags datasets linked to no specification, so they no longer sit invisible.
- Capabilities can generate datasets now, matching policies, and accuracy specifications can generate datasets too, from a chosen test case count.
- Specifications separate into two scopes: capabilities and policies under a product’s Specifications tab, inabilities under Settings.
- A specification’s Sessions and Evaluations tabs now show a total, counting a session or evaluation reached through the specification’s datasets or its metrics, so production traffic a monitor scores now shows up instead of an empty tab.
Smarter Onboarding
- Onboarding classifies your product’s kind before generating specifications and names the generated specifications after your product instead of its category, so a document-processing product and a conversational one no longer get the same generic catalog.
- A document-processing product can ask for a Knowledge Base file during onboarding when you pick an accuracy specification, instead of that specification silently staying hidden.
- Onboarding confirms what it generated, listing every specification by name before moving to the tutorial.
- The tutorial’s review step reorganizes into three ideas: What we will check, How we will judge it, and Example challenges, each with its own loading and retry state.
Analytics: One View, Filtered
Analytics now shows every specification in a single view instead of separate Overview, Accuracy, Security & Safety, and Behavior tabs. A Dataset type filter (Development only) replaces the tabs, and the metric/specification switches that used to differ per tab are now one toggle for the whole page. The filter persists in the URL, so a filtered view is shareable and survives a refresh.AI-Assisted Metric Revision, Inline
Revising a metric with AI now happens inside the form instead of a separate dialog. Describe the change in a panel beside the fields, and the description, evaluation parameters, and judge prompt rewrite in place. Undo restores a single field to its last saved value, so one bad field no longer costs the other two.Product Settings, One Card Per Setting
The product Settings page is now one column of full-width cards, one setting per card, matching the user Settings page: Metadata, Session Auto-close, Inabilities, Product Properties, and, when configured, the OTel Output Source Rule. Monitors and Connections moved to their own sidebar entries.A Late Message Reopens a Closed Session
A message that arrives after a production session closes now reopens it instead of being dropped. Any monitor evaluations already on the session are marked outdated, so the monitor scores it again once it closes for good. A session that keeps reopening is re-scored up to five times; after that it still accepts the late message, but its existing scores stay as they are.This is the new default for every product, set under Session Auto-close on the product’s Settings page. You can still choose to ignore a late message or refuse it with an error. Development sessions never reopen, because monitors only score production traffic.Platform Improvements
- A row’s actions menu stays usable when a related read fails: opening the three-dot menu on a session’s Evaluations tab no longer opens empty.
- A deep link now opens the section it names: “View evaluations” from Analytics used to leave you on Analytics; it opens Evaluations now.
- A session can be evaluated even when its version has no conversation target.
- Deleting an item that is still in use now returns a clear error instead of a server error.
Platform Improvements
We are working on a new version of the platform, and it will come out soon. In the meantime, this release collects a set of smaller improvements.- More intuitive table row behavior: Clicking anywhere in a table row now opens that row. Before, clicking a shortened cell opened a dialog with the full text instead, which got in the way of simply moving around. The full text is still one click away, from a small button in the cell’s top-right corner: shown on hover or keyboard focus, always shown on touch, and present only when the text is really cut off.
-
More evaluator models available in the platform: An evaluator model is the LLM that scores a metric. You can pick from the GPT-5 family (
GPT-5.2,GPT-5.1,GPT-5,GPT-5-mini) and five EU data zone models (eu.claude-sonnet-4-5,eu.gpt-4.1,eu.gpt-4.1-mini,eu.gpt-4o,eu.gpt-4o-mini) that run on European infrastructure, for when your data must stay in the EU. See the full list.Claude-Sonnet-3.7andGemini-2.0-flashwere disabled in May and can no longer be used. Model names are case-insensitive. -
API keys verify about 400 ms faster: Every request authenticated with a
gsk_API key spent around 400 ms checking that key before doing any work. That check is now near-instant, so the Python SDK, the CLI, and trace ingestion all answer faster. Your existing keys keep working and their format is unchanged, so there is nothing for you to update. -
Retiring an evaluator model now retires the metrics built on it: When Galtea stops serving an evaluator model, every metric that uses it is reported as legacy or disabled on its own, instead of each one being marked by hand. If the model comes back, so do the metrics. In the SDK, a
Metricnow carriesdisabled_atnext tolegacy_at. Adisabled_atdate in the future is how an upcoming retirement is announced, so compare it against the current time before treating a metric as disabled. - An evaluation that names an unknown evaluator model now fails instead of scoring: Asking for an evaluator model the platform does not serve returns an error. Before, the evaluation fell back to a default model and returned a normal-looking score, so a result could come from a judge you did not choose.
- A monitor’s Sessions and Evaluations tabs show their rows again: Both tabs read zero over an empty table. Neither was asking for production traffic, which is the only kind a Monitor scores. Both now do, and each tab’s count is built from the same filters as the table under it, so a count can no longer disagree with the rows it sits above.
- The Platform version card covers every service: The Platform version card at the end of the Settings page listed only the API. It now has one row for the dashboard, one for the API, and one for each of the ten backend services, each showing a readable name plus the exact identifier used in logs and deployments. A service that has not reported its version yet says so, with a tooltip explaining what to check, rather than showing a blank version that would make it look healthy. Visible to enterprise members and admins.
Monitor Only the Production Traffic That Matters
A Monitor scores a sampled share of your production sessions. The new Content filter section of the create and edit form tells it which sessions are worth scoring at all: refund and cancellation traffic for a support agent, everything except your internal test phrases for a medical assistant.- Include and exclude terms. Include terms keeps only the sessions containing one of your terms. Exclude terms drops the sessions containing one of them, even if they matched an include term. Both are optional.
- Write terms the simple way. Separate terms with a comma, for example
refund, cancel, chargeback. A session matches when any term appears anywhere in a turn, and matching is case-insensitive. - Choose which side of the turn to search. Each group has its own Match in setting: the user’s
Input, your product’sOutput, orEither. - See the effect before you save. A Preview panel tests the filter against your recent sessions and spends no credits: how many contain your terms, roughly how many would be evaluated at your current sampling percentage, and which ones match, so you can click into one and check.
POST /monitors and PATCH /monitors/{id}. The Python SDK does not set them yet, so a Monitor created with monitors.create has no filter until you add one in the dashboard.Label Your Comments and Find the Same Problem Across Sessions
Comments on an inference result can now carry labels, so error analysis produces something you can search instead of a pile of notes.- Label as you write. The Comments tab of a turn has a Labels for this comment field. Labels are free text you invent as you go, for example
hallucinationorwrong tone, up to 10 per comment. - Reuse instead of retyping. Every label already used on that product appears above the field as a chip you can click, so the same wording lands on the same problem across many turns, by anyone on your team.
- Filter sessions by label. The sessions list has a Comments filter offering No comments, Has comments, and one entry per label your product uses. Picking more than one widens the result.
hasComments parameter announced in the 2026-07-27 release with commentLabels, which takes __none__, __any__, or any label. The Python SDK does not cover comments yet.A Version Does Not Need a Connection Anymore
The version form now starts with a Link Connection toggle, off by default, instead of asking for connection fields whether or not you have a live product to point at. Leave it off for a version that only records the inferences you send from your own code: the conversation target and connection fields stay hidden. Turn it on to have Galtea run conversations against your product, then pick the target (HTTP endpoint, phone, or WebRTC) and a connection, now mandatory, so no version is half-connected by accident. Turning the toggle off on an existing version and saving removes its connection.See Version for what each connection type does, or Direct Inferences and Evaluations for the flow that needs a linked endpoint.Platform Improvements
- Check which platform version you are running: An About card at the end of the Settings page shows the deployed Version, Commit, and Build date, so an enterprise organization on its own infrastructure can compare it against the hosted platform. Visible to enterprise members and admins, and also available with
galtea build-info. - Full session lifecycle documented: The Session page now covers which
Statusvalues mean a session is open, the two ways it closes (sessions.finishor the auto-close inactivity window), and what the closed-session policy does to a turn that arrives late. - Specification descriptions during onboarding: The step for choosing which specifications to evaluate now shows a Description column, so you can tell similarly named specifications apart.
- Detail tabs renamed: The first tab on a version, session, test, test case, monitor, or organization page is now Overview instead of Details, matching the specification and user group pages. On a product it is now Settings.
- The dashboard follows your system appearance: With the theme set to System, the dashboard switches between light and dark the moment your operating system does, without a reload.
Breaking Change: Claude Sonnet 4.5 Model Name
The evaluator model name for Claude Sonnet 4.5 changed to match its real name on our LLM gateway."Claude-Sonnet-4.5"is now"claude-sonnet-4-5"."eu.claude-sonnet-4.5"is now"eu.claude-sonnet-4-5".
evaluator_model_name, through the SDK, the CLI, or the API directly. That call now returns a 400 error. Update it to use the new name. See the full list of available models.What does not break: the dashboard. Its model list comes from our API, so it already shows the new names.What does not break: metrics you already created. We renamed the model on our side and they now point at the new name automatically. Your existing evaluations keep running.Monitor Your Production Traffic Automatically
Testing tells you an AI product is ready to ship. Monitors tell you it is still behaving once real users arrive. A Monitor is an always-on evaluation rule: point it at a product, pick the metrics that matter, and Galtea keeps scoring your live production sessions on its own. There is no run to trigger, no schedule to maintain, and no extra code once your product logs its sessions.This is the difference between finding a quality regression in a weekly report and catching it as it happens. A Monitor watches the traffic your users actually send, so a drop in answer quality, a new failure pattern, or a prompt change that quietly hurt responses shows up on its own.- Set it once, watch continuously. Create a Monitor from a product’s Monitors tab (Production view). Each Monitor scores a sampled share of that product’s production sessions — 10% by default — so you get a representative quality signal without paying to score every single conversation. Raise or lower the sampling rate as your traffic and budget change.
- Metrics stay current on their own. A Monitor binds a metric family, not a fixed version. When you revise a metric, the Monitor picks up the new revision automatically — you never re-edit the Monitor.
- Spend stays under control. An optional monthly credit cap protects each Monitor’s spend, and your organization balance is always the hard limit. The Monitor’s status tells you exactly why it is or is not scoring right now:
ACTIVE,PAUSED,CAPPED, orNO_CREDITS.
See the Score Trend Across the Whole Product
A Monitor’s own page shows the results of that one Monitor. To see the trend across every production evaluation of a product, open its Analytics tab in Production view mode: an evaluation scores over time chart plots the average score per day. Switch between a by metric view (one line per metric family) and a by specification view (one line per specification, built from the metrics linked to it). The chart follows the same filters as the rest of the page (version, test, metric, language, and time frame) and includes every matching production evaluation, including the ones you triggered by hand.Sessions Close Themselves, so Monitors Know When to Score
A Monitor scores a session only once the conversation is over. So that Galtea knows when that is, a session now closes on its own after a period of inactivity, instead of staying open forever waiting for your integration to end it.- Configure it per product. In the product’s advanced settings you set the auto-close scope (
NONE,DEVELOPMENT,PRODUCTION, orALL— the default isPRODUCTION), the inactivity window in minutes (default 30), and what happens if a new turn arrives for an already-closed session:REJECTit with an error, orIGNOREit silently. - Close a conversation the moment it ends. When you already know a conversation is finished, close it right away with the SDK’s
sessions.finish(session_id)so its Monitor scores it on the next scan instead of waiting out the inactivity window. - Know when production will not be scored. If a product’s auto-close scope excludes production, open production sessions never close on inactivity, so their Monitors score only the sessions you finish explicitly. The dashboard warns you when that is the case.
Evaluation Scores Over Time Moved to the Production View
The scores-over-time chart announced in the 2026-07-21 release now appears only in the Production view of a product’s Analytics tab. It plots one average score per day, which answers “is the live product drifting”, so it belongs next to your Monitors rather than beside your test runs. Development still compares versions with the existing scores and performance charts. Details in Monitor.Manage Metrics and Tests Directly from a Specification
A specification is now the single place to manage its metrics and tests.- Link an existing metric or test. From a specification’s Hub tabs, the Metrics tab’s create dropdown gains a Link existing metric action, and the Tests tab’s create button becomes a dropdown with Create new test (opens the form already linked to the specification) and Link existing test.
- Linking moves a test. A test belongs to only one specification. If the test you pick is already linked elsewhere, a warning icon shows which specification it will move from.
- Unlink instead of delete. Inside a specification’s Hub tabs, the bulk row action is Unlink from specification rather than delete, so removing a metric or test from a specification never deletes it.
- See the link from the other side. A metric’s or test’s own detail page now shows its linked specification(s) read-only, and the test creation form always states whether the new test will be linked, and to which specification.
Comment on Inference Results
You can now leave free-form comments on an inference result to annotate a single turn during error analysis, without going through the metric-bound evaluation flow.- Add and delete comments from the new Comments tab in the inference result detail panel. Comments support Markdown and show the author and timestamp.
- A turn card shows a comment icon (with a count when there is more than one) so you can spot annotated turns without opening them.
- The sessions list has a new Has comments / No comments filter.
Platform Improvements
- Auto-close settings visible on product details: The product details page now shows auto-close scope, inactivity window, and closed-session inference policy read-only. Previously these were visible only on the edit page, behind Advanced Options.
Retrieval Context Now Comes from Traces
retrieval_context is now derived from RETRIEVER trace spans, the same way tools_used is derived from TOOL spans, instead of living only as a standalone field.- When you pass
retrieval_contextexplicitly (SDK parameter, endpoint output mapping, or session turns), Galtea also records it as aRETRIEVERtrace linked to the inference result. - The dashboard reads retrieval context from those traces everywhere it is displayed: the inference result details, the evaluations table and its CSV export, and the human-annotation dialog.
- Metrics that need
retrieval_contextcan now get it from your instrumented retriever spans directly — if your RAG pipeline emitsRETRIEVERspans, there is nothing extra to send. See Send OpenTelemetry Traces to Galtea for the span conventions.
Import Traces from More Frameworks
Trace ingestion recognizes more of the ecosystem, so conversations assemble with real content instead of empty turns:- MLflow Tracing. OTLP spans exported by MLflow-instrumented products are now decoded natively (
mlflow.spanInputs/mlflow.spanOutputs, model and token-usage metadata included). - Standard OpenTelemetry session attributes. Traces that follow the GenAI semantic conventions group into conversations by
gen_ai.conversation.idorsession.idout of the box — no Galtea-specific attributes needed. - Langfuse imports work for any framework. The importer no longer requires a LangChain-family signal: any Langfuse-shaped export with real conversation content imports, every observation type maps to its matching trace type (
retriever,tool,agent,chain, and more), and outputs are stored verbatim. See the Langfuse integration guide. - Answers produced without an LLM are recovered. When a turn is answered by a guardrail replacement, a templated redirect, or a tool-terminal reply, you can now teach assembly where that reply lives with a per-product (or per-version) output source rule in the advanced settings. Details in Monitor Real User Traffic with OpenTelemetry.
Replay Evaluations After Revising a Test Case
When you edit a test case’s content, Galtea forks a new revision — and now offers to bring your evaluation history along. Right after the edit, a dialog lists every product that evaluated the test case family before, each with its evaluation count, and re-runs them against the new revision with fresh inferences. Deselect any product you want to leave out, or skip the dialog entirely.Log Inference Results with Your Own Conversation ID
You can now append inference results to an existing session using your own conversation identifier instead of a Galtea session ID. Passsession_custom_id together with a version_id (or product_id) anchor to inference_results.create, and Galtea finds the session by the custom ID you set when creating it. If no session matches, the call fails with a clear error — session creation stays where it belongs, in the sessions API.Specifications Get a Name
Specifications now have a short, human-readablename alongside the long description, so lists and pickers are scannable at a glance. The field is editable across the dashboard, API, SDK, and CLI; existing specifications were backfilled from their description.Evaluator Model Retirement: Bedrock Claude Sonnet 4
As announced in the deprecation notice, the grace window for Claude Sonnet 4 (Bedrock) as an evaluator model has closed and the model is now retired, ahead of Bedrock’s end-of-life for it. Metrics that used it were migrated during the grace period; Claude Sonnet 4.5 and 4.6 remain available.Platform Improvements
- Monitors have their own tab: The monitor list moved from the collapsible card in Analytics to a Monitors tab in the Product Hub, between Sessions and Evaluations (Production view).
- Production mode sticks: Clicking Production on a product with no production evaluations no longer reverts to Development a second later. Your explicit choice is remembered.
- Connection fields match the target: Version forms only show the Initialization/Finalization connection fields when the Conversation Target is an HTTP endpoint; for Phone and WebRTC targets they are hidden, and stale selections are cleared when you switch.
- Copied products keep their files: Copying a product into another organization re-homes its stored assets (including call recordings), so downloads no longer fail in the destination organization.
- One score, everywhere: The Analytics insight for low specification scores now shows the same sample-weighted score as the Overview ring, instead of two different numbers for the same specification.
- Production session details filled in: The Details tab of a production session (which has no test case) now shows the inference metrics and stopping reason instead of rendering mostly empty.
- Clearer skip reasons: When an evaluation is skipped because a metric is missing parameters, the reason now lists every missing parameter, not just some of them.
- Dashboard form polish: The caret no longer jumps to the end while typing in list fields; the role dropdown for a new user starts empty instead of silently pre-selecting Member; and the organization settings page was redesigned into clear General, Security, and SAML SSO cards with an instant-save MFA toggle.
- Deleted organization handling: If your organization is removed while you are signed in, you are logged out instead of seeing broken pages.
Evaluate Your Voice Agents over WebRTC
You can now evaluate AI voice agents that connect over WebRTC, the standard for real-time audio over the internet. Instead of calling an HTTP API, Galtea opens a live WebRTC session with your agent through Pipecat Cloud and runs the conversation as a real voice call.Set it up once: create a WebRTC Connection with your Pipecat Cloud public API key and the name of the agent to target, then attach it to a version. A version reaches your product through exactly one target — an endpoint, a phone connection, or a WebRTC connection — so choose WebRTC when Galtea should open a WebRTC session. The API key is stored encrypted and never returned; a fetched connection reports only whether a key is set.WebRTC connections run any test type with no extra setup:- Behavior test cases run multi-turn, with the conversation simulator driving each user turn from the test case’s persona, goal, and scenario.
- Accuracy and Security test cases run single-turn: Galtea speaks the test case’s input line into the call and evaluates the reply.
webrtc-connection resource.Launch Runs Without Picking a Version
You no longer have to select a version every time you start work. Point Galtea at a product and it resolves the version for you.- Evaluations: In the Run Evaluation dialog, the version field now offers Automatic (latest configured version). Leave it on Automatic and Galtea clones your latest properly configured version, then runs the evaluation against that new copy. The clone keeps a link back to its parent, so your version history stays clear.
- Sessions and single-turn logging: When you log a session or a single-turn result without a version, Galtea reuses the product’s latest version instead of asking you to choose one.
- SDK: Session and evaluation methods now accept
product_idas an alternative toversion_id, so you can write scripts against a product and let Galtea pick the version. Seesessions.createandevaluations.create_single_turn.
Platform Improvements
- Lists and details refresh on their own: Pending evaluations, sessions, and the evaluation details page now update about every 5 seconds while work is in progress, so a finished status, score, and reason appear without a page reload. Polling stops once nothing is pending.
- Simulated conversations speak the regional variety: When a test case uses a regional language tag such as
es-MX(Mexican Spanish) orpt-BR(Brazilian Portuguese), the simulated user now writes in that local variety, not just generic Spanish or Portuguese. - Human evaluation is reachable again: The Manual Evaluation button and the per-row Annotate action are visible once more. A permission-check bug had hidden them for every role, which blocked starting a human evaluation from the dashboard.
- Clearer status filter and Manual Evaluation button: The evaluations status filter now shows readable labels (Pending, Pending Human, Success, Failed, Skipped), and the Manual Evaluation button hides when there is nothing left to annotate instead of showing a “(0)” count.
- More accurate coverage insight: The analytics coverage insight now counts only Policy specifications. Capability and inability specifications, which never carry evaluation scores, no longer inflate the “specifications with no scores yet” count.
- No credit charge for deterministic-only evaluations: Evaluations scored only by deterministic metrics use no LLM credits, but each one was still charged a single credit. They are now correctly free.
- ZIP uploads with special characters now process: Uploading a ZIP for dataset generation whose files have spaces, accented characters, or subfolders in their names failed with an opaque error, while single files worked. Those ZIPs now parse correctly.
- Cleaner, more reliable voice test calls: Voice runs over a phone connection are steadier. The per-turn timer now starts when the caller’s audio finishes playing, so slow personas no longer trigger a false “agent did not respond in time”. A genuinely slow agent reply is saved as a partial turn instead of being dropped, and captured agent audio has its leading and trailing silence trimmed for cleaner playback.
Track Changes to Your Test Cases
Test cases now keep a revision history, like metric revisions. When you edit a test case’s evaluation content, the platform creates a new revision instead of overwriting the old one. Annotation-only edits, such as reviewer or score, still edit in place. The list shows only the latest revision by default, with an opt-in filter for older ones, and each revision links back to the one it came from. Evaluations that ran against an old test case now show an Outdated Test Case badge, and the SDK warns you when this happens.Compare Version Coverage in Analytics
The analytics page now shows shared test-case coverage so you can compare version scores fairly. Two versions can have the same average score even if one was evaluated against many more test cases, and shared coverage makes that gap visible. Each version shows how many test cases it was evaluated against, out of all test cases used across the shown versions, as a count and a percentage, with a Total / Used / Ignored summary. The selected-version area of each analytics tab is now two cards: radial gauges for the selected version, and a scores radar across all shown versions. Both stay visible and follow one grouping toggle (Specifications / Metrics). All numbers respect the active filters.Platform Improvements
- Iterate a version from an existing one: You can now create a version from an existing one to track how it evolved. Each version records its parent as an Evolved from link, available in the dashboard, API, and SDK.
- Persona gender on test cases: Test cases can now store and show the persona’s gender.
- Attach session metadata at creation: The SDK can now add metadata to a session when it is created.
- Cancelled evaluation status in the SDK: The SDK
EvaluationStatusenum now includesCANCELLED. - Development routes hidden in production: Dev and demo routes are no longer shown in the production dashboard.
- Clearer report export errors: Exporting a report for a product with no data now returns a clear client error instead of a server error.
- Faster permission checks: Permission checks now run in parallel, so permission-heavy requests are faster.
- Sturdier SDK networking: The SDK now reuses one HTTP session with retries and a timeout, and only prints colored output to a terminal.
- Dashboard polish: Reduced oversized text, fixed clickable empty space in truncated table cells, restored non-admin assistant routes to the catalog, and refreshed the sidebar create-product button.
Run Voice Simulations on Your Phone Agents
You can now run voice simulations against AI agents that answer the phone. A new phone connection tells Galtea how to reach your agent: you set the phone number once, and Galtea dials your agent and runs the conversation as a real voice call. This is Direct Inference over the phone.A few things that make it powerful:- The caller sounds human, in your agent’s language. Set a test case’s language and the synthetic caller speaks it with a natively-accented voice (a Spanish test case is answered in Spanish), so one phone connection covers many languages with no extra setup.
- Voice agents are scored like text agents. Galtea transcribes each agent reply with speech-to-text automatically, so every metric you already use works over the phone, including security tests that speak adversarial prompts out loud.
- You can replay the call however you like. The entire conversation is captured as a continuous call recording that you can play back in the dashboard, either as a single clip or turn by turn.
Update Specifications from a Document
As your product evolves, its documentation (PRDs, design docs, spec sheets) often changes before its specifications do. A new Update from document action on the product detail page bridges that gap.Upload a document and the platform compares its content against your product’s existing specifications. Each suggestion is classified as one of:- New — a behavioral expectation found in the document that has no matching specification yet.
- Updated — an existing specification that the document refines or extends.
- Contradictory — an existing specification that conflicts with something in the document.
Product Cards Show Version Scores
Product cards now tell you more at a glance. Each card shows two version rows:- Best — the version with the highest average evaluation score, shown with a percentage (e.g.,
v2 (55%)). - Newest — the most recently created version.
(-) instead of a percentage. The Evaluations counter and Total Cost row have been removed from cards to keep the view focused.Cross-Product Ranking in Val
Val (the Galtea AI assistant) can now compare your products against each other on a shared metric. Ask Val something like “how does the whole product family perform on Faithfulness?” and it returns a worst-first ranking with the evaluation count, mean, and min/max score for each product, plus an overall evaluation-weighted score across all of them.If you do not specify a metric, Val first lists the metrics shared across your selected products and waits for you to pick one before returning numbers.Platform Improvements
- Try a demo product during onboarding: New users can now clone a ready-made Galtea demo during the onboarding flow.
- Faster audio loading in inference results: Audio files in inference result sessions now load lazily and in a single batched request as turns scroll into view, rather than fetching all URLs and preloading all audio on render. The audio player loads only the duration header for visible turns, so the seek bar shows the correct length without downloading the full file. Play latency is unchanged.
- Val assistant fixes: Several usability issues in the Val widget are resolved. The ⌘K / Ctrl+K shortcut no longer fires while you are typing in another field. Pressing Enter mid-stream inserts a newline as expected. Citation chips no longer appear for links inside code blocks. The clipboard copy button falls back gracefully on plain HTTP. The route catalog no longer suggests admin-only pages that regular members cannot access.
- Tooltips close when the pointer moves onto their content: Tooltips now close as soon as the pointer leaves the trigger, like any standard hover element. Previously, moving the pointer from the trigger onto the tooltip kept it open and could block clicks on elements beneath it.
- Long entity names stay inside their column: Long names and identifiers in link cells (Test, Version, and similar columns in data tables) now wrap and clamp to two lines inside their column instead of overflowing into the next one.
Platform Improvements
- Create specifications during onboarding: You can now create your custom specifications during the onboarding flow to cover gaps you would like to evaluate that our AI system did not detect or did not expect to be a primary focus.
- Filter sessions by status: The session list now has a status filter — select
PENDING,COMPLETED, orFAILED(or any combination) to focus on the runs that matter. Available in the dashboard, the API (statusesquery parameter onGET /sessions), and the SDK viasessions.list(status=…). - See direct inference progress right away: When you start a direct inference run, every session it will produce now appears in the list as
PENDINGimmediately — instead of trickling in as each one is picked up — so you can track progress from the moment the run starts. - Security metrics now evaluate correctly: Data Leakage, Jailbreak Resilience, and Misuse Resilience evaluations were being silently marked
SKIPPEDbecause they depended on a field the product form no longer surfaces. These metrics now draw context fromproduct_capabilitiesandproduct_inabilities(populated from your specifications) and run as expected. - “New Metric from template” fixed: Clicking From template in the New Metric screen was throwing a routing error and closing the dialog immediately. The dialog now opens and works correctly.
Documentation and Examples
- SDK evaluation examples now show the specification-first approach — passing
specification_ids=instead of metric names — side by side with the metric-name approach, in tabbed code blocks, on every relevant tutorial page.
Smoother Onboarding
Getting your first product evaluated now takes less waiting and less typing.Generation runs in the background. When the onboarding tutorial creates the tests and metrics for your selected specifications, it no longer holds you on the review step until everything finishes. Generation continues while you keep moving through setup — only the final Launch Evaluation button waits for it to be ready, and if anything fails to generate you get a clear retry instead of a stuck screen.Your first product is named for you. The onboarding form no longer asks you to name your product. The platform derives a concise name from the description you provide as part of the same step, so there’s one less field between you and your first evaluation.Platform Improvements
- Endpoint connections are checked before any work is queued: When you start an evaluation, the platform now runs a quick pre-flight probe against the version’s endpoint connection first. If the endpoint is unreachable or returns an error, the request is rejected immediately with a clear message — instead of queuing inferences that were always going to fail.
- Faster large evaluation runs: Evaluation runs now process their sessions in parallel rather than one after another, so large runs finish sooner and a single big run no longer blocks others behind it. Each run also respects a per-endpoint rate limit, so your endpoint is never flooded with requests.
Documentation and Examples
- A new Improve Your Product with Evaluations and an AI Agent playbook walks through the full evaluate → diagnose → improve → compare loop: hand your failing evaluations to an AI coding agent using the Galtea Agent Skill and CLI, let it propose a fix, then re-run the evaluations to prove it helped — illustrated end to end on a RAG support chatbot.
- The Direct Inferences & Evaluations from Platform tutorial now documents the full round-trip for stateful multi-turn conversations: how to extract a value from one response into session metadata, and then send it back on the next turn.
Analytics Insights
The Analytics Overview tab now includes an Insights card that automatically scans your evaluation data and surfaces what needs your attention. The card applies deterministic rules across three categories:- Score health — flags metrics that fall below the warning (85) or critical (60) thresholds, and links each flag directly to the relevant evaluations with the judge’s reasoning so you can dig in immediately.
- Specification coverage — identifies specifications that have no linked tests yet, so coverage gaps don’t stay invisible.
- Version regressions — detects metric scores that dropped 10 or more points from one version to another, distinguishing warnings from critical degradations.
Smarter Run Evaluation Flow
Running an evaluation from a version or product now opens a Specifications tab by default. It pre-selects the specifications that have runnable tests and submits via the existing batch endpoint — the platform resolves linked tests and metrics server-side, so you don’t have to select them manually. If a product has no specifications, the dialog falls back to the familiar manual test and metric selector.In the manual selector, the metric picker now ranks and pre-selects metrics based on relevance:- Linked — metrics directly linked to the specification of your selected test are pre-selected and appear first.
- Recommended — metrics linked to any specification sharing the same test type appear next with a badge.
Complete Metric with AI
Two places in the dashboard now offer a Complete With AI button on metrics: the manual create-metric form and the candidate edit dialog inside the AI metric generation review step.Fill in the metric name, description, and evaluation type — and let the AI generate the judge prompt and evaluation parameters for you. Your name, description, and tags are never overwritten. If you already have a judge prompt, the platform prompts for confirmation before replacing it. AI-generated fields display a badge so you always see what was filled by the model.This works for AI Evaluation and Human Evaluation metrics. Self-Hosted metrics are excluded (they have no judge prompt).AI-Assisted Specification Creation
Two improvements to the specification creation flow:Fill with AI (the dialog on the create and update pages) now rewrites your rough note into a properly written specification description and classifies the type in one step, returningdescription, type, testType, and testVariant together. Previously it only filled the category selectors and never touched the description.Complete with AI (the inline button) sends the description you already wrote, applies only type, testType, and testVariant, and never touches your text. The button was previously labeled “Categorize with AI” — the new name reflects that it completes the fields you left empty without altering the ones you filled.In Detailed generation mode, the AI now also considers a built-in catalog of safety and quality specifications that apply to most AI products: jailbreak resistance, illegal-activity refusal, system-prompt non-disclosure, cross-user data leakage, AI-identity disclosure, and others. Specifications that don’t fit (for example, safety specs for backend-only products with no human users) are excluded automatically. You review and edit everything before any of it is saved.Import Endpoint Connection from cURL
Creating an endpoint connection now offers an Import from cURL option. Paste a cURL command — from your API docs, a Postman collection, or a terminal history — and the platform extracts the URL, headers, and request body, then maps the body to a Jinja2 input template with{{ input.user_message }} pre-wired. For conversation-type connections, a lightweight AI pass enriches the template further. The New Endpoint Connection button is now a From cURL / Blank dropdown.In the onboarding flow, the cURL paste step is the default view; the traditional form is available via Skip — fill in manually. Fields that couldn’t be recovered from the cURL command are flagged inline, and a mandatory probe test confirms the connection works before saving.Platform Improvements
- Row click navigates to entity details: Every table row in the dashboard now navigates to the entity’s detail page on click. The primary cell renders a real link, so Ctrl/Cmd/middle-click opens it in a new tab. Clicking on a text cell opens a full-text dialog with a copy button instead of navigating. Drag-to-select on table text still works.
- Assistant renamed to Val: The Galtea platform assistant is now named Val across the dashboard panel and the public docs site. The panel header, launcher pill (“Ask Val”), and the agent’s own persona are all updated.
- Breadcrumb navigation restores your position: Clicking a breadcrumb link now returns you to the exact tab and pagination page you were on when you drilled into the current entity. For a fresh deep link, it falls back to the tab that lists the entity you were viewing.
- Analytics score charts use a fixed 0–100 axis: Score charts in the Analytics section are anchored to a 0–100 scale so visual proportions are consistent across products and time windows. Previously the axis auto-scaled to the data range, making small absolute differences appear large.
- List endpoints default to your organization: Every
GETlist endpoint now scopes to the caller’s organization automatically, so you no longer need to pass an organization filter to list your own data. Admins can opt into a cross-organization view with theallOrganizationsflag.
Galtea Assistant Redesign
The Galtea AI assistant is now a non-modal docked side panel that sits alongside the dashboard instead of covering it. When open, the page content reflows into a true split view — the panel holds a fixed width on the right while the rest of the dashboard fills the remaining space. You can resize the split by dragging the divider. Open and close the panel from anywhere in the dashboard with ⌘K (Mac) or Ctrl+K (Windows/Linux).The assistant now shows its work while it processes your message. A live step trace displays each phase of the agent’s reasoning — searching the docs, calling API tools, writing the answer — with the active step pulsing in real time. When the response arrives, the trace collapses to a compact summary (for example, “Worked for 2.3s · 4 steps”) that you can expand at any time to review the full trajectory.For write operations, the assistant proposes an action and you approve it before anything changes. Each proposal shows a preview of what will happen, and destructive actions display how many items would be affected before you confirm. When you approve, the dashboard navigates you directly to the affected product, version, or evaluation. The assistant can now reach any allow-listed resource in the platform — reads are driven by the live API catalog, and writes cover 28 curated operations — so it adapts as new endpoints are added without requiring new tool code.Redesigned Onboarding Experience
New users now flow through a focused, full-page onboarding shell that adapts to how you want to run evaluation — import traces, connect your endpoint, or use the SDK — and pre-selects the specifications matching the goal you gave at signup, so first evaluation results arrive faster.Word Document Upload for Product Setup
When setting up your product with AI assistance, you can now upload Word documents (.docx) alongside PDFs and other formats. The platform extracts the content and uses it to generate specifications, so teams that maintain documentation in Word can feed it directly into Galtea without converting files first.Job Cancellation
You can now cancel in-progress async jobs from both the API and the SDK. When you start an inference batch — for example, viaevaluations.run() in endpoint-connection mode — and need to stop it early, call galtea.jobs.cancel(job_id). Queued jobs are removed from the queue; jobs already running receive a cooperative cancellation signal and stop at their next checkpoint. Work already dispatched when the signal is observed is not rolled back. Cancellation is idempotent — calling cancel() on an already-cancelled job returns state='cancelled' rather than raising an error. Use galtea.jobs.get_status(job_id) to check the current state before deciding whether to cancel. See the Jobs Service reference for the full API.Platform Improvements
- PASS/FAIL verdict shown in the annotation dialog: When annotating evaluations, the score pill now displays the binary PASS/FAIL outcome alongside the numeric score so you can see the verdict at a glance.
- “See details” button on every table row: All entity tables — sessions, evaluations, tests, and more — now show a direct-navigation icon button to the right of the row actions menu. Click to open the detail page; hold Cmd/Ctrl to open it in a new tab.
- Unified test creation form: Creating a test from a single specification now uses the same creation form as batch creation. The previous dedicated single-spec flow has been retired.
- Accuracy-requirement specifications correctly classified: Specifications of the accuracy-requirement type are now saved as
POLICY + QUALITY. Previously, creating one via the API produced an incorrect type. - Coverage radar chart renders correctly in edge cases: The coverage radar chart now renders when two specifications share the same name or when a version has no scores for one of the axes. Previously, either condition caused the chart to be skipped.
- Clearer SDK error messages for agent integrations: When an agent skill invocation fails due to an argument shape mismatch, the SDK now surfaces a clear description of the issue instead of a cryptic internal error.
- LLM configuration errors surface immediately: Misconfigured API keys, unsupported model names, and other non-retryable LLM errors now cause evaluations to fail on the first attempt rather than consuming the full retry budget before reporting the problem.
- Red-teaming generation handles content filters: When a content filter blocks a generation step, the red-teaming generator now retries. If the seed prompt itself keeps triggering the filter after retries, that seed is skipped and generation continues with the remaining prompts.
- Specification linking documented in CLI and SDK: The
galtea spec link-testsandgaltea spec unlink-testscommands and their SDK equivalents —galtea.specifications.link_tests()andgaltea.specifications.unlink_tests()— are now documented in the CLI--helpoutput and SDK docstrings, making them easier to discover.
Metric Optimization Lifecycle
When you optimize an AI Evaluation metric, the result now behaves like any other revision of that metric. The optimized version joins the same revision lineage as a manual “Create new revision” — so you can trace the full optimization history in the metric’s revision panel and see how the judge prompt has evolved across generations.The Optimize button now shows the correct annotation threshold. It counts only the human annotations where your score and the AI’s score fall on opposite sides of the PASS/FAIL boundary — evaluations where your annotation genuinely overrides the AI’s verdict. Previously, any numeric difference between scores counted, even when both scores agreed on the outcome.Platform Improvements
- Incompatible evaluations now show
SKIPPEDinstead ofFAILED: Evaluations that cannot run — such as sessions where the agent produced no output, or deterministic metrics applied to multi-turn sessions — are now caught at pre-flight and markedSKIPPEDimmediately, before being dispatched to the evaluator. Previously these were dispatched and returnedFAILEDwith no clear explanation. The evaluation result now includes the specific reason the evaluation was skipped.
Documentation and Examples
- Direct Inference OTel collector: authentication guide updated: The Direct Inference OTel exporter tutorial now covers the required
Authorization: Bearer <Galtea-API-key>header with four supply patterns — SDK environment variables over HTTP, SDK environment variables over gRPC, programmatic Python, and OTel Collector YAML relay. The section also documents shell-expansion pitfalls withOTEL_EXPORTER_OTLP_TRACES_HEADERS, acurlsmoke-test command for self-validation, and a note on the current gRPC TLS handshake issue with HTTP/4318 as the validated path.
Import Langfuse Traces
If you already export traces from Langfuse, you can now bring them into Galtea without any code changes. Open New Session → Import traces (Beta), upload your Langfuse export file — JSON, JSONL, or CSV are all accepted, and both the Traces and Observations export tabs from Langfuse work — select the version you want to link them to, and click Import. Each trace becomes a session with its observations mapped to inference results, ready to evaluate immediately.The import runs as a background job, so you can navigate away while large files are processed. The dialog stays open and shows progress; once complete, the sessions table refreshes automatically.This is the recommended path for teams already using Langfuse’s LangChain or LangGraph integration and wanting to run Galtea evaluations on their existing trace history without modifying their agent code.SDK: Model Management
The SDK now includes aModelService, accessible as galtea.models. It supports the full lifecycle — list, get, get_by_name, create, update, and delete — bringing programmatic parity with the dashboard’s model management for cost tracking and estimation. list() accepts both a partial-match name filter and an exact-match names list, alongside the standard ids, pagination, and date-range parameters.See the Model Service reference for the full API.Platform Improvements
FULL_PROMPTmetric creation is now blocked:POST /metricsandgaltea.metrics.create(...)reject requests that setsource=FULL_PROMPTor that supply ajudgePromptwithoutevaluationParams. UsePARTIAL_PROMPTfor new AI Evaluation metrics — it covers the same use cases with explicit, user-configurable evaluation parameters. ExistingFULL_PROMPTmetrics are unaffected and continue to be evaluated end to end.- SDK
generate()reflects live agent status:inference_results.generate()now creates the record asPENDINGwhile your agent runs and only transitions toGENERATEDon a successful return. Previously the record was immediately markedGENERATEDwith a null output, which broke the contract thatGENERATEDmeans the agent completed successfully. - Onboarding video on the welcome screen and Quickstart: The product welcome screen and the Quickstart page now include an embedded walkthrough video so new users can see the end-to-end setup flow before writing any code.
- More accurate AI evaluation scores: Judge prompts now produce
reasonbeforescore. This chain-of-thought ordering consistently improves score quality by forcing the model to articulate its reasoning before committing to a number. - Table header overflow fixed: Long column headers — such as version hashes in the analytics comparison table — are now clipped at the cell boundary instead of visually bleeding into adjacent columns.
Metric Optimization
When you create a custom AI Evaluation metric, the judge prompt rarely scores responses exactly the way your team would on day one. The new Optimize action closes that gap: it takes a metric’s current judge prompt, learns from the human annotations your team has already submitted, and rewrites the prompt so the AI score lines up more closely with the human score. The result is saved as a sibling metric named<original name> (Optimized), so you can compare both versions side by side before adopting the new one.The Optimize button appears on the details page of any AI Evaluation metric created by your organization. It becomes available once enough annotations have been collected, with a particular focus on cases where the annotator disagreed with the AI score: those disagreements are the signal the optimizer learns from.Galtea Assistant (Beta)
A new in-product AI assistant is now available. The floating launcher in the bottom-right corner of the dashboard opens a chat panel that can answer questions about the platform, point you to the right page, and walk you through common workflows. The same assistant is also embedded in the docs site.Form Improvements
Submitting a new product now uses AI to validate that the description matches what Galtea can test. Submissions that look voice, image, or multimodal, or that read more like a system prompt or build instructions than a product description, open a confirmation dialog before saving. The check is advisory and never blocks creation, so you can review the warning and proceed or go back and edit.Navigating away from any creation or edit form with unsaved changes now opens a confirmation dialog with “Discard” and “Keep editing” choices, so accidental clicks no longer lose your work. Forms protected include metric and product creation and edit, test edit, endpoint connection edit, and dialog-level flows such as Add Specification.Platform Improvements
- Full customization when creating tests from Behavior specs: Creating a test from a Behavior specification now gives you the complete test creation form, with every customization option available, including data-catalog file uploads.
- Faster AI-assisted product setup: Generating a product description and specifications during product creation is now noticeably faster. Specification file uploads are also accepted in both quick and detailed generation modes.
- Clearer status for empty-parameter evaluations: Evaluations whose required fields (such as
retrieval_context) are empty or whitespace-only strings now resolve asSKIPPEDupfront with a clear “missing required parameters” reason, instead of dispatching and landing asFAILED. - CLI: background spec refresh and safer plaintext login: The Galtea CLI now refreshes its OpenAPI spec cache in a detached background process, so commands no longer pay the refresh latency. Logging in to a non-localhost HTTP host now requires
--insecureto prevent accidental credential exposure over plaintext; the consent is sticky on that host so re-logins do not re-prompt.
Specifications View Redesign
The specifications list is now split into two tables: Policies and Capabilities & Inabilities. Each row shows separate Metrics and Tests columns — a green check with a count when the resource exists, or a “Missing” pill that routes you to the test creation form when it does not.Two-Factor Authentication
Email two-factor authentication is now available under Settings → Security, and organization admins can enforce MFA for all members from the organization settings page. See Registration and Sign-In for details.Documentation and Examples
- Galtea Agent Skill: The Galtea Agent Skill page has been refreshed with updated install instructions and the Claude Code plugin install path.
Platform Improvements
- Fraction input for annotation scores: The human evaluation score field now accepts fractions (e.g., 7 out of 10) alongside percentages.
- Max iterations for behavior test generation: Behavior test AI generation now exposes a
max_iterationssetting, with the default lowered from 20 to 10. - Language field promoted in test creation: The Language dropdown is now visible directly in the test creation form, not only under Advanced Options.
- CLI via Homebrew: The Galtea CLI is now installable via Homebrew. See the CLI installation page for all options.
- Same-name metric revisions: Creating a metric revision that reuses the parent’s name no longer fails.
Galtea CLI
The Galtea CLI is now generally available. Install it viaapt, dnf, or pip, then authenticate with galtea login or a GALTEA_API_KEY environment variable. The CLI auto-generates a galtea <resource> <verb> command for every public API endpoint, so new endpoints appear as new commands without a CLI release. See the new Installation and Usage pages to get started.The Galtea Agent Skill now drives Galtea through the CLI, which improves how reliably coding assistants like Claude Code and Cursor can manage Galtea on your behalf.Product Tags
Products now support tags. You can add and edit tags from the create and update forms, filter the product list by one or more tags, and see tags as chips on product cards and the details page.Smarter Pre-Evaluation Guidance
Before launching an evaluation, the run dialog now checks whether each selected metric can be satisfied by the current product and endpoint connection configuration, and flags any that cannot with a tooltip describing what is missing.When an evaluation is still SKIPPED at runtime, the error message now explains what is missing and where to fix it, grouped by source, with actionable guidance included.Structured Inputs for AI-Generated Tests
Structured JSON inputs were previously limited to uploaded or manually created test cases. AI-generated test cases — scenarios, red-teaming, and gold-standard — now also receive a structuredinput populated from the endpoint connection’s JSON Schema.AI-assisted product and specification generation also now respects the language of the source material.Documentation and Examples
- New CLI Installation and CLI Usage pages.
- New
AgentInputreference page with all fields, theConversationMessagestructure, and updated SDK code examples.
Platform Improvements
- Endpoint connection picker in test creation: Test creation and the AI candidate review screen now offer an explicit endpoint connection picker under Advanced Options.
- Filter test cases by input substring: The Test Cases list has a new debounced text filter, including matches inside structured JSON inputs.
- Metric lineage on the details page: Revised metrics display an “Evolved from” link to the parent metric, and the revision flow now shows a warning before marking the source metric as legacy.
- Inference Results in the session dropdown: The session row dropdown now includes an “Inference Results” option alongside “Evaluations” and “Traces”.
- Sidebar cleanup: The Models entry has been removed from the sidebar. Models remain reachable from the organization dropdown and the version forms.
- Optional secret fields on endpoint connection updates: Credential fields are now optional when editing an existing endpoint connection — leave them empty to keep the stored value.
- SDK
@observefix for async agents: Theobservedecorator now correctly preserves Galtea context across async coroutines. - Clearer max-iterations stopping reason: Conversations that hit the iteration cap now record a “Max iterations reached” stopping reason instead of a generic error.
- Updated wording: Product creation now reads “Register product” throughout the platform.
Metric Versioning with Evaluation Replay
Metrics now support versioning. The new “New Revision” action on any metric opens the creation form pre-filled with the source metric’s name, description, tags, evaluation type, judge prompt, evaluation parameters, evaluator model, and user groups. Revised metrics stay linked to their predecessor through a parent-child lineage, so you can trace the evolution of a metric over time. When a revision is created, only the direct parent is marked as legacy, allowing parallel branches from the same source.After saving a revision, you are prompted to replay historical evaluations from the metric family against the new version. The replay dialog lists your products, shows how many previous evaluations are eligible per product, and lets you select which ones to re-run. Evaluations that have already been replayed are automatically deduplicated.The metric creation flow now surfaces all three modes upfront: Blank, From Template, and Metric Revision, each with a description of what it does. “From Template” (previously called “Duplicate”) creates an independent copy with no lineage link, while “Metric Revision” creates a linked successor.Galtea Agent Skill
The Galtea Agent Skill is now available as an open-source Agent Skill for AI coding assistants. Once installed, tools like Claude Code, Cursor, and Windsurf can authenticate against Galtea, manage products, run evaluations, inspect sessions and traces, and follow validated workflows without you having to explain Galtea first.Install it by pointing your coding agent to theGaltea-AI/skills repository, or use the skills CLI: npx skills add Galtea-AI/skills --skill "galtea".Run Evaluations from Specifications
You can now run evaluations directly from a specification page. The action is available both on the specification detail page header and as a row action on the specification list. The dialog shows a version dropdown (filtered to versions with an endpoint connection, latest pre-selected) and displays the specification’s linked tests and metrics as read-only badges, so you can verify the evaluation scope at a glance before launching.The action is gated on the specification having at least one linked metric and one linked test. When either is missing, the button is disabled with a tooltip explaining the requirement.Language Variables in Input Templates
Two new built-in template variables are available in endpoint connection input templates:{{ language_code }} resolves to the test case’s ISO 639-1 code (e.g. es), and {{ language_name }} resolves to the full language name (e.g. Spanish). Both resolve to an empty string when the test case has no language set, keeping templates compatible with language-agnostic tests. The placeholder toolbar in the dashboard includes both variables for easy insertion. For the full template syntax reference, see Structured Input Template Syntax.Reliable Trace-Dependent Evaluations
Evaluations that depend on trace data (TRACES or TOOLS_USED parameters) now wait for traces to be ingested before running. Previously, these evaluations could execute against empty trace data when traces had not yet arrived. The platform now defers them with automatic retries, and if traces do not arrive within the retry window, the evaluations land as SKIPPED with an actionable reason explaining what was missing. Sessions already marked as completed are detected immediately, avoiding unnecessary waiting. No credits are consumed for skipped evaluations.Export Evaluation Results as CSV
You can now export evaluation results directly from the dashboard as a CSV file. The export action appears in the row selection toolbar on the evaluation list page. With no rows selected, all evaluations matching the current filters are exported. When specific rows are selected, only those are included.The CSV includes columns for Score, Metric, Input, Output, Expected Output, Retrieval Context, Version, and Test, along with export-only identifier columns (Evaluation ID, Session ID, Test Case ID) that do not appear in the table UI but provide full traceability for downstream analysis.Inference Result Error Tracking
When an agent crashes or returns an empty response during simulation, the corresponding inference result is now properly marked asFAILED with the error details attached, instead of remaining stuck in a PENDING state. Each inference result carries its own error information, so you can see exactly what went wrong at the individual turn level.The evaluation pipeline also respects these statuses: evaluations are automatically skipped for non-ready inference results, with a clear reason recorded explaining which turns were not ready and why. In the dashboard, failed turns display a red error badge with the error message visible inline.One-Click Metric Duplication
You can now duplicate an existing metric as a starting point for a new one. The “Use as Template” action in the metric dropdown navigates to the creation form pre-filled with the source metric’s name, description, tags, evaluation type, judge prompt, evaluation parameters, evaluator model, and user groups. The duplicated metric is independent from the original, so you can adjust any field before saving.Platform Improvements
- Test case source file tracking: Test cases generated from uploaded documents now display the name of the original source file, improving traceability for file-based test generation.
- Annotate action on evaluation details: The Annotate (or Evaluate, for pending human review) action is now accessible directly from the evaluation detail page header dropdown, in addition to the list view context menu.
- Clearer error messages for test generation: When QUALITY test generation fails, the error message now includes the specific reason returned by the generator, replacing generic HTTP error responses with actionable guidance.
- Analytics radar chart fix: The radar chart in the Analytics tab now works correctly when a specification section has fewer than 3 scored items. The view automatically switches to radial value cards with a tooltip explaining the minimum requirement.
- Smoother table text filters: Text filter inputs across all dashboard tables no longer flash or revert while typing. Filter changes are debounced before syncing to the URL, reducing unnecessary API calls and eliminating the visible blink caused by intermediate re-renders.
Human Annotation for All Evaluations
Human annotation is no longer limited to human evaluation metrics. You can now annotate any completed evaluation with a human score and reasoning, regardless of its metric source. This means AI-evaluated results from full-prompt, partial-prompt, deterministic, or self-hosted metrics can all receive a parallel human assessment. Unlike human evaluation metrics (which contribute to the evaluation score and analytics), human annotations are stored as supplementary feedback alongside the original result.SDK: Endpoint Connection Management
The SDK now provides a full CRUD service for endpoint connections, so you can manage your agent connections entirely from code. Available operations includecreate, get, get_by_name, list, update, delete, and test.The service supports all authentication types (Bearer, API Key, Basic), retry configuration with customizable backoff strategies, and partial updates where you can modify individual fields without resending the entire configuration.SDK: Job-Based wait_for
The evaluations.wait_for() method now accepts a job_id parameter in addition to evaluation_ids. When you run an evaluation against an endpoint connection without a local agent, run() returns a jobId. You can now pass that ID directly to wait_for(job_id=...) and let the SDK handle the entire lifecycle: polling the job status, discovering the evaluations it produced, and waiting for them to complete.SDK: Langfuse CallbackHandler
Building on the Langfuse integration from the previous release, the SDK now includes a CallbackHandler for LangChain users. Instead of decorating individual functions with @observe, you can pass the handler directly to chain.invoke(config={"callbacks": [handler]}) and every LLM call, tool invocation, and retriever query in the chain will be traced automatically.The handler manages Galtea context through depth tracking, so nested callbacks are handled correctly without manual cleanup. It works as a singleton (set the inference result ID per request) or as a per-request instance. See the CallbackHandler integration guide for setup instructions.Structured JSON Inputs
This release extends JSON-structured inputs across the platform.SDK:input_data and context_data on AgentInput. When using the structured agent signature, your agent function now receives context_data alongside the conversation messages. If your test cases include structured context fields (e.g., customer_tier, language, or any custom key-value pairs), they flow automatically into AgentInput.context_data during evaluation and simulation. The generate() method also accepts structured input as a dictionary with multiple fields.CSV import with JSON values. When importing test cases from CSV, the platform now detects and preserves JSON objects in the input and context columns. You no longer need to flatten structured data before uploading. For the time being, structured JSON inputs are only available for uploaded or manually created tests.Langfuse Integration
If you already use Langfuse to observe your LLM application, you can now send the same traces to Galtea for evaluation by swapping a single import. Replacefrom langfuse import observe with from galtea.integrations.langfuse import observe, and every span will flow to both Langfuse and Galtea simultaneously.The integration is fully transparent to Langfuse: your Langfuse dashboard, trace IDs, and alerts remain completely unaffected. Galtea only exports traces when you explicitly link an inference_result_id, so there is no overhead for unrelated function calls.Two wrappers are available:observedecorator for the standard@observepatternstart_as_current_observationcontext manager for manual span control
Public REST API Reference
The platform now exposes a public OpenAPI specification covering approximately 81 endpoints across products, versions, specifications, tests, sessions, evaluations, metrics, traces, and more. Internal-only endpoints (authentication flows, billing, user administration) are excluded automatically.You can import the spec into Postman, Cursor, Claude, or any OpenAPI-compatible tool to explore and call the API directly. Every endpoint description now includes a link back to the relevant documentation page, making it easier to navigate between the spec and the guides.Platform Improvements
- Improved code and JSON display: Code and JSON data across evaluation details, trace details, session details, and endpoint connection test results now render with proper formatting and syntax highlighting instead of plain preformatted text.
- Flexible behavior test case editing: The input field is no longer required when editing behavior test cases, since behavior-type tests do not always need explicit input values.
- Specification type tooltips: Hovering over the specification type pill (Capability, Inability, Policy) in the dashboard now shows a short description of what each type means, linking to the Specifications documentation for more detail.
Zero-Code Trace Ingestion via OpenTelemetry
If your service is already instrumented with OpenTelemetry, you can now send traces to Galtea without writing any Galtea-specific code. The platform exposes an OTLP-compatible trace ingestion endpoint that accepts standard OTLP/JSON payloads from your existing OTel Collector.When you run a direct inference through Galtea, the platform automatically injects a W3Ctraceparent header into the request to your endpoint. If your service propagates that header internally (as most OTel-instrumented services do), all resulting spans are automatically correlated with the corresponding Galtea inference result. No manual ID passing, no SDK calls for trace creation, just your existing instrumentation working end to end.For services that are not OTel-instrumented, the Galtea SDK’s @trace decorator and start_trace context manager continue to work as before. A step-by-step guide covering all tracing methods, including W3C trace context propagation, is available in the documentation.Structured JSON Inputs
Test case inputs and session contexts now support structured JSON objects in addition to plain text strings. This means you can store complex, multi-field payloads directly in the platform database. Existing string-based inputs continue to work unchanged.In upcoming releases, structured JSON inputs will be more broadly supported across the platform, including endpoint connections and SDK methods.SDK: Synchronous Evaluation Polling
The Python SDK now includesevaluations.wait_for(), which polls one or more evaluations until they leave PENDING status and raises a TimeoutError if the deadline is exceeded. This simplifies scripts and CI pipelines that need to block until results are ready.Normalized JSON Field Match Metric
The JSON Field Match metric now supports a normalized comparison mode that performs case-insensitive and accent-insensitive matching within field values. For example,"Sí" and "SI" are treated as equal. This is useful when evaluating outputs in multilingual contexts or when casing differences are not meaningful.Editable Test Metadata
Tests now include an editablemetadata field where you can store free-form notes or custom tracking information. The field is available in the dashboard, the API, and the SDK, giving you a lightweight way to annotate tests without modifying their structure.Platform Improvements
- Time-of-day filters: Date filters throughout the dashboard now support hour, minute, and second precision, so you can narrow results to a specific time window rather than a full day.
- Augmented column: The test cases table now includes an Augmented column indicating which test cases were generated through AI augmentation.
- Double-click prevention: The Save Accepted button during metric generation no longer triggers duplicate submissions if clicked rapidly.
- Pagination stability: Table pagination no longer resets unexpectedly when navigating between pages via URL.
- Show Details navigation: The Show Details action in entity tables now opens directly to the details tab.
Specifications in Analytics
The analytics dashboard now includes a dedicated Specifications section that aggregates evaluation scores per specification, giving you a clear picture of how each policy area is performing across versions. Specification-level radar charts let you compare coverage at a glance, and clicking any specification label filters the entire dashboard to show only metrics linked to that specification.Version comparison charts have also been improved with click-to-highlight and double-click-to-navigate directly to the evaluations list.Test Case CSV Export
You can now export your test cases as CSV files directly from the dashboard. Select specific test cases or export all visible ones — the platform generates a CSV with 22 columns covering input, expected output, strategy, language, scores, and more, then provides a direct download link. Use exported data for offline review, spreadsheet analysis, or integration with external workflows.Documentation and Video Guides
The documentation has been significantly expanded with new video walkthroughs and reference pages:- Video guides now cover specifications, versions, traces, human evaluation, endpoint connection setup, and AI-powered specification generation — each embedded directly in the relevant docs page
- A new Writing Specifications guide covers how to define specifications for your product — behavioral expectations, policies, and rules that drive metric generation and test creation
- New reference pages for evaluation types, evaluation parameters, and endpoint connection configuration provide deeper detail on evaluation setup
- A runnable evaluation demo covers endpoint-based, specification-filtered, and local-agent evaluation modes
Platform Improvements
- JSON field matching metric: A new deterministic metric that compares top-level JSON fields between actual and expected output, returning the ratio of matching fields as the score. Supports lenient parsing of actual output wrapped in markdown code fences or surrounding text.
- Endpoint connection validation: Before running an evaluation, the platform now validates that all required endpoint connections on the version still exist — preventing runtime failures from deleted connections.
- Credit warning badge: A visual indicator in the sidebar shows your organization’s credit status — amber when credits are running low, red when exhausted — so you always know where you stand before launching evaluations.
- Retry defaults: New endpoint connections default to retries enabled with 3 attempts and exponential backoff, improving reliability out of the box.
- Sidebar overlay: The navigation sidebar now overlays content on hover instead of pushing the page layout, reducing visual disruption while browsing.
- Form and table polish: Clearer validation errors when switching to AI generation mode, a clear button for optional form fields, improved column width handling in tables, better form field reset behavior, and more consistent UI across the dashboard.
Improved Golden Dataset Generation
The golden dataset generation pipeline has been redesigned. You can expect more diverse questions, more accurate source attribution, and fewer duplicated or low-quality entries in your generated test cases.Platform Improvements
- Automatic text truncation: Long text in table cells across the dashboard is now automatically clamped to two rows. Click any truncated cell to expand and view the full content rendered as Markdown. This replaces inconsistent truncation behavior across tables and keeps the interface clean without hiding information.
- AI metric saving fix: An issue that prevented non-admin users from saving AI-generated metric candidates has been resolved. All team members can now review and accept metric suggestions without encountering permission errors.
- Evaluation link fix: The evaluation link generated in the SDK code snippets now navigates to the correct tab, ensuring you land on the evaluations view as expected.
AI-Powered Metric Creation
You can now generate evaluation metrics directly from your product’s specifications. Select one or more policy specifications, and the AI analyzes your product description and specification rules to produce ready-to-use metric candidates — complete with judge prompt, evaluation parameters, tags, and evaluator model. You review each candidate, edit anything that needs adjusting, and save the ones you want. Saved metrics are automatically linked back to their source specification.Learn more about the full workflow in the AI Metric Generation documentation.Data Catalogs for Behavior Tests
Behavior tests can now be grounded in real data. When creating a generated behavior test, you can upload a Data Catalog — a file containing real values like names, IDs, amounts, and dates. The scenario generator injects catalog values into the generated test cases, producing scenarios that reflect your actual data distribution instead of relying on fictional placeholders.Simplified Test Creation
The test creation form has been reorganized to reduce visual complexity. Optional fields — Custom User Focus, Data Catalog, and Language — are now grouped behind an Advanced Options toggle, keeping the main form focused on required fields and essential choices.Test Your Endpoint Connection
You can now test your endpoint connection directly from the creation and edit forms. A dedicated test panel sends a sample request to your endpoint and displays the raw response, so you can verify connectivity and response format before saving.Run Evaluations from Any Level
The evaluation run dialog now supports two new entry points: you can trigger a run from a specific test or from an individual test case, in addition to the existing version, session, and inference result levels. The dialog title, description, and available options adapt to the context you start from. As part of this change, the action has been renamed from “Run Inference” to “Run Evaluation” across the dashboard to better reflect what it does.SDK: Full Conversation Access in Custom Metrics
TheCustomScoreEvaluationMetric.measure() method now receives an optional inference_results parameter containing all conversation turns for the current session. This lets custom metrics reason over the full conversation history — not just the latest turn — when computing scores. This also fixes an issue where evaluating sessions with custom metrics only fetched the most recent turn, making multi-turn custom metric evaluation unreliable. See the custom metrics tutorial for usage details.Platform Improvements
- Failed turn tracking: The evaluation engine now detects which conversation turns failed during a simulated conversation and records them in a new
failedTurnsfield on each evaluation. Failed turns are visible in the evaluation detail view and included in CSV exports, making it easier to pinpoint where multi-turn conversations break down. - Clearer metric validation errors: When a metric is missing required evaluation parameters, the error message now lists exactly which parameters are absent — instead of the generic “at least one is missing” message.
- Inference error visibility: When a direct inference call results in an error, the error text is now displayed directly in the inference result view.
- Unified form components: AI-assisted creation forms and table selection dialogs across the dashboard now share a consistent design and interaction model. This also fixes an issue where discarding all AI-generated metric candidates left the review step in a dead end with no navigation options.
Human Evaluators
You can now bring humans into the evaluation loop. Human evaluation lets your team members review and annotate inference results directly from the Galtea platform — alongside the automated LLM-as-a-judge scores you already rely on.To get started, create a metric with the evaluation type set to Human, assign it to the relevant user group, and invite your evaluators. Each evaluator sees only the inference results assigned to their group and can submit scores through a dedicated annotation interface.User Groups
User Groups are the organizational layer behind human evaluation. A user group links a set of team members to specific metrics, so you can control who evaluates what. You can manage user groups from the dashboard or programmatically through the User Group Service in the SDK.AI-Assisted Specification Configuration
Specification creation is now smarter. AI-assisted configuration has been extended to all specification types — not just policies. When you create a new specification, the system analyzes your description and suggests the test type, test variant, and specification type automatically. You can accept the suggestions with a single click or adjust them before saving.For Policy specifications specifically, the AI also suggests which test type best validates the rule you are defining — helping you choose betweenACCURACY, SECURITY, and BEHAVIOR based on the policy’s intent.You can manage specifications from the dashboard or via the Specification Service in the SDK.Inference Monitoring and Tracing
Direct inference sessions now surface more visibility into what is happening during evaluation runs:- Session status: Each inference session displays a status indicator —
COMPLETED,PENDING, orFAILED— so you can quickly identify which sessions need attention. - Failed inference visibility: When a direct inference call fails, the error is now surfaced directly in the session view, rather than being silently logged.
- Distributed trace collection: Remote agents can now send traces back to Galtea automatically. Pass the
AgentInputobject to your agent handler, and all@trace-decorated calls are linked to the correct session. See Tracing Agent Operations for the full setup guide.
Metrics and Evaluation
- Flexible text metric validation: Text-based metrics no longer require a strict
answerkey in the response payload, making it easier to integrate custom output formats. - Version endpoint validation: Starting an evaluation now checks that the selected version has a valid endpoint connection configured, catching misconfigurations before the run begins.
- SDK fix — session custom metrics: An issue where
actual_outputwas alwaysNonewhen evaluating sessions with custom metrics has been resolved. - Filter tests by version: You can now filter the test list by the selected version, making it faster to find the tests relevant to your current evaluation run.
- Specification-aware validation: Products are now enriched with their linked specifications before evaluation validation runs, ensuring the evaluation engine has the full context it needs.
Platform Improvements
- Report terminology: Automated reports now use clearer dimension names — Accuracy instead of “Quality” and Security & Safety instead of “Red Teaming” — matching the terminology you see in the dashboard.
- Report visualization: Metric comparison charts now support multi-page groups, so products with more than six metrics per group render correctly without truncation.
- Evaluation run form: The evaluation run form layout has been reorganized for a cleaner workflow when configuring new runs.
- Improved augmentation: The data augmentation pipeline has received quality improvements to produce more diverse and representative test case variations.
- SDK improvements: The SDK now correctly supports JSON request bodies in HTTP DELETE operations and properly escapes JSON string values in inference input templates.
Documentation
A new Human Evaluation tutorial walks you through the full workflow — from creating a human metric and setting up user groups to annotating results in the platform. The tutorial includes step-by-step screenshots showing the metric creation form, the evaluator sidebar, and the annotation interface.Specifications
Specifications are now live — a structured way to define and test the behavioral expectations of your product.A Specification represents a single, testable claim about what your product should or should not do. Each specification has a type that classifies the expectation:- Capability — a core function the product can perform
- Inability — something the product fundamentally cannot do, regardless of user input
- Policy — a rule the product must follow, such as refusing certain requests or always adding a disclaimer
ACCURACY, SECURITY, or BEHAVIOR) that determines how the specification is evaluated.Specifications replace the legacy free-text Capabilities, Inabilities, and Policies fields that used to live on the Product form. Your behavioral expectations are now discrete, individually traceable, and can be linked directly to the metrics that validate them.You can manage specifications from the dashboard or via the Specification Service in the SDK.AI-Assisted Configuration
Specification creation is AI-assisted. With a single click, the system suggests the specification type, test type, and test variant based on your description — so you can focus on defining what your product should do, not on configuring how it gets tested.Improved Report Generation
The automated report generation pipeline has received a set of quality upgrades:- Realism analysis: Before generating narrative content, the engine now runs a realism check on the underlying data to ensure summaries reflect actual conditions rather than statistical artifacts.
- Pattern analysis: A new pattern analysis step examines the diversity and distribution of your test results, informing the structure of each report section. You can toggle this per generation run.
Better Data Augmentation
Data Augmentation now uses a dedicated diversity model alongside the main generation pass. Instead of producing variations that converge on the same patterns, augmented test cases are explicitly steered toward different scenarios, phrasings, and edge cases — giving you broader coverage from the same seed data.A demo video has also been added to the Data Augmentation documentation to walk you through the full workflow.Simplified Metrics
The Metric entity has been streamlined. Legacy fields — includingcriteria, evaluation_steps, test_type, user_persona, and stopping_criterias — have been removed from the metric creation form. These were carry-overs from an older architecture that are no longer needed. If you were passing any of these fields through the SDK, you can safely remove them.Platform Improvements
- 403 Forbidden page: Accessing a route without the required permissions now shows a clear, dedicated error page with an explanatory message, rather than a silent redirect or blank screen.
- Sorting fixes: Table columns that do not support server-side sorting no longer display a sort indicator, removing a common source of confusion in large result sets.
- Evaluation prompts: User prompts in evaluations are now rendered through Jinja2, enabling richer template-based customization.
Trace Collection During Platform Simulations
When running evaluations via Direct Inference, you can now collect traces from your endpoint and link them back to Galtea automatically.Add the new{{ inference_result_id }} placeholder to your Endpoint Connection input template. Your handler receives the ID, passes it to the SDK’s set_context, and all @trace-decorated calls made during that request are automatically linked to the correct inference result in Galtea. See Collecting Traces During Direct Inference for the full walkthrough.Documentation and Examples
Demo videos are now embedded throughout the documentation — covering test generation, Accuracy tests, Security & Safety tests, Behavior tests, endpoint connection setup, and data augmentation — so you can see each workflow in action without leaving the docs.Automated Report Generation
You can now export your analytics data as a comprehensive PDF report directly from the Galtea dashboard. The report is generated automatically and includes AI-written summaries for each section — covering scope, methodology, product evaluation, and conclusions — alongside dashboard-style visualizations.One-Click Test Case Augmentation
Scaling your Test Cases just got much simpler. If you have a small base of known-good test cases, you can now augment them with a single click directly from the dashboard. Galtea will automatically generate additional test cases based on your existing ones, letting you quickly expand coverage without manually writing each entry.New Quality Test Tasks
We have expanded the Quality Test configuration with two new task types: Correction and Other. When selecting Other, you can provide a custom task description to better classify your tests. This makes it easier to define quality evaluations that go beyond the predefined categories.New Tutorial: Direct Inferences and Evaluations from the Platform
A new step-by-step guide walks you through the full workflow of running inferences and evaluations directly from the Galtea dashboard — no SDK code required. It covers creating an Endpoint Connection, attaching it to a Version, running tests, and reviewing results.Performance and Reliability
- Faster Evaluations: The evaluation pipeline has been optimized to reduce redundant database lookups. Pre-fetched entities are reused across batch operations, validation order has been improved, and retry logic now groups evaluations by session for significantly faster batch processing.
- Improved Error Messages: The IOU and Spatial Match metrics now return clearer error messages specifying the expected JSON formats for bounding box inputs.
- Jinja2 Template Validation Fix: The conversation simulator template validator no longer raises false positives for valid Jinja2 for-loop patterns in JSON templates, such as the common OpenAI-compatible message format.
- Dashboard Polish: Several UI improvements including better dark mode contrast, refined Pill component styling, and a smarter toast notification system that reduces interruptions for frequent users.
SDK & Simulation Upgrades
We have significantly improved the flexibility of the Galtea SDK to better fit your existing workflows.-
Unified Simulation Method:
The
simulator.simulatemethod has been upgraded to handle both single-turn and multi-turn simulations. You no longer need to switch methods based on the test case complexity. -
Simplified Agent Integration:
We’ve made the
agentproperty much more flexible. You can now pass methods with various signatures directly (including async generators), removing the strict requirement to wrap everything in a specific Agent class.
New Platform Features
AI-Powered Product Definition
Bootstrapping a new product context is now faster than ever. You can upload a set of files (documentation, knowledge base, etc.), and Galtea will use AI to automatically generate a comprehensive Product Definition for you.Advanced Endpoint Connections
For more complex conversational agents, we have refined how connections are established. You can now configure distinct behaviors for:- The Initialization of the conversation.
- The Conversation messages.
- The Finalization of the conversation.
Single Inference Evaluation
You can now evaluate a single inference result directly from the platform UI. This is perfect for spot-checking model behavior without running a full simulation suite.Reliability and UX Improvements
- Graceful Metric Handling: Previously, using a metric incompatible with a specific Test Type would throw an error. Now, the system intelligently marks these metrics as “skipped” in the results, allowing the rest of your evaluation to proceed uninterrupted.
- Enhanced Custom Metrics: We have deprecated all DeepEval metrics in favor of our own Custom Metrics architecture. This transition significantly enhances the reliability, speed, and flexibility of your evaluations.
- Test Creation Experience: The Test creation form has been polished for better usability, making it easier to define and organize your test cases.
- Platform Stability: General improvements to platform stability and reliability to ensure a smoother experience during heavy load.
Evaluate Single-Turn Interactions
You can now run evaluations on specific single turns (even when directly creating an Inference Result), rather than being restricted to evaluating full threads or sessions. This granularity allows for more precise analysis, enabling you to pinpoint and score individual exchanges within a larger conversation context without the noise of the surrounding dialogue.Bulk Actions in Tables
We’ve enhanced table functionality across the dashboard to improve your productivity. You can now select multiple rows to perform actions in bulk, such as deleting multiple Test Cases or cleaning up old Sessions simultaneously.
Stability & Performance
- Metric Stability: We’ve deployed updates to improve the consistency and reliability of several core metrics.
- Platform Resilience: Significant backend optimizations have been implemented to ensure platform stability and maintain low latency, even under periods of high load.
Run Inferences and Evaluations from the Platform
You can now execute inferences and run evaluations directly from the Galtea dashboard. This new capability allows you to quickly test your Product Versions and validate performance without writing a single line of code or switching to your IDE.This streamlined workflow is perfect for:- Quick sanity checks on new model versions.
- Running specific test cases ad-hoc.
- Validating changes instantly before full-scale testing.

General Improvements
- Bug Fixes & UX Polish: We’ve addressed various minor bugs and refined the user interface to provide a smoother, more stable experience across the platform.
Standardized Tracing with OpenTelemetry
Galtea Traces have been upgraded to use the OpenTelemetry (OTel) standard. This major infrastructure update aligns our tracing capabilities with industry standards, paving the way for future seamless integrations with your existing observability tools and monitoring stacks. Read more about traces.New Models Available
We have expanded the selection of models available for evaluating your products. You can now leverage the latest capabilities of:- Claude-Sonnet-4.5
- GPT-5.2, GPT-5.1, GPT-5, and GPT-5-mini
Revamped Onboarding & Development Modes
We’ve improved the onboarding experience to better match your workflow:- Use-Case Based Onboarding: The setup process is now split between two clear paths: evaluating a product during Development or monitoring a product in Production with real users.
- Platform View Modes: You can now select your preferred view mode in the platform: Development or Production. This automatically filters the UI to remove unnecessary options, keeping your workspace clean and focused on the task at hand.

General Improvements
- Enhanced Forms: We’ve standardized and improved forms across the platform for better usability and a more consistent experience.
AI-Assisted Product Onboarding
Getting started with Galtea got even simpler. We’ve improved the Product creation flow by simplifying the integration of AI assistance directly into the product creation form. The system helps you draft comprehensive product descriptions, capabilities, and security boundaries, ensuring your product context is perfectly optimized for our test generation engine from day one.
Enhanced Inference Result Visualization
We have revamped how Inference Result metrics are visualized. The new layout provides a clearer breakdown of performance data, making it easier to correlate specific input/output pairs with their respective scores. This improvement allows for quicker diagnosis of issues within specific conversation turns.
Documentation Refresh
We have reorganized our documentation to improve discoverability and ease of use. The structure is now streamlined into clearer categories, making it easier to find SDK references, conceptual guides, and tutorials. Check out here the new docs structure here.General Improvements & Bug Fixes
- General Bug Fixes: Addressed various minor issues to improve platform stability and performance.
- UI Polish: Minor visual updates across the dashboard for a more consistent user experience.
Enhanced Product Creation Experience
We have completely revamped the onboarding flow for new products. A new, intuitive form is now available to help you create AI-based products faster and more efficiently. This update streamlines the initial setup configuration, improving the overall user experience when onboarding new agents or LLM apps.
Evaluating Tool Usage
As agents become more autonomous, validating how they use tools is just as important as the final answer. We have introduced new capabilities to strictly evaluate tool calls:New Property in Test Cases
Test cases now accept a new property:expected_tools. This allows you to define exactly which tools an agent should invoke during a specific test scenario. Read more about Test Case structure.New Evaluation Parameters
To support this validation, the evaluation process now accepts two specific parameters regarding tool usage:tools_used: The actual list of tools invoked by the model.expected_tools: The ground truth list of tools that should have been used.
New Metric: Tool Correctness
We have added a specialized metric to our library: Tool Correctness.This metric automatically compares thetools_used against the expected_tools to determine if the agent selected the right functions to solve the user’s problem. This is critical for ensuring reliability in agentic workflows. See metric details.
Full Visibility with Agent Tracing
You can now add Traces to your Sessions and Inference Results. This unlocks deep observability into your AI agents, allowing you to understand exactly what they are doing at each step of execution—not just the final output.Traces capture internal operations such as:- Tool Calls: API requests, calculations, or data fetching.
- Retrieval Steps: Vector database searches and RAG context retrieval.
- Chain Orchestration: Internal routing and decision-making logic.
- LLM Invocations: Prompts sent to underlying models.

Granular Credit Consumption and Transparency
We’ve updated our credit system to provide more granularity and transparency. You can now see exactly where your credits are being consumed, giving you detailed visibility into your usage patterns and helping you manage resources more effectively.Platform Robustness and Security
We’ve improved the overall robustness of the platform to ensure higher uptime and reliability. Additionally, we have increased safety by implementing rate limiting on our authentication endpoints, providing better protection against abuse and unauthorized access attempts.Improved SDK File Validation
We’ve enhanced file validation within the SDK. When uploading files—for example, when creating a Test usingtest_file_path or ground_truth_file_path—you will now receive stricter validation and clearer, more descriptive error messages to help you debug issues faster.Simplified and More Powerful Metric Creation with Partial Prompts
We’ve upgraded how custom judge Metrics are created by introducing a new Partial Prompt method. This approach simplifies the process by letting you focus on the core evaluation logic while Galtea handles the final prompt construction. This not only makes creating custom metrics faster but also significantly increases the quality and consistency of the evaluation results. Learn more about the new method.Clearer Pass/Fail with Binarized Metric Scores
To provide more decisive validation, we’ve updated three of our key metrics to return a clear binary score (0 for fail, 1 for pass). This change eliminates ambiguous 0.5 scores, making it easier to determine success or failure for critical test cases. The updated metrics are:More Robust Quality Test Generation
When creating Quality Tests from the dashboard, we now validate your uploaded files to ensure they are in the expected format. This proactive check helps prevent errors during test generation, leading to a smoother and more reliable workflow.Platform Enhancements
We’re always working to improve the core platform experience. This week’s updates include:- Improved Performance: We’ve optimized our underlying LLM usage for better concurrency, scalability, and robustness, resulting in a faster and more reliable platform.
- Enhanced Email Validation: The process for validating user emails has been made more robust to ensure better security and deliverability.
Flexible Sign-In with Google, GitHub, and GitLab
You can now sign in to Galtea using your Google, GitHub, or GitLab accounts! This makes accessing your workspace faster and more secure, providing a seamless single sign-on (SSO) experience alongside our traditional email and password login.
Simplified Metric Versioning with “Legacy” Tag
We’ve introduced a new way to handle updated Metrics. Galtea metrics might be marked as “Legacy,” allowing us to improve them over time with the same name. This approach simplifies versioning by removing the need to embed version numbers in metric names, ensuring that your historical evaluations remain linked to the correct metric version.Test Generation Transparency and Enhancements
We’re bringing more transparency and efficiency to our test generation process:- See What’s Under the Hood: The models used to generate a Test are now listed in the test’s details section, providing clearer insight into how your test cases were created.
- Faster Test Generation: We’ve significantly sped up the processing of multiple files within ZIP archives when generating tests from a knowledge base.
- Smarter Test Case Handling: The engine for generating Quality Tests now handles the
max_test_casesparameter more effectively.
Fixes & Improvements
- Fixed a bug where file names from ZIP archives were not displayed correctly in the UI.
- Resolved an issue that prevented the processing of ZIP files containing two files with the same name (e.g.,
folder1/file.txtandfolder2/file.txt).
A Fresh New Look for the Dashboard
We’ve polished the dashboard with a subtle but significant UI refresh. Expect updated colors, clearer icons, and more consistent dialogs designed to create a smoother, more intuitive, and visually consistent workflow.Enhanced Security and Performance
We’ve bolstered the platform’s security with improved user authentication mechanisms to better protect your data. Alongside this, we’ve enhanced how we handle concurrent requests, leading to better performance and reliability, especially during high-traffic periods.Smarter & Faster Red Teaming Generation
Creating Red Teaming tests is now better than ever. We’ve improved language control during generation for more robust and relevant tests, even when no specific language is provided. Plus, we’ve optimized the generation process for significantly faster test creation, allowing you to build robust security evaluations more efficiently.Introducing Partial Prompts: Create Custom Metrics Faster
We’re making it easier than ever to create custom judge metrics with our new Partial Prompt evaluation type. Now, you can focus solely on defining your evaluation criteria and simply select the data parameters you need—likeinput or retrieval_context. Galtea automatically constructs the full, correctly formatted judge prompt for you, saving you time and reducing complexity.This new approach streamlines the creation of powerful, tailored Metrics, allowing you to concentrate on what matters most: the evaluation logic itself.
Quality Generator with Improved Language Handling
The LLM language confusion is now patched for the quality generator. Languages are formally detected for the whole document, and test cases are ensured to be generated with the correct language. This ensures more accurate and contextually appropriate test case generation across different language inputs.Improved Long Context Processing for Quality Test Cases
Our quality generator now intelligently processes long documents, ensuring optimal context is provided for the best quality generation of test cases. This enhancement streamlines the evaluation process and improves the reliability of generated test cases for comprehensive quality assessments.Platform Experience Enhancements
We’ve also rolled out several quality-of-life improvements and bug fixes for a smoother experience. A notable update is that tables across the platform now refresh instantly after you delete an item with a right-click, making data management more fluid and intuitive.Define Product Behavior with Policies
Products now accept a set of policies describing how they should behave in specific situations. This allows you to set explicit rules, such as requiring a disclaimer for financial advice or defining standard responses for out-of-scope questions, to ensure consistent and safe model behavior.Advanced Red Teaming Strategies
We’ve added five new, sophisticated red teaming strategies to help you uncover more complex vulnerabilities. These strategies use your Product Description to create highly contextual and evasive prompts:- Persuasive Content: Disguises malicious requests as legitimate business tasks.
- Creative Writing: Reframes harmful prompts as creative exercises.
- Data Analysis: Hides malicious intent within analytical or data-generation tasks.
- Bait and Switch: Lowers model defenses with benign queries before introducing the adversarial prompt.
- Empathetic Framing: Uses emotional manipulation to pressure the model into unsafe compliance.
Smarter Test Generation Engine
Our test generation capabilities have been significantly upgraded for both Quality and Red Teaming tests:- New Quality Test Engine: The engine for generating Quality Tests has been rebuilt to produce higher-quality test cases. It can now also incorporate and validate the new product policies you define.
- Faster Red Teaming Generation: The generation process for Red Teaming Tests is now more reliable and significantly faster, allowing you to build robust security evaluations more efficiently.
Platform Usability and Performance
We’ve rolled out several enhancements to improve your workflow and the platform’s reliability:- Deletion Confirmations: To prevent accidental data loss, the platform will now ask for confirmation before deleting key entities like Test Cases, Evaluations, and Versions.
- Improved Table Experience: The performance and user experience of data tables and their filters have been optimized for a smoother, more responsive interface.
Platform Simplification and SDK v3.0
We’ve undertaken a major simplification of our core concepts to make the platform more intuitive.Evaluation Tasks are now simply Evaluations, and Metric Types have been renamed to Metrics. The old parent Evaluations entity has been removed entirely. These changes streamline the workflow and clarify the relationship between different parts of Galtea.To support this, we’ve released a new major version of our SDK.New Conversational Metrics
We’re excited to introduce two new metrics designed specifically for evaluating conversational AI:- User Objective Accomplished: Evaluates whether the user’s stated goal was successfully and correctly achieved during the conversation.
- User Satisfaction: Assesses the user’s overall experience, focusing on efficiency and sentiment, to gauge their satisfaction with the interaction.
Enhanced Test Case Feedback and Management
Improving test quality is now a more collaborative process. When upvoting or downvoting a Test Case, you can now add auser_score_reason to provide valuable context for your feedback.Additionally, you can now filter test cases by their score directly via the SDK using the user_score parameter in the test_cases.list() method.Dashboard and SDK Usability Improvements
We’ve rolled out several updates to make your workflow smoother and more efficient:- Improved Dashboard Navigation: Navigating between related entities like Tests, Test Cases, and Evaluations is now more intuitive. We’ve also adjusted table interactions—you can now single-click a row to select and copy text without navigating away. To see an entity’s details, simply right-click the row.
- Efficient Batch Fetching in SDK: The SDK now allows you to fetch objects by providing a list of IDs (e.g., fetching all Test Cases for a list of Test IDs at once), significantly improving usability for batch operations.
General Improvements
This release also includes numerous performance optimizations and minor UI/UX enhancements across the entire platform to provide a faster and more polished experience.Create Conversational Scenarios from Your Quality Tests
You can now generate comprehensive Scenario Based Tests directly from your existing Quality Tests. This powerful feature allows you to transform your single-turn, gold-standard test cases into realistic, multi-turn conversational scenarios, significantly accelerating the process of evaluating your AI’s dialogue capabilities.New Deterministic Metric: URL Validation
We’ve added the URL Validation metric to our deterministic evaluation suite. It ensures that all URLs in your model’s output are safe, properly formatted with HTTPS, and resolvable. It includes strict validation and SSRF protection, making it essential for any application that generates external links.Platform Enhancements and Performance Boost
We’ve rolled out major performance improvements across the platform, with a special focus on the analytics views, making them faster and more responsive. This update also includes several minor visual fixes, such as ensuring icons render correctly in different themes, to provide a smoother and more polished user experience.Consistent Sorting and Enhanced Navigation
We’ve made significant usability improvements to the platform. Table sorting is now more consistent and persistent; you can navigate away, refresh the page, and your chosen sort order will remain, ensuring a smoother workflow. Additionally, navigating from an Evaluation is easier than ever. You can now right-click on any evaluation in the table or use the dropdown menu in the details page to jump directly to the related Test Case or Session.
Upvote and Downvote Test Cases
To help teams better curate their test suites, we’ve introduced a voting system for Test Cases. You can now upvote or downvote test cases directly from the Dashboard, providing a quick feedback loop on test case quality and relevance.
SAML SSO Authentication
Organizations can now enhance their security by configuring SAML SSO for authentication. This allows for seamless and secure access to the Galtea platform through your existing identity provider.Improved Misuse Resilience Metric
The Misuse Resilience metric has been enhanced to accept the full product context. This allows for a more accurate and comprehensive evaluation of your model’s ability to resist misuse by leveraging a deeper understanding of your product’s intended capabilities and boundaries.New Analytics Filter
The analytics page now includes a filter for Test Case language. This allows you to narrow down your analysis and gain more precise insights into the performance of your multilingual models.Confidence Scores for Generated Test Cases
We’re introducing confidence scores for all generated Test Cases. This new feature provides a clear indicator of the quality and reliability of each test case, helping you better understand your test suites and prioritize human review efforts. Higher scores indicate greater confidence in the test case’s relevance and accuracy.Simplified and More Flexible Metric Creation
Creating Metrics is now more intuitive and flexible. Metrics are now directly linked to a specific test category (QUALITY, RED_TEAMING, or SCENARIOS), which simplifies the creation process by tailoring the available parameters to the relevant test category. This change makes it easier to define metrics that are perfectly aligned with your evaluation goals. See the updated creation guide for more details.Custom Judges for Conversational Evaluation
You can now create your own Custom Judges specifically for evaluating multi-turn conversations. This powerful feature allows you to define complex, stateful evaluation logic that assesses the entire dialogue, enabling you to measure nuanced aspects like task completion, context retention, and persona adherence across multiple turns. Learn more in our guide to evaluating conversations.Conversation Simulator Enhancements
We’ve added more control and realism to the Conversation Simulator:- Agent-First Interactions: The
simulatemethod now includes anagent_goes_firstparameter, allowing you to test scenarios where the agent initiates the conversation. See the SDK docs. - Selectable Conversation Styles: When creating a Scenario Based Test, you can now choose a conversation style (
writtenorspoken). This influences the tone and formality of the simulated user’s dialogue, enabling more realistic testing. This is available under thestrategiesparameter in the test creation method.
More Realistic Conversation Simulations
Our Conversation Simulator has been significantly upgraded to generate more realistic and human-like interactions. The simulator’s evaluation of stopping criteria is now more precise, ensuring that your multi-turn dialogue tests conclude under the correct conditions and provide more accurate insights into your AI’s conversational abilities.Additionally, conversations generated during simulations are now linked to the specific product version they were created with, allowing for better tracking and traceability.Bulk Document Uploads for Test Generation
You can now upload multiple files at once using ZIP archives to generate comprehensive test suites. This feature streamlines the process of creating tests from large document collections, saving you time and effort when building out your evaluation scenarios. For more details, see the test creation documentation.Streamlined Onboarding Experience
Getting started with a new product is now faster than ever. When you register a new product in the Galtea dashboard, an initial version is now created automatically. This simplification streamlines the setup process and helps you get to your first evaluation more quickly.Enhanced Document Processing
We’ve improved our backend for processing documents used in test generation. This enhancement leads to more accurate and relevant test cases being created from your knowledge bases, improving the overall quality of your evaluations.Fresh New Look: Galtea Rebranding is Live!
We’re excited to unveil Galtea’s complete visual transformation! Our new branding includes a refreshed logo, updated slogan, and a modern look across the entire platform. Experience the new design on our website and explore the updated dashboard interface. This rebrand reflects our continued commitment to providing a more intuitive and visually appealing user experience.
Enhanced Analytics with Improved Version Comparisons
The Analytics views have received significant improvements to make data analysis more efficient and user-friendly. Key enhancements include:- Streamlined Chart Comparisons: Charts now support easier side-by-side version comparisons, helping you quickly identify performance differences across iterations.
- Optimized Filter Layout: Filters no longer occupy valuable screen real estate, giving you more space to focus on your analytics data and insights.
- Improved Visual Clarity: Enhanced data presentation makes it easier to interpret results and make informed decisions about your AI systems.
Strengthened Authentication and Security
We’ve implemented more secure and robust authentication mechanisms across the platform. These behind-the-scenes security enhancements provide better protection for your data and ensure reliable access to your Galtea workspace, giving you peace of mind when working with sensitive AI evaluation data.Advanced PDF Knowledge Extraction
Our document processing capabilities have been significantly improved for PDF files. The enhanced extraction algorithms now provide more accurate and comprehensive knowledge retrieval from PDF documents, making it easier to create relevant test cases and evaluation scenarios from your existing documentation and knowledge bases.TestCase Scenarios Creation and Edition via Dashboard
You can now create and edit TestCase Scenarios directly through the Dashboard interface. This streamlined workflow allows you to define complex multi-turn conversation scenarios without leaving the platform, making it easier to set up comprehensive testing workflows for your conversational AI systems.Enhanced TestCase Dashboard with Test-Specific Columns
We’ve improved the TestCase Dashboard with smarter table displays that now only show columns relevant to each specific test. This reduces visual clutter and makes it easier to focus on the information that matters most for your particular testing scenario, whether you’re working with single-turn evaluations, conversation scenarios, or other tests.New Judge Template Selector for Judge Metrics
When creating Judge metrics, you can now select from pre-built judge templates to accelerate your metric setup process. This feature provides a starting point for common evaluation patterns while still allowing full customization of your evaluation prompts and scoring logic.Infrastructure Improvements for Event Resilience
We’ve enhanced our event handling infrastructure to provide better resilience against unexpected system events. These improvements help ensure that your tests and evaluations are preserved and continue running smoothly, even during system maintenance or unexpected interruptions.Create Custom Judges via Your Own Prompts
You can now define Custom Judge metrics by crafting your own evaluation prompts during metric creation. This allows you to encode your domain-specific rubrics, product constraints, or behavioral guidelines directly into the metric—giving you precise control over how LLM outputs are assessed. Simply write your prompt, specify the scoring logic, and Galtea will leverage LLM-as-a-judge techniques to evaluate outputs according to your standards.New Deterministic Metrics
Four new deterministic metrics are now available:- Text Similarity: Quantifies how closely two texts resemble each other using character-level fuzzy matching.
- Text Match: Checks if generated text is sufficiently similar to a reference string, returning a pass/fail based on a threshold.
- Spatial Match: Verifies if a predicted box aligns with a reference box using IoU scoring, producing a pass/fail result.
- IoU (Intersection over Union): Computes the overlap ratio between predicted and reference boxes for alignment and detection tasks.
Dashboard Redesign: Second Iteration
We’ve launched the second iteration of our redesigned dashboard with a refreshed visual language focused on clarity and usability. Key improvements include:- Modern Forms: Forms have been modernized to provide a more intuitive and visually appealing user experience, as well as giving a more professional look to the Dashboard.
Expanded Evaluator Model Support
We’ve added support for more evaluator models to enhance your evaluation capabilities:- Gemini-2.5-Flash: Google’s latest high-performance model optimized for speed and accuracy
- Gemini-2.5-Flash-Lite: A lightweight variant offering faster processing with efficient resource usage
- Gemini-2.0-Flash: Google’s established model providing reliable evaluation performance
Enhanced Conversation Simulation
Testing conversational AI just got more powerful with two major improvements:- Visible Stopping Reasons: You can now see exactly why simulated conversations ended in the dashboard, providing crucial insights into dialogue flow and helping you identify areas for improvement.
- Custom User Persona Definitions: Create highly specific user personas when generating Scenario Based Tests. Define detailed user backgrounds, goals, and behaviors to test how your AI handles diverse user interactions more effectively.
Classic NLP Metrics Now Available
We’ve expanded our metric library with three essential deterministic metrics for precise text evaluation:- BLEU: Measures n-gram overlap between generated and reference text, ideal for machine translation and constrained generation tasks.
- ROUGE: Evaluates summarization quality by measuring the longest common subsequence between candidate and reference summaries.
- METEOR: Assesses translation and paraphrasing by aligning words using exact matches, stems, and synonyms for more nuanced evaluation.
Enhanced Red Teaming with Jailbreak Resilience v2
Security testing gets an upgrade with Jailbreak Resilience v2, an improved version of our jailbreak resistance metric. This enhanced evaluation provides more comprehensive assessment of your model’s ability to resist adversarial prompts and maintain safety boundaries across various attack vectors.Dashboard Redesign: First Iteration
We’ve launched the first iteration of our redesigned dashboard with a refreshed visual language focused on clarity and usability. Key improvements include:- Modern Typography: Cleaner, more readable text throughout the platform
- Refined UI Elements: Updated buttons, cards, and form elements with reduced rounded corners for a more contemporary look
- Streamlined Tables: Enhanced data presentation with improved content layout
- Updated Login Experience: A more polished and user-friendly authentication flow
Improved SDK Documentation
We’ve enhanced our SDK documentation with clearer guidance on defining evaluator models for metrics, making it easier to configure and customize your evaluation workflows.Test Your Chatbots with Simulated Conversations
It is now possible to generate tests that simulate realistic, multi-turn user interactions. Our new Scenario Based Tests allow you to define user personas and goals to evaluate how well your conversational AI handles complex dialogues.This feature is powered by the Conversation Simulator, which programmatically runs these scenarios to test dialogue flow, context handling, and task completion. Get started with our new Simulating User Conversations tutorial.New Red Teaming Metric: Data Leakage
We’ve added the Data Leakage metric to our suite of Red Teaming evaluations. This metric assesses whether your LLM returns content that could contain sensitive information, such as PII, financial data, or proprietary business data. It is crucial for ensuring your applications are secure and privacy-compliant.Enhanced Metric Management
We’ve rolled out several improvements to make metric creation and management more powerful and intuitive:- Link Metrics to Specific Models: You can now associate a Metric with a specific evaluator model (e.g., “GPT-4.1”). This ensures consistency across evaluation runs and allows you to use specialized models for certain metrics.
- Simplified Custom Scoring: We’ve introduced a more streamlined method for defining and calculating scores for your own deterministic metrics using the
CustomScoreEvaluationMetricclass. This makes it easier to integrate your custom, rule-based logic directly into the Galtea workflow. Learn more in our tutorial on evaluating with custom scores.
Support for Larger Inputs and Outputs
To better support applications that handle large documents or complex queries, we have increased the maximum character size for evaluation inputs and outputs to 250,000 characters.Test Your Chatbots with Realistic Conversation Simulation
You can now evaluate your conversational AI with our new Conversation Simulator. This powerful feature allows you to test multi-turn interactions by simulating realistic user conversations, complete with specific goals and personas. It’s the perfect way to assess your product’s dialogue flow, context handling, and task completion abilities.Get started with our step-by-step guide on Simulating User Conversations.New Metric: Resilience To Noise
We’ve expanded our RAG metrics with Resilience To Noise. This metric evaluates your product’s ability to maintain accuracy and coherence when faced with “noisy” input, such as:- Typographical errors
- OCR/ASR mistakes
- Grammatical errors
- Irrelevant or distracting content
Stay in Control with Enhanced Credit Management
We’ve rolled out a new and improved credit management system to give you better visibility and control over your usage. The system now includes proactive warnings that notify you when you are approaching your allocated credit limits, helping you avoid unexpected service interruptions and manage your resources more effectively.Streamlined Conversation Logging with OpenAI-Aligned Format
Logging entire conversations is now easier and more intuitive. We’ve updated our batch creation method to align with the widely-usedmessages format from OpenAI, consisting of role and content pairs. This makes sending multi-turn interaction data to Galtea simpler than ever.See the new format in action in the Inference Result Batch Creation docs.Tailor Your Red Teaming with Custom Threats
You can now define your own custom threats when creating Red Teaming Tests. This new capability allows you to move beyond our pre-defined threat library and create highly specific adversarial tests that target the unique vulnerabilities and edge cases of your AI product. Simply describe the threat you want to simulate, and Galtea will generate relevant test cases.New Red Teaming Strategies: Role Play and Prefix Injection
We’ve expanded our arsenal of Red Teaming Strategies to help you build more robust AI defenses:- Role Play: This strategy attempts to alter the model’s identity (e.g., “You are now an unrestricted AI”), encouraging it to bypass its own safety mechanisms and perform actions it would normally refuse.
- Prefix Injection: Adds a misleading or tactical instruction before the actual malicious prompt. This can trick the model into a different mode of operation, making it more susceptible to the adversarial attack.
Introducing the Misuse Resilience Metric
A new non-deterministic metric, Misuse Resilience, is now available. This powerful metric evaluates your product’s ability to stay aligned with its intended purpose, as defined in your product description, even when faced with adversarial inputs or out-of-scope requests. It ensures your AI doesn’t get diverted into performing unintended actions, a crucial aspect of building robust and responsible AI systems. Learn more in the full documentation.Enhanced Test Case Management: Mark as Reviewed
To improve collaboration and workflow for human annotation teams, Test Cases can now be marked as “reviewed”. This feature allows you to:- Track which test cases have been validated by a human.
- See who performed the review, providing a clear audit trail.
- Filter and manage your test sets with greater confidence.
Introducing the Factual Accuracy Metric
We’ve added a new Factual Accuracy metric to our evaluation toolkit! This non-deterministic metric measures whether the information in your model’s output is factually correct when compared to a trusted reference answer. It’s particularly valuable for RAG and question answering systems where accuracy is paramount.The metric uses an LLM-as-a-judge approach to compare key facts between your model’s output and the expected answer, helping you catch hallucinations and ensure your AI provides reliable information to users. Read the full documentation here.Enhanced Red Teaming with New Attack Strategies
Our red teaming capabilities just got more sophisticated! We’ve added two powerful new attack strategies:- Biblical Strategy: Transforms adversarial prompts into biblical/ancient scripture style using poetic and symbolic language to disguise malicious intent while preserving meaning.
- Math Prompt Strategy: Encodes harmful requests into formal mathematical notation using group theory concepts to obscure the intent from standard text analysis.
Smarter Red Teaming Test Generation
We’ve significantly improved how red teaming tests are generated. Our system now takes even more factors into account when creating adversarial test cases:- Product-Aware Generation: Tests are now more precisely tailored to your specific product’s strengths, weaknesses, and operational boundaries.
- Context-Sensitive Attacks: The generation process better understands your product’s intended use cases to craft more relevant and challenging scenarios.
- Enhanced Threat Modeling: Our algorithms now consider a broader range of factors when determining the most effective attack vectors for your particular AI system.
Better Metric Source Visibility and Management
Understanding where your metrics come from is now easier than ever! We’ve enhanced the platform to provide clearer visibility into metric sources:- Source Classification: All metrics are now clearly labeled with their source - whether they’re from established frameworks, custom Galtea implementations, or other origins.
- Enhanced Filtering: You can now filter metrics by their source to quickly find the evaluation criteria that best fit your needs.
- Improved Descriptions: Metric descriptions now include more detailed information about their origins and implementation, with links to relevant documentation.

- Galtea: Custom metrics designed specifically for your needs, like our new Factual Accuracy metric
- G-Eval: Framework-based metrics that use evaluation criteria or steps for assessment
- Established Frameworks: Metrics adapted from proven evaluation libraries and methodologies
Generate Quality Tests from Examples
You can now create Quality Tests directly from your own examples using the new Few Shots parameter. This makes it easier to tailor tests to your specific use cases and ensure your models are evaluated on the scenarios that matter most. Learn more about test creation.Metric Tags
Metrics now support tags for easier classification and discovery. Quickly find and organize metrics relevant to your projects. See all metrics.Enhanced Product Details
Products now include new detail fields:- Capabilities: What your product can do.
- Inabilities: Known limitations.
- Security Boundaries: Define the security scope and constraints.
Improved Q&A Generation
Question-answer pairs are now generated with improved accuracy and clarity, thanks to better text filtering and processing.New Guide: Setting Up Your Product Description
We’ve created a comprehensive guide to help you set up your product descriptions effectively. This guide covers best practices, examples, and tips to ensure your product is presented in the best light. Check it out hereGeneral Improvements
We’ve made various bug fixes and UX/UI improvements across the Dashboard, SDK, and more, making your experience smoother and more reliable.Major Platform Overhaul
We’ve been hard at work reorganizing and expanding the Galtea platform to handle more use cases and prepare for exciting future features. This release brings significant improvements to the dashboard, SDK, and test generation.Dashboard Enhancements
-
Reorganized Version, Test, and Evaluation Views:
Detailed views have been streamlined and improved for clearer insights. -
New Sessions Visualizations:
Easily organize and navigate conversations through our new Sessions feature. -
Evaluations Visualization Removed:
The dashboard now focuses on Sessions and Evaluations as the primary elements. -
Better Filters Across Tables:
Quickly find what you need with improved filtering capabilities on the dashboard. -
General Bug Fixes & UX Improvements:
Enjoy smoother interactions, clearer tooltips, and more intuitive code snippets.
SDK v2 Released
The new Galtea SDK v2 is here! It includes breaking changes to simplify workflows and add session support.-
Implicit Session Creation:
Sessions are created automatically when needed for evaluations. -
Repurposed evaluations.create():
The old method is replaced bycreate_single_turn()for test-based evaluations, whilecreate()now exclusively handles session-based evaluations. -
New evaluations.create_single_turn() Method:
Use this for single-turn test cases. It now requiresversion_idinstead ofevaluation_id. -
Simplified Version Creation:
Thegaltea.versions.create()method now accepts all properties directly, no need for anoptional_propsdictionary. -
Sessions Support:
Group multiple inference results under a single session for better multi-turn tracking usinggaltea.sessions.create().
Improved Test Case Generation
-
Smarter Test Coverage:
Test cases are now distributed more intelligently across your documents for better coverage based on the number of questions you choose to generate. -
Single Threat per Red Teaming Test:
Red Teaming Tests now only allow a single threat per test, ensuring clearer results.
Improved Test Generation
Our test generation capabilities have been significantly upgraded:- Versatile Red Teaming: Red Teaming tests are now more powerful, allowing you to employ multiple attack strategies to thoroughly probe your AI’s defenses.
- Better Synthetic Data: We’ve made general improvements to the quality of synthetic data generation, ensuring your tests are more effective and realistic.
Code Snippets Now Available on the Dashboard
We’re making it easier than ever to integrate Galtea into your development process!- Simplified Evaluation Setup: The “Create Evaluation” form on the dashboard has been replaced with a convenient code snippet. Simply copy and paste it directly into your project to get started.
- Streamlined Creation: Similarly, a new code snippet for “Create Evaluation” is now available on the dashboard, simplifying how you send evaluation data to Galtea. You can easily copy and paste this into your project.

Usability Improvements
We’ve also rolled out several usability enhancements based on your feedback:- Enhanced Readability in Tables: Table cells now correctly render line breaks, making it easier to view multi-line content and detailed information at a glance.
- Controlled Test Case Generation: To ensure optimal performance and manageability, the maximum number of test cases automatically generated for a single test from a knowledge base is now capped at 1000.
Streamlined Onboarding and Quicker Starts
We’ve revamped the platform onboarding! It’s now more visually intuitive and to help new users get evaluating in no time, we now provide a default Metric and a default Test. This makes it easier than ever to get started with Galtea and run your first evaluation quickly.Deeper Insights with Visible Conversation Turns
Understanding the full context of interactions is key. You can now view the complete conversation turns associated with your test cases directly within the dashboard. This offers richer context, aiding in more thorough analysis and debugging of your conversational AI products.
Dashboard Usability Boost
We’re continually refining the Galtea experience. This update brings several UI enhancements across the dashboard, designed to improve overall usability and make your workflow smoother and more intuitive.Tailor Your Test Generation: Selectable Test Case Counts
Gain more control over your testing process! When generating tests, you can now specify the exact number of test cases you want Galtea to create. This allows you to fine-tune the scope and depth of your tests according to your needs.Track Your Team’s Work: Creator Attribution Displayed
Clarity in collaboration is important. Now, the user who created a Product, Test, Version, or other key assets will be clearly displayed on their respective details pages. This helps in tracking ownership and contributions within your team.Enhanced Table Functionality for Easier Data Navigation
Working with data tables in the dashboard is now more efficient:- Clear Filter Indicators: Easily see which filters are currently applied to any table.
- Quick Filter Reset: A new “Clear All Filters” button allows you to reset your view with a single click.
New Conversation Evaluation Metrics
You can now evaluate conversations using these new metrics:- Role Adherence - Assess how well an AI stays within its defined role
- Knowledge Retention - Measure how effectively information is remembered throughout a conversation
- Conversation Completeness - Evaluate whether all user queries were fully addressed
- Conversation Relevancy - Determine if responses remain on-topic and purposeful
Enhanced Security Framework
We’ve significantly improved user access management by implementing an Attribute-Based Access Control (ABAC) strategy, providing more granular control over who can access what within your organization.Extended Data Generation Capabilities
Our data generation tools have been expanded with:- Catalan Language Support - Create synthetic data in Catalan to enhance your multilingual applications
- Added support for text-based files - Upload your knowledge base in virtually any text-based format including JSON, HTML, Markdown, and more
Improved Test Creation Experience
We’ve enhanced the clarity of threat selection in the Test Creation form. The selection now displays both the threat and which security frameworks that threat covers, making it easier to align your testing with specific security standards.
Analytics & Navigation Enhancements
- Reduced Clutter in Analytics Filters - Tests and Versions filtering now only display elements that have been used in an evaluation
- Streamlined Navigation - Clicking the “input” cell in the evaluations table now navigates directly to the associated Test Case
Bug Fixes & Improvements
We’ve resolved several issues to ensure a smoother experience:- Fixed a bug that could trigger an infinite loop in the Test Cases List of the dashboard
- Addressed multiple small UI glitches and errors throughout the platform
Improvements in Red Teaming Tests
-
New “misuse” threat implemented
Now red teaming incorporates a new threat, misuse, which are queries that not necessaryly malicious however out-of-scope for you specific product. You can now test whether your product can successfully block these queries by marking “Mitre Atlas: Ambiguous prompts” in the threat list. -
Better “data leakage” and “toxicity” tests
The red teaming tests incorporate better your product meta data, to generate the most adequate test cases for “data leakage” and “toxicity”.
Analytics Page Upgrades
We’re continuing to expand the power of the Analytics page! This update introduces:-
Radar View for Version Comparison
You can now visualize performance across multiple metrics for a single version using the brand-new radar view. It provides a quick way to understand strengths and weaknesses at a glance. -
Smarter Metric Filters
Filters now only show metrics that have actually been used in evaluations—removing unnecessary clutter and making it easier to find relevant data. -
Graph Tooltips
Hovering over truncated names now reveals full labels with tooltips, helping you understand graph contents more clearly.

SDK Safeguards
We’ve added protections to ensure your SDK integration is as smooth and reliable as possible:-
Version Compatibility Checks
If the SDK version you’re using is not compatible with the current API, it will now throw a clear error to prevent unexpected behavior. -
Update Notifications
When a new SDK version is available, you’ll get a console message with update information—keeping you in the loop without being intrusive.
Bug Fixes
- Metric Range Calculation
Some default metrics were previously displaying inverted scoring scales (e.g., treating 0% as best and 100% as worst). This is now resolved for accurate interpretation. - Test Creation Not Possible Through
.txtKnowledge Base Files
Due to a recent refactor, the creation of tests using knowledge base files with.txtextensions was not possible. This has been fixed and you can now create tests using.txtfiles as the knowledge base again.
Monitoring Is Live!
Real-world user interactions with your products can now be fully monitored and analyzed. Using the Galtea SDK, you can trigger evaluations in a production environment and view how different versions perform with real users. Read more here.Improved Galtea Red Teaming Tests
Our simulation-generated tests have been upgraded—delivering higher-quality outcomes. Red teaming tests can now be directed to validate even more specific aspects of various security standards, such as OWASP, MITRE ATLAS, and NIST. Specifically, we have improved jailbreak attacks, in addition to new financial attacks and toxicity prompts.New Analytics Page
A completely redesigned analytics page is now available! It features:- Enhanced Filtering Capabilities.
- Improved Data Clarity and Layout.

And with monitoring active, you can see production evaluation results in real time on this page!
User Experience Enhancements
We’re continuously refining the platform based on your feedback. This week’s improvements include:-
Customizable Evaluations List:
You can now select which metrics you are interested in, so the evaluations list only shows the ones you need. -
Enhanced Evaluation List Filtering:
Easily filter evaluations by versions, evaluations, tests and test groups. -
Enhanced Test List Filtering:
Easily filter tests by its group. -
Smart Table Sorting:
When you apply a custom sort, the default (usually creation date) is automatically disabled.