> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Changelog
> The updates for Galtea
## Track Changes to Your Test Cases
[Test cases](/concepts/product/test/case) now keep a revision history, like metric revisions. When you edit a test case's evaluation content, the platform creates a new revision instead of overwriting the old one. Annotation-only edits, such as reviewer or score, still edit in place. The list shows only the latest revision by default, with an opt-in filter for older ones, and each revision links back to the one it came from. Evaluations that ran against an old test case now show an **Outdated Test Case** badge, and the SDK warns you when this happens.
## Compare Version Coverage in Analytics
The analytics page now shows **shared test-case coverage** so you can compare version scores fairly. Two versions can have the same average score even if one was evaluated against many more test cases, and shared coverage makes that gap visible. Each version shows how many test cases it was evaluated against, out of all test cases used across the shown versions, as a count and a percentage, with a Total / Used / Ignored summary. The selected-version area of each analytics tab is now two cards: radial gauges for the selected version, and a scores radar across all shown versions. Both stay visible and follow one grouping toggle (Specifications / Metrics). All numbers respect the active filters.
## Platform Improvements
* **Iterate a version from an existing one:** You can now create a [version](/concepts/product/version) from an existing one to track how it evolved. Each version records its parent as an **Evolved from** link, available in the dashboard, API, and SDK.
* **Persona gender on test cases:** Test cases can now store and show the persona's gender.
* **Attach session metadata at creation:** The SDK can now add metadata to a [session](/concepts/product/version/session) when it is created.
* **Cancelled evaluation status in the SDK:** The SDK `EvaluationStatus` enum now includes `CANCELLED`.
* **Development routes hidden in production:** Dev and demo routes are no longer shown in the production dashboard.
* **Clearer report export errors:** Exporting a report for a product with no data now returns a clear client error instead of a server error.
* **Faster permission checks:** Permission checks now run in parallel, so permission-heavy requests are faster.
* **Sturdier SDK networking:** The SDK now reuses one HTTP session with retries and a timeout, and only prints colored output to a terminal.
* **Dashboard polish:** Reduced oversized text, fixed clickable empty space in truncated table cells, restored non-admin assistant routes to the catalog, and refreshed the sidebar create-product button.
## Run Voice Simulations on Your Phone Agents
You can now run voice simulations against AI agents that answer the phone. A new [phone connection](/concepts/product/phone-connection) tells Galtea how to reach your agent: you set the phone number once, and Galtea **dials your agent and runs the conversation as a real voice call**. This is Direct Inference over the phone.
A few things that make it powerful:
* **The caller sounds human, in your agent's language.** Set a [test case](/concepts/product/test/case)'s language and the synthetic caller speaks it with a natively-accented voice (a Spanish test case is answered in Spanish), so one phone connection covers many languages with no extra setup.
* **Voice agents are scored like text agents.** Galtea transcribes each agent reply with speech-to-text automatically, so every [metric](/concepts/metric) you already use works over the phone, including [security tests](/concepts/product/test/security-tests) that speak adversarial prompts out loud.
* **You can replay the call however you like.** The entire conversation is captured as a continuous [call recording](/concepts/product/version/session) that you can play back in the dashboard, either as a single clip or [turn by turn](/concepts/product/version/session/inference-result).
## Update Specifications from a Document
As your product evolves, its documentation (PRDs, design docs, spec sheets) often changes before its [specifications](/concepts/product/specification) do. A new **Update from document** action on the product detail page bridges that gap.
Upload a document and the platform compares its content against your product's existing specifications. Each suggestion is classified as one of:
* **New** — a behavioral expectation found in the document that has no matching specification yet.
* **Updated** — an existing specification that the document refines or extends.
* **Contradictory** — an existing specification that conflicts with something in the document.
You review all suggestions and choose which to apply. Accepted **new** suggestions create a new specification. Accepted **updated** and **contradictory** suggestions replace the referenced one. Anything you leave unselected stays untouched. Nothing changes without your explicit confirmation.
## Product Cards Show Version Scores
[Product](/concepts/product) cards now tell you more at a glance. Each card shows two [version](/concepts/product/version) rows:
* **Best** — the version with the highest average [evaluation](/concepts/product/version/session/evaluation) score, shown with a percentage (e.g., `v2 (55%)`).
* **Newest** — the most recently created version.
A version with no evaluations shows `(-)` instead of a percentage. The Evaluations counter and Total Cost row have been removed from cards to keep the view focused.
## Cross-Product Ranking in Val
Val (the Galtea AI assistant) can now compare your [products](/concepts/product) against each other on a shared [metric](/concepts/metric). Ask Val something like "how does the whole product family perform on Faithfulness?" and it returns a worst-first ranking with the evaluation count, mean, and min/max score for each product, plus an overall evaluation-weighted score across all of them.
If you do not specify a metric, Val first lists the metrics shared across your selected products and waits for you to pick one before returning numbers.
## Platform Improvements
* **Try a demo product during onboarding:** New users can now clone a ready-made Galtea demo during the onboarding flow.
* **Faster audio loading in inference results:** Audio files in [inference result](/concepts/product/version/session/inference-result) sessions now load lazily and in a single batched request as turns scroll into view, rather than fetching all URLs and preloading all audio on render. The audio player loads only the duration header for visible turns, so the seek bar shows the correct length without downloading the full file. Play latency is unchanged.
* **Val assistant fixes:** Several usability issues in the Val widget are resolved. The ⌘K / Ctrl+K shortcut no longer fires while you are typing in another field. Pressing Enter mid-stream inserts a newline as expected. Citation chips no longer appear for links inside code blocks. The clipboard copy button falls back gracefully on plain HTTP. The route catalog no longer suggests admin-only pages that regular members cannot access.
* **Tooltips close when the pointer moves onto their content:** Tooltips now close as soon as the pointer leaves the trigger, like any standard hover element. Previously, moving the pointer from the trigger onto the tooltip kept it open and could block clicks on elements beneath it.
* **Long entity names stay inside their column:** Long names and identifiers in link cells (Test, Version, and similar columns in data tables) now wrap and clamp to two lines inside their column instead of overflowing into the next one.
This release focuses on usability, bug fixes, and small improvements while we work on bigger features coming in the next weeks.
## Platform Improvements
* **Create specifications during onboarding:** You can now create your custom [specifications](/concepts/product/specification) during the onboarding flow to cover gaps you would like to evaluate that our AI system did not detect or did not expect to be a primary focus.
* **Filter sessions by status:** The [session](/concepts/product/version/session) list now has a status filter — select `PENDING`, `COMPLETED`, or `FAILED` (or any combination) to focus on the runs that matter. Available in the dashboard, the API (`statuses` query parameter on `GET /sessions`), and the SDK via [`sessions.list(status=…)`](/sdk/api/session/list).
* **See direct inference progress right away:** When you start a direct inference run, every [session](/concepts/product/version/session) it will produce now appears in the list as `PENDING` immediately — instead of trickling in as each one is picked up — so you can track progress from the moment the run starts.
* **Security metrics now evaluate correctly:** [Data Leakage](/concepts/metric/data-leakage), [Jailbreak Resilience](/concepts/metric/jailbreak-resilience), and [Misuse Resilience](/concepts/metric/misuse-resilience) evaluations were being silently marked `SKIPPED` because they depended on a field the product form no longer surfaces. These metrics now draw context from `product_capabilities` and `product_inabilities` (populated from your [specifications](/concepts/product/specification)) and run as expected.
* **"New Metric from template" fixed:** Clicking *From template* in the New Metric screen was throwing a routing error and closing the dialog immediately. The dialog now opens and works correctly.
## Documentation and Examples
* [SDK evaluation examples](/sdk/tutorials/specification-driven-evaluations) now show the specification-first approach — passing `specification_ids=` instead of metric names — side by side with the metric-name approach, in tabbed code blocks, on every relevant tutorial page.
## Smoother Onboarding
Getting your first [product](/concepts/product) evaluated now takes less waiting and less typing.
**Generation runs in the background.** When the onboarding tutorial creates the [tests](/concepts/product/test) and [metrics](/concepts/metric) for your selected [specifications](/concepts/product/specification), it no longer holds you on the review step until everything finishes. Generation continues while you keep moving through setup — only the final **Launch Evaluation** button waits for it to be ready, and if anything fails to generate you get a clear retry instead of a stuck screen.
**Your first product is named for you.** The onboarding form no longer asks you to name your product. The platform derives a concise name from the description you provide as part of the same step, so there's one less field between you and your first [evaluation](/concepts/product/version/session/evaluation).
## Platform Improvements
* **Endpoint connections are checked before any work is queued:** When you start an evaluation, the platform now runs a quick pre-flight probe against the version's [endpoint connection](/concepts/product/endpoint-connection) first. If the endpoint is unreachable or returns an error, the request is rejected immediately with a clear message — instead of queuing inferences that were always going to fail.
* **Faster large evaluation runs:** Evaluation runs now process their [sessions](/concepts/product/version/session) in parallel rather than one after another, so large runs finish sooner and a single big run no longer blocks others behind it. Each run also respects a per-endpoint rate limit, so your endpoint is never flooded with requests.
## Documentation and Examples
* A new [Improve Your Product with Evaluations and an AI Agent](/playbooks/improve-product-with-ai-agent) playbook walks through the full evaluate → diagnose → improve → compare loop: hand your failing evaluations to an AI coding agent using the [Galtea Agent Skill](/sdk/integrations/agent-skill) and CLI, let it propose a fix, then re-run the evaluations to prove it helped — illustrated end to end on a RAG support chatbot.
* The [Direct Inferences & Evaluations from Platform](/sdk/tutorials/direct-inferences-and-evaluations-from-platform) tutorial now documents the full **round-trip** for stateful multi-turn conversations: how to extract a value from one response into session metadata, and then send it back on the next turn.
## Analytics Insights
The **Analytics Overview** tab now includes an Insights card that automatically scans your evaluation data and surfaces what needs your attention. The card applies deterministic rules across three categories:
* **Score health** — flags [metrics](/concepts/metric) that fall below the warning (85) or critical (60) thresholds, and links each flag directly to the relevant evaluations with the judge's reasoning so you can dig in immediately.
* **Specification coverage** — identifies [specifications](/concepts/product/specification) that have no linked tests yet, so coverage gaps don't stay invisible.
* **Version regressions** — detects metric scores that dropped 10 or more points from one [version](/concepts/product/version) to another, distinguishing warnings from critical degradations.
Flags are sorted by severity. Each carries a named action — clicking it navigates you to the right place in the platform rather than leaving you to find it yourself. When nothing is flagged, the card shows an explicit **all-good state** so you know the check ran. A plain-language narrative at the top summarizes the findings.
Val (the platform assistant) now answers evaluation-quality questions using the same backend rules as this card, so the assistant and the dashboard always agree on what counts as a red flag.
## Smarter Run Evaluation Flow
Running an [evaluation](/concepts/product/version/session/evaluation) from a version or product now opens a **Specifications** tab by default. It pre-selects the [specifications](/concepts/product/specification) that have runnable [tests](/concepts/product/test) and submits via the existing batch endpoint — the platform resolves linked tests and metrics server-side, so you don't have to select them manually. If a product has no specifications, the dialog falls back to the familiar manual test and metric selector.
In the manual selector, the metric picker now ranks and pre-selects metrics based on relevance:
* **Linked** — metrics directly linked to the specification of your selected test are pre-selected and appear first.
* **Recommended** — metrics linked to any specification sharing the same test type appear next with a badge.
Incompatible metrics are still listed at the bottom so you can override, but the common case no longer requires manual searching.
## Complete Metric with AI
Two places in the dashboard now offer a **Complete With AI** button on [metrics](/concepts/metric): the manual create-metric form and the candidate edit dialog inside the AI metric generation review step.
Fill in the metric name, description, and evaluation type — and let the AI generate the judge prompt and [evaluation parameters](/concepts/metric/evaluation-parameters) for you. Your name, description, and tags are never overwritten. If you already have a judge prompt, the platform prompts for confirmation before replacing it. AI-generated fields display a badge so you always see what was filled by the model.
This works for [AI Evaluation](/concepts/metric/evaluation-types#ai-evaluation) and [Human Evaluation](/concepts/metric/evaluation-types#human-evaluation) metrics. Self-Hosted metrics are excluded (they have no judge prompt).
## AI-Assisted Specification Creation
Two improvements to the specification creation flow:
**Fill with AI** (the dialog on the create and update pages) now rewrites your rough note into a properly written specification description *and* classifies the type in one step, returning `description`, `type`, `testType`, and `testVariant` together. Previously it only filled the category selectors and never touched the description.
**Complete with AI** (the inline button) sends the description you already wrote, applies only `type`, `testType`, and `testVariant`, and never touches your text. The button was previously labeled "Categorize with AI" — the new name reflects that it completes the fields you left empty without altering the ones you filled.
In **Detailed** generation mode, the AI now also considers a built-in catalog of safety and quality [specifications](/concepts/product/specification) that apply to most AI products: jailbreak resistance, illegal-activity refusal, system-prompt non-disclosure, cross-user data leakage, AI-identity disclosure, and others. Specifications that don't fit (for example, safety specs for backend-only products with no human users) are excluded automatically. You review and edit everything before any of it is saved.
## Import Endpoint Connection from cURL
Creating an [endpoint connection](/concepts/product/endpoint-connection) now offers an **Import from cURL** option. Paste a cURL command — from your API docs, a Postman collection, or a terminal history — and the platform extracts the URL, headers, and request body, then maps the body to a Jinja2 input template with `{{ input.user_message }}` pre-wired. For conversation-type connections, a lightweight AI pass enriches the template further. The **New Endpoint Connection** button is now a **From cURL / Blank** dropdown.
In the onboarding flow, the cURL paste step is the default view; the traditional form is available via **Skip — fill in manually**. Fields that couldn't be recovered from the cURL command are flagged inline, and a mandatory probe test confirms the connection works before saving.
## Platform Improvements
* **Row click navigates to entity details:** Every table row in the dashboard now navigates to the entity's detail page on click. The primary cell renders a real link, so Ctrl/Cmd/middle-click opens it in a new tab. Clicking on a text cell opens a full-text dialog with a copy button instead of navigating. Drag-to-select on table text still works.
* **Assistant renamed to Val:** The Galtea platform assistant is now named **Val** across the dashboard panel and the public docs site. The panel header, launcher pill ("Ask Val"), and the agent's own persona are all updated.
* **Breadcrumb navigation restores your position:** Clicking a breadcrumb link now returns you to the exact tab and pagination page you were on when you drilled into the current entity. For a fresh deep link, it falls back to the tab that lists the entity you were viewing.
* **Analytics score charts use a fixed 0–100 axis:** Score charts in the Analytics section are anchored to a 0–100 scale so visual proportions are consistent across products and time windows. Previously the axis auto-scaled to the data range, making small absolute differences appear large.
* **List endpoints default to your organization:** Every `GET` list endpoint now scopes to the caller's organization automatically, so you no longer need to pass an organization filter to list your own data. Admins can opt into a cross-organization view with the `allOrganizations` flag.
## Galtea Assistant Redesign
The Galtea AI assistant is now a **non-modal docked side panel** that sits alongside the dashboard instead of covering it. When open, the page content reflows into a true split view — the panel holds a fixed width on the right while the rest of the dashboard fills the remaining space. You can resize the split by dragging the divider. Open and close the panel from anywhere in the dashboard with **⌘K** (Mac) or **Ctrl+K** (Windows/Linux).
The assistant now shows its work while it processes your message. A live step trace displays each phase of the agent's reasoning — searching the docs, calling API tools, writing the answer — with the active step pulsing in real time. When the response arrives, the trace collapses to a compact summary (for example, "Worked for 2.3s · 4 steps") that you can expand at any time to review the full trajectory.
For write operations, the assistant proposes an action and you approve it before anything changes. Each proposal shows a preview of what will happen, and destructive actions display how many items would be affected before you confirm. When you approve, the dashboard navigates you directly to the affected [product](/concepts/product), [version](/concepts/product/version), or [evaluation](/concepts/product/version/session/evaluation). The assistant can now reach any allow-listed resource in the platform — reads are driven by the live API catalog, and writes cover 28 curated operations — so it adapts as new endpoints are added without requiring new tool code.
## Redesigned Onboarding Experience
New users now flow through a focused, full-page onboarding shell that adapts to how you want to run evaluation — **import traces**, **connect your endpoint**, or **use the SDK** — and pre-selects the [specifications](/concepts/product/specification) matching the goal you gave at signup, so first [evaluation](/concepts/product/version/session/evaluation) results arrive faster.
## Word Document Upload for Product Setup
When setting up your product with AI assistance, you can now upload **Word documents** (`.docx`) alongside PDFs and other formats. The platform extracts the content and uses it to generate [specifications](/concepts/product/specification), so teams that maintain documentation in Word can feed it directly into Galtea without converting files first.
## Job Cancellation
You can now cancel in-progress async jobs from both the API and the SDK. When you start an inference batch — for example, via [`evaluations.run()`](/sdk/api/evaluation/run) in endpoint-connection mode — and need to stop it early, call [`galtea.jobs.cancel(job_id)`](/sdk/api/job/cancel). Queued jobs are removed from the queue; jobs already running receive a cooperative cancellation signal and stop at their next checkpoint. Work already dispatched when the signal is observed is not rolled back. Cancellation is idempotent — calling `cancel()` on an already-cancelled job returns `state='cancelled'` rather than raising an error. Use [`galtea.jobs.get_status(job_id)`](/sdk/api/job/get-status) to check the current state before deciding whether to cancel. See the [Jobs Service reference](/sdk/api/job/service) for the full API.
## Platform Improvements
* **PASS/FAIL verdict shown in the annotation dialog:** When annotating [evaluations](/concepts/product/version/session/evaluation), the score pill now displays the binary PASS/FAIL outcome alongside the numeric score so you can see the verdict at a glance.
* **"See details" button on every table row:** All entity tables — [sessions](/concepts/product/version/session), [evaluations](/concepts/product/version/session/evaluation), [tests](/concepts/product/test), and more — now show a direct-navigation icon button to the right of the row actions menu. Click to open the detail page; hold Cmd/Ctrl to open it in a new tab.
* **Unified test creation form:** Creating a [test](/concepts/product/test) from a single [specification](/concepts/product/specification) now uses the same creation form as batch creation. The previous dedicated single-spec flow has been retired.
* **Accuracy-requirement [specifications](/concepts/product/specification) correctly classified:** Specifications of the accuracy-requirement type are now saved as `POLICY + QUALITY`. Previously, creating one via the API produced an incorrect type.
* **Coverage radar chart renders correctly in edge cases:** The coverage radar chart now renders when two [specifications](/concepts/product/specification) share the same name or when a [version](/concepts/product/version) has no scores for one of the axes. Previously, either condition caused the chart to be skipped.
* **Clearer SDK error messages for agent integrations:** When an [agent skill](/sdk/integrations/agent-skill) invocation fails due to an argument shape mismatch, the SDK now surfaces a clear description of the issue instead of a cryptic internal error.
* **LLM configuration errors surface immediately:** Misconfigured API keys, unsupported model names, and other non-retryable LLM errors now cause [evaluations](/concepts/product/version/session/evaluation) to fail on the first attempt rather than consuming the full retry budget before reporting the problem.
* **Red-teaming generation handles content filters:** When a content filter blocks a generation step, the red-teaming generator now retries. If the seed prompt itself keeps triggering the filter after retries, that seed is skipped and generation continues with the remaining prompts.
* **[Specification](/concepts/product/specification) linking documented in CLI and SDK:** The `galtea spec link-tests` and `galtea spec unlink-tests` commands and their SDK equivalents — [`galtea.specifications.link_tests()`](/sdk/api/specification/link-tests) and [`galtea.specifications.unlink_tests()`](/sdk/api/specification/unlink-tests) — are now documented in the CLI `--help` output and SDK docstrings, making them easier to discover.
## Metric Optimization Lifecycle
When you optimize an [AI Evaluation metric](/concepts/metric/evaluation-types#ai-evaluation), the result now behaves like any other revision of that metric. The optimized version joins the same revision lineage as a manual "Create new revision" — so you can trace the full optimization history in the metric's revision panel and see how the judge prompt has evolved across generations.
The **Optimize** button now shows the correct annotation threshold. It counts only the [human annotations](/concepts/metric/evaluation-types#human-evaluation) where your score and the AI's score fall on opposite sides of the PASS/FAIL boundary — evaluations where your annotation genuinely overrides the AI's verdict. Previously, any numeric difference between scores counted, even when both scores agreed on the outcome.
## Platform Improvements
* **Incompatible evaluations now show `SKIPPED` instead of `FAILED`:** [Evaluations](/concepts/product/version/session/evaluation) that cannot run — such as sessions where the agent produced no output, or [deterministic metrics](/concepts/metric/evaluation-types) applied to multi-turn sessions — are now caught at pre-flight and marked `SKIPPED` immediately, before being dispatched to the evaluator. Previously these were dispatched and returned `FAILED` with no clear explanation. The evaluation result now includes the specific reason the evaluation was skipped.
## Documentation and Examples
* **Direct Inference OTel collector: authentication guide updated:** The [Direct Inference OTel exporter tutorial](/sdk/tutorials/direct-inferences-and-evaluations-from-platform) now covers the required `Authorization: Bearer ` header with four supply patterns — SDK environment variables over HTTP, SDK environment variables over gRPC, programmatic Python, and OTel Collector YAML relay. The section also documents shell-expansion pitfalls with `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, a `curl` smoke-test command for self-validation, and a note on the current gRPC TLS handshake issue with HTTP/4318 as the validated path.
## Import Langfuse Traces
If you already export traces from [Langfuse](https://langfuse.com), you can now bring them into Galtea without any code changes. Open **New Session → Import traces (Beta)**, upload your Langfuse export file — JSON, JSONL, or CSV are all accepted, and both the Traces and Observations export tabs from Langfuse work — select the [version](/concepts/product/version) you want to link them to, and click **Import**. Each trace becomes a [session](/concepts/product/version/session) with its observations mapped to [inference results](/concepts/product/version/session/inference-result), ready to evaluate immediately.
The import runs as a background job, so you can navigate away while large files are processed. The dialog stays open and shows progress; once complete, the sessions table refreshes automatically.
This is the recommended path for teams already using [Langfuse's LangChain or LangGraph integration](/sdk/integrations/langfuse/integration-guide) and wanting to run Galtea evaluations on their existing trace history without modifying their agent code.
## SDK: Model Management
The SDK now includes a `ModelService`, accessible as `galtea.models`. It supports the full lifecycle — `list`, `get`, `get_by_name`, `create`, `update`, and `delete` — bringing programmatic parity with the dashboard's [model management](/concepts/model) for cost tracking and estimation. `list()` accepts both a partial-match `name` filter and an exact-match `names` list, alongside the standard `ids`, pagination, and date-range parameters.
See the [Model Service reference](/sdk/api/model/service) for the full API.
## Platform Improvements
* **`FULL_PROMPT` metric creation is now blocked:** `POST /metrics` and `galtea.metrics.create(...)` reject requests that set `source=FULL_PROMPT` or that supply a `judgePrompt` without `evaluationParams`. Use `PARTIAL_PROMPT` for new [AI Evaluation metrics](/concepts/metric/evaluation-types#ai-evaluation) — it covers the same use cases with explicit, user-configurable [evaluation parameters](/concepts/metric/evaluation-parameters). Existing `FULL_PROMPT` metrics are unaffected and continue to be evaluated end to end.
* **SDK `generate()` reflects live agent status:** `inference_results.generate()` now creates the record as `PENDING` while your agent runs and only transitions to `GENERATED` on a successful return. Previously the record was immediately marked `GENERATED` with a null output, which broke the contract that `GENERATED` means the agent completed successfully.
* **Onboarding video on the welcome screen and Quickstart:** The product welcome screen and the [Quickstart](/quickstart) page now include an embedded walkthrough video so new users can see the end-to-end setup flow before writing any code.
* **More accurate AI evaluation scores:** Judge prompts now produce `reason` before `score`. This chain-of-thought ordering consistently improves score quality by forcing the model to articulate its reasoning before committing to a number.
* **Table header overflow fixed:** Long column headers — such as version hashes in the analytics comparison table — are now clipped at the cell boundary instead of visually bleeding into adjacent columns.
## Metric Optimization
When you create a custom [AI Evaluation metric](/concepts/metric/evaluation-types#ai-evaluation), the judge prompt rarely scores responses exactly the way your team would on day one. The new Optimize action closes that gap: it takes a metric's current judge prompt, learns from the [human annotations](/concepts/metric/evaluation-types#human-evaluation) your team has already submitted, and rewrites the prompt so the AI score lines up more closely with the human score. The result is saved as a sibling metric named ` (Optimized)`, so you can compare both versions side by side before adopting the new one.
The Optimize button appears on the details page of any AI Evaluation [metric](/concepts/metric) created by your organization. It becomes available once enough annotations have been collected, with a particular focus on cases where the annotator disagreed with the AI score: those disagreements are the signal the optimizer learns from.
## Galtea Assistant (Beta)
A new in-product AI assistant is now available. The floating launcher in the bottom-right corner of the dashboard opens a chat panel that can answer questions about the platform, point you to the right page, and walk you through common workflows. The same assistant is also embedded in the docs site.
## Form Improvements
Submitting a new [product](/concepts/product) now uses AI to validate that the description matches what Galtea can test. Submissions that look voice, image, or multimodal, or that read more like a system prompt or build instructions than a product description, open a confirmation dialog before saving. The check is advisory and never blocks creation, so you can review the warning and proceed or go back and edit.
Navigating away from any creation or edit form with unsaved changes now opens a confirmation dialog with "Discard" and "Keep editing" choices, so accidental clicks no longer lose your work. Forms protected include [metric](/concepts/metric) and [product](/concepts/product) creation and edit, [test](/concepts/product/test) edit, [endpoint connection](/concepts/product/endpoint-connection) edit, and dialog-level flows such as Add [Specification](/concepts/product/specification).
## Platform Improvements
* **Full customization when creating tests from Behavior specs:** Creating a [test](/concepts/product/test/behavior-tests) from a [Behavior specification](/concepts/product/specification) now gives you the complete test creation form, with every customization option available, including data-catalog file uploads.
* **Faster AI-assisted product setup:** Generating a product description and specifications during product creation is now noticeably faster. Specification file uploads are also accepted in both quick and detailed generation modes.
* **Clearer status for empty-parameter evaluations:** [Evaluations](/concepts/product/version/session/evaluation) whose required fields (such as `retrieval_context`) are empty or whitespace-only strings now resolve as `SKIPPED` upfront with a clear "missing required parameters" reason, instead of dispatching and landing as `FAILED`.
* **CLI: background spec refresh and safer plaintext login:** The [Galtea CLI](/cli/installation) now refreshes its OpenAPI spec cache in a detached background process, so commands no longer pay the refresh latency. Logging in to a non-localhost HTTP host now requires `--insecure` to prevent accidental credential exposure over plaintext; the consent is sticky on that host so re-logins do not re-prompt.
## Specifications View Redesign
The [specifications](/concepts/product/specification) list is now split into two tables: **Policies** and **Capabilities & Inabilities**. Each row shows separate **Metrics** and **Tests** columns — a green check with a count when the resource exists, or a "Missing" pill that routes you to the [test](/concepts/product/test) creation form when it does not.
## Two-Factor Authentication
Email two-factor authentication is now available under **Settings → Security**, and organization admins can enforce MFA for all members from the organization settings page. See [Registration and Sign-In](/registration) for details.
## Documentation and Examples
* **Galtea Agent Skill:** The [Galtea Agent Skill](/sdk/integrations/agent-skill) page has been refreshed with updated install instructions and the Claude Code plugin install path.
## Platform Improvements
* **Fraction input for annotation scores:** The [human evaluation](/concepts/metric/evaluation-types) score field now accepts fractions (e.g., 7 out of 10) alongside percentages.
* **Max iterations for behavior test generation:** [Behavior test](/concepts/product/test/behavior-tests) AI generation now exposes a `max_iterations` setting, with the default lowered from 20 to 10.
* **Language field promoted in test creation:** The Language dropdown is now visible directly in the test creation form, not only under Advanced Options.
* **CLI via Homebrew:** The Galtea CLI is now installable via Homebrew. See the [CLI installation page](/cli/installation) for all options.
* **Same-name metric revisions:** Creating a [metric](/concepts/metric) revision that reuses the parent's name no longer fails.
## Galtea CLI
The Galtea CLI is now generally available. Install it via `apt`, `dnf`, or `pip`, then authenticate with `galtea login` or a `GALTEA_API_KEY` environment variable. The CLI auto-generates a `galtea ` command for every public API endpoint, so new endpoints appear as new commands without a CLI release. See the new [Installation](/cli/installation) and [Usage](/cli/usage) pages to get started.
The [Galtea Agent Skill](/sdk/integrations/agent-skill) now drives Galtea through the CLI, which improves how reliably coding assistants like Claude Code and Cursor can manage Galtea on your behalf.
## Product Tags
[Products](/concepts/product) now support tags. You can add and edit tags from the create and update forms, filter the product list by one or more tags, and see tags as chips on product cards and the details page.
## Smarter Pre-Evaluation Guidance
Before launching an [evaluation](/concepts/product/version/session/evaluation), the run dialog now checks whether each selected [metric](/concepts/metric) can be satisfied by the current product and endpoint connection configuration, and flags any that cannot with a tooltip describing what is missing.
When an evaluation is still [SKIPPED](/concepts/metric/evaluation-parameters#troubleshooting-skipped-evaluations) at runtime, the error message now explains what is missing and where to fix it, grouped by source, with actionable guidance included.
## Structured Inputs for AI-Generated Tests
Structured JSON inputs were previously limited to uploaded or manually created [test cases](/concepts/product/test/case). AI-generated test cases — scenarios, red-teaming, and gold-standard — now also receive a structured `input` populated from the endpoint connection's JSON Schema.
AI-assisted product and specification generation also now respects the language of the source material.
## Documentation and Examples
* New [CLI Installation](/cli/installation) and [CLI Usage](/cli/usage) pages.
* New [`AgentInput` reference page](/sdk/api/agent/input) with all fields, the `ConversationMessage` structure, and updated SDK code examples.
## Platform Improvements
* **Endpoint connection picker in test creation:** [Test creation](/concepts/product/test/case) and the AI candidate review screen now offer an explicit endpoint connection picker under Advanced Options.
* **Filter test cases by input substring:** The [Test Cases](/concepts/product/test/case) list has a new debounced text filter, including matches inside structured JSON inputs.
* **Metric lineage on the details page:** Revised [metrics](/concepts/metric) display an "Evolved from" link to the parent metric, and the revision flow now shows a warning before marking the source metric as legacy.
* **Inference Results in the session dropdown:** The session row dropdown now includes an "Inference Results" option alongside "Evaluations" and "Traces".
* **Sidebar cleanup:** The Models entry has been removed from the sidebar. Models remain reachable from the organization dropdown and the version forms.
* **Optional secret fields on endpoint connection updates:** Credential fields are now optional when editing an existing [endpoint connection](/concepts/product/endpoint-connection) — leave them empty to keep the stored value.
* **SDK `@observe` fix for async agents:** The `observe` decorator now correctly preserves Galtea context across async coroutines.
* **Clearer max-iterations stopping reason:** Conversations that hit the iteration cap now record a "Max iterations reached" stopping reason instead of a generic error.
* **Updated wording:** Product creation now reads "Register product" throughout the platform.
## Metric Versioning with Evaluation Replay
[Metrics](/concepts/metric) now support versioning. The new "New Revision" action on any metric opens the creation form pre-filled with the source metric's name, description, tags, evaluation type, judge prompt, [evaluation parameters](/concepts/metric/evaluation-parameters), evaluator model, and user groups. Revised metrics stay linked to their predecessor through a parent-child lineage, so you can trace the evolution of a metric over time. When a revision is created, only the direct parent is marked as legacy, allowing parallel branches from the same source.
After saving a revision, you are prompted to replay historical [evaluations](/concepts/product/version/session/evaluation) from the metric family against the new version. The replay dialog lists your products, shows how many previous evaluations are eligible per product, and lets you select which ones to re-run. Evaluations that have already been replayed are automatically deduplicated.
The metric creation flow now surfaces all three modes upfront: Blank, From Template, and Metric Revision, each with a description of what it does. "From Template" (previously called "Duplicate") creates an independent copy with no lineage link, while "Metric Revision" creates a linked successor.
## Galtea Agent Skill
The [Galtea Agent Skill](/sdk/integrations/agent-skill) is now available as an open-source [Agent Skill](https://github.com/anthropics/skills) for AI coding assistants. Once installed, tools like Claude Code, Cursor, and Windsurf can authenticate against Galtea, manage [products](/concepts/product), run [evaluations](/concepts/product/version/session/evaluation), inspect [sessions](/concepts/product/version/session) and [traces](/concepts/product/version/session/trace), and follow validated workflows without you having to explain Galtea first.
Install it by pointing your coding agent to the [`Galtea-AI/skills`](https://github.com/Galtea-AI/skills) repository, or use the [skills CLI](https://github.com/anthropics/skills): `npx skills add Galtea-AI/skills --skill "galtea"`.
## Run Evaluations from Specifications
You can now run [evaluations](/concepts/product/version/session/evaluation) directly from a [specification](/concepts/product/specification) page. The action is available both on the specification detail page header and as a row action on the specification list. The dialog shows a version dropdown (filtered to versions with an [endpoint connection](/concepts/product/endpoint-connection), latest pre-selected) and displays the specification's linked tests and metrics as read-only badges, so you can verify the evaluation scope at a glance before launching.
The action is gated on the specification having at least one linked metric and one linked test. When either is missing, the button is disabled with a tooltip explaining the requirement.
## Language Variables in Input Templates
Two new built-in template variables are available in [endpoint connection](/concepts/product/endpoint-connection) input templates: `{{ language_code }}` resolves to the test case's ISO 639-1 code (e.g. `es`), and `{{ language_name }}` resolves to the full language name (e.g. `Spanish`). Both resolve to an empty string when the test case has no language set, keeping templates compatible with language-agnostic tests. The placeholder toolbar in the dashboard includes both variables for easy insertion. For the full template syntax reference, see [Structured Input Template Syntax](/concepts/product/endpoint-connection-template-syntax).
## Reliable Trace-Dependent Evaluations
Evaluations that depend on [trace](/concepts/product/version/session/trace) data (`TRACES` or `TOOLS_USED` [parameters](/concepts/metric/evaluation-parameters)) now wait for traces to be ingested before running. Previously, these evaluations could execute against empty trace data when traces had not yet arrived. The platform now defers them with automatic retries, and if traces do not arrive within the retry window, the evaluations land as `SKIPPED` with an actionable reason explaining what was missing. Sessions already marked as completed are detected immediately, avoiding unnecessary waiting. No credits are consumed for skipped evaluations.
## Export Evaluation Results as CSV
You can now export [evaluation](/concepts/product/version/session/evaluation) results directly from the dashboard as a CSV file. The export action appears in the row selection toolbar on the evaluation list page. With no rows selected, all evaluations matching the current filters are exported. When specific rows are selected, only those are included.
The CSV includes columns for Score, Metric, Input, Output, Expected Output, Retrieval Context, Version, and Test, along with export-only identifier columns (Evaluation ID, Session ID, Test Case ID) that do not appear in the table UI but provide full traceability for downstream analysis.
## Inference Result Error Tracking
When an agent crashes or returns an empty response during [simulation](/sdk/api/simulator/simulate), the corresponding [inference result](/concepts/product/version/session/inference-result) is now properly marked as `FAILED` with the error details attached, instead of remaining stuck in a `PENDING` state. Each inference result carries its own error information, so you can see exactly what went wrong at the individual turn level.
The evaluation pipeline also respects these statuses: evaluations are automatically skipped for non-ready inference results, with a clear reason recorded explaining which turns were not ready and why. In the dashboard, failed turns display a red error badge with the error message visible inline.
## One-Click Metric Duplication
You can now duplicate an existing [metric](/concepts/metric) as a starting point for a new one. The "Use as Template" action in the metric dropdown navigates to the creation form pre-filled with the source metric's name, description, tags, evaluation type, judge prompt, evaluation parameters, evaluator model, and user groups. The duplicated metric is independent from the original, so you can adjust any field before saving.
## Platform Improvements
* **Test case source file tracking:** [Test cases](/concepts/product/test/case) generated from uploaded documents now display the name of the original source file, improving traceability for file-based test generation.
* **Annotate action on evaluation details:** The Annotate (or Evaluate, for pending human review) action is now accessible directly from the [evaluation](/concepts/product/version/session/evaluation) detail page header dropdown, in addition to the list view context menu.
* **Clearer error messages for test generation:** When [QUALITY test](/concepts/product/test/case) generation fails, the error message now includes the specific reason returned by the generator, replacing generic HTTP error responses with actionable guidance.
* **Analytics radar chart fix:** The radar chart in the Analytics tab now works correctly when a specification section has fewer than 3 scored items. The view automatically switches to radial value cards with a tooltip explaining the minimum requirement.
* **Smoother table text filters:** Text filter inputs across all dashboard tables no longer flash or revert while typing. Filter changes are debounced before syncing to the URL, reducing unnecessary API calls and eliminating the visible blink caused by intermediate re-renders.
## Human Annotation for All Evaluations
Human annotation is no longer limited to [human evaluation metrics](/concepts/metric/evaluation-types). You can now annotate any completed evaluation with a human score and reasoning, regardless of its metric source. This means AI-evaluated results from full-prompt, partial-prompt, deterministic, or self-hosted metrics can all receive a parallel human assessment. Unlike human evaluation metrics (which contribute to the evaluation score and analytics), human annotations are stored as supplementary feedback alongside the original result.
## SDK: Endpoint Connection Management
The SDK now provides a full CRUD service for [endpoint connections](/concepts/product/endpoint-connection), so you can manage your agent connections entirely from code. Available operations include [`create`](/sdk/api/endpoint-connection/create), [`get`](/sdk/api/endpoint-connection/get), [`get_by_name`](/sdk/api/endpoint-connection/get-by-name), [`list`](/sdk/api/endpoint-connection/list), [`update`](/sdk/api/endpoint-connection/update), [`delete`](/sdk/api/endpoint-connection/delete), and [`test`](/sdk/api/endpoint-connection/test-connection).
The service supports all authentication types (Bearer, API Key, Basic), retry configuration with customizable backoff strategies, and partial updates where you can modify individual fields without resending the entire configuration.
## SDK: Job-Based `wait_for`
The [`evaluations.wait_for()`](/sdk/api/evaluation/wait-for) method now accepts a `job_id` parameter in addition to `evaluation_ids`. When you run an evaluation against an [endpoint connection](/concepts/product/endpoint-connection) without a local agent, [`run()`](/sdk/api/evaluation/run) returns a `jobId`. You can now pass that ID directly to `wait_for(job_id=...)` and let the SDK handle the entire lifecycle: polling the job status, discovering the evaluations it produced, and waiting for them to complete.
## SDK: Langfuse `CallbackHandler`
Building on the [Langfuse integration](/sdk/integrations/langfuse/overview) from the previous release, the SDK now includes a [`CallbackHandler`](/sdk/api/langfuse/callback-handler) for LangChain users. Instead of decorating individual functions with `@observe`, you can pass the handler directly to `chain.invoke(config={"callbacks": [handler]})` and every LLM call, tool invocation, and retriever query in the chain will be traced automatically.
The handler manages Galtea context through depth tracking, so nested callbacks are handled correctly without manual cleanup. It works as a singleton (set the inference result ID per request) or as a per-request instance. See the [CallbackHandler integration guide](/sdk/integrations/langfuse/integration-guide) for setup instructions.
## Structured JSON Inputs
This release extends JSON-structured inputs across the platform.
**SDK: `input_data` and `context_data` on AgentInput.** When using the structured agent signature, your agent function now receives [`context_data`](/sdk/api/simulator/simulate) alongside the conversation messages. If your [test cases](/concepts/product/test/case) include structured context fields (e.g., `customer_tier`, `language`, or any custom key-value pairs), they flow automatically into `AgentInput.context_data` during evaluation and simulation. The [`generate()`](/sdk/api/inference-result/generate) method also accepts structured `input` as a dictionary with multiple fields.
**CSV import with JSON values.** When importing test cases from CSV, the platform now detects and preserves JSON objects in the input and context columns. You no longer need to flatten structured data before uploading. For the time being, structured JSON inputs are only available for uploaded or manually created tests.
## Langfuse Integration
If you already use [Langfuse](https://langfuse.com) to observe your LLM application, you can now send the same traces to Galtea for evaluation by swapping a single import. Replace `from langfuse import observe` with `from galtea.integrations.langfuse import observe`, and every span will flow to both Langfuse and Galtea simultaneously.
The integration is fully transparent to Langfuse: your Langfuse dashboard, trace IDs, and alerts remain completely unaffected. Galtea only exports traces when you explicitly link an `inference_result_id`, so there is no overhead for unrelated function calls.
Two wrappers are available:
* [`observe`](/sdk/api/langfuse/observe) decorator for the standard `@observe` pattern
* [`start_as_current_observation`](/sdk/api/langfuse/start-as-current-observation) context manager for manual span control
Get started with the [Langfuse Integration overview](/sdk/integrations/langfuse/overview), or check the [Integration Guide](/sdk/integrations/langfuse/integration-guide) for a detailed walkthrough of the code changes needed to add Galtea evaluation alongside Langfuse.
## Public REST API Reference
The platform now exposes a public [OpenAPI specification](https://api.galtea.ai/openapi.json) covering approximately 81 endpoints across products, versions, specifications, tests, sessions, evaluations, metrics, traces, and more. Internal-only endpoints (authentication flows, billing, user administration) are excluded automatically.
You can import the spec into Postman, Cursor, Claude, or any OpenAPI-compatible tool to explore and call the API directly. Every endpoint description now includes a link back to the relevant [documentation page](/introduction), making it easier to navigate between the spec and the guides.
## Platform Improvements
* **Improved code and JSON display:** Code and JSON data across evaluation details, trace details, session details, and endpoint connection test results now render with proper formatting and syntax highlighting instead of plain preformatted text.
* **Flexible behavior test case editing:** The input field is no longer required when editing [behavior test cases](/concepts/product/test/case), since behavior-type tests do not always need explicit input values.
* **Specification type tooltips:** Hovering over the specification type pill (Capability, Inability, Policy) in the dashboard now shows a short description of what each type means, linking to the [Specifications documentation](/concepts/product/specification) for more detail.
## Zero-Code Trace Ingestion via OpenTelemetry
If your service is already instrumented with [OpenTelemetry](https://opentelemetry.io/), you can now send traces to Galtea without writing any Galtea-specific code. The platform exposes an OTLP-compatible trace ingestion endpoint that accepts standard OTLP/JSON payloads from your existing OTel Collector.
When you run a [direct inference](/concepts/product/endpoint-connection) through Galtea, the platform automatically injects a W3C `traceparent` header into the request to your endpoint. If your service propagates that header internally (as most OTel-instrumented services do), all resulting spans are automatically correlated with the corresponding Galtea [inference result](/concepts/product/version/session/inference-result). No manual ID passing, no SDK calls for trace creation, just your existing instrumentation working end to end.
For services that are not OTel-instrumented, the Galtea SDK's [`@trace` decorator](/sdk/api/trace/trace-decorator) and [`start_trace` context manager](/sdk/api/trace/start-trace) continue to work as before. A step-by-step guide covering all tracing methods, including [W3C trace context propagation](/sdk/tutorials/direct-inferences-and-evaluations-from-platform#option-3-w3c-trace-context-propagation), is available in the documentation.
## Structured JSON Inputs
[Test case](/concepts/product/test/case) inputs and [session](/concepts/product/version/session) contexts now support structured JSON objects in addition to plain text strings. This means you can store complex, multi-field payloads directly in the platform database. Existing string-based inputs continue to work unchanged.
In upcoming releases, structured JSON inputs will be more broadly supported across the platform, including endpoint connections and SDK methods.
## SDK: Synchronous Evaluation Polling
The Python SDK now includes [`evaluations.wait_for()`](/sdk/api/evaluation/wait-for), which polls one or more evaluations until they leave `PENDING` status and raises a `TimeoutError` if the deadline is exceeded. This simplifies scripts and CI pipelines that need to block until results are ready.
## Normalized JSON Field Match Metric
The [JSON Field Match](/concepts/metric/json-field-match-normalized) metric now supports a normalized comparison mode that performs case-insensitive and accent-insensitive matching within field values. For example, `"Sí"` and `"SI"` are treated as equal. This is useful when evaluating outputs in multilingual contexts or when casing differences are not meaningful.
## Editable Test Metadata
[Tests](/concepts/product/test) now include an editable `metadata` field where you can store free-form notes or custom tracking information. The field is available in the dashboard, the API, and the [SDK](/sdk/api/test/service), giving you a lightweight way to annotate tests without modifying their structure.
## Platform Improvements
* **Time-of-day filters:** Date filters throughout the dashboard now support hour, minute, and second precision, so you can narrow results to a specific time window rather than a full day.
* **Augmented column:** The test cases table now includes an **Augmented** column indicating which test cases were generated through AI augmentation.
* **Double-click prevention:** The **Save Accepted** button during metric generation no longer triggers duplicate submissions if clicked rapidly.
* **Pagination stability:** Table pagination no longer resets unexpectedly when navigating between pages via URL.
* **Show Details navigation:** The **Show Details** action in entity tables now opens directly to the details tab.
## Specifications in Analytics
The analytics dashboard now includes a dedicated [Specifications](/concepts/product/specification) section that aggregates evaluation scores per specification, giving you a clear picture of how each policy area is performing across [versions](/concepts/product/version). Specification-level radar charts let you compare coverage at a glance, and clicking any specification label filters the entire dashboard to show only [metrics](/concepts/metric) linked to that specification.
Version comparison charts have also been improved with click-to-highlight and double-click-to-navigate directly to the [evaluations](/concepts/product/version/session/evaluation) list.
## Test Case CSV Export
You can now export your [test cases](/concepts/product/test/case) as CSV files directly from the dashboard. Select specific test cases or export all visible ones — the platform generates a CSV with 22 columns covering input, expected output, strategy, language, scores, and more, then provides a direct download link. Use exported data for offline review, spreadsheet analysis, or integration with external workflows.
## Documentation and Video Guides
The documentation has been significantly expanded with new video walkthroughs and reference pages:
* **Video guides** now cover [specifications](/concepts/product/specification), [versions](/concepts/product/version), [traces](/sdk/tutorials/tracing-agent-operations), [human evaluation](/sdk/tutorials/human-evaluation), [endpoint connection setup](/concepts/product/endpoint-connection), and AI-powered specification generation — each embedded directly in the relevant docs page
* A new [Writing Specifications](/sdk/tutorials/writing-specifications) guide covers how to define specifications for your product — behavioral expectations, policies, and rules that drive metric generation and test creation
* New reference pages for [evaluation types](/concepts/metric/evaluation-types), [evaluation parameters](/concepts/metric/evaluation-parameters), and [endpoint connection configuration](/concepts/product/endpoint-connection-configuration) provide deeper detail on evaluation setup
* A runnable [evaluation demo](/sdk/api/evaluation/run) covers endpoint-based, specification-filtered, and local-agent evaluation modes
## Platform Improvements
* **JSON field matching metric:** A new deterministic [metric](/concepts/metric/json-field-match) that compares top-level JSON fields between actual and expected output, returning the ratio of matching fields as the score. Supports lenient parsing of actual output wrapped in markdown code fences or surrounding text.
* **Endpoint connection validation:** Before running an [evaluation](/concepts/product/version/session/evaluation), the platform now validates that all required [endpoint connections](/concepts/product/endpoint-connection) on the version still exist — preventing runtime failures from deleted connections.
* **Credit warning badge:** A visual indicator in the sidebar shows your organization's credit status — amber when credits are running low, red when exhausted — so you always know where you stand before launching evaluations.
* **Retry defaults:** New [endpoint connections](/concepts/product/endpoint-connection) default to retries enabled with 3 attempts and exponential backoff, improving reliability out of the box.
* **Sidebar overlay:** The navigation sidebar now overlays content on hover instead of pushing the page layout, reducing visual disruption while browsing.
* **Form and table polish:** Clearer validation errors when switching to AI generation mode, a clear button for optional form fields, improved column width handling in tables, better form field reset behavior, and more consistent UI across the dashboard.
## Improved Golden Dataset Generation
The [golden dataset generation](/concepts/product/test/quality-tests) pipeline has been redesigned. You can expect more diverse questions, more accurate source attribution, and fewer duplicated or low-quality entries in your generated test cases.
## Platform Improvements
* **Automatic text truncation:** Long text in table cells across the dashboard is now automatically clamped to two rows. Click any truncated cell to expand and view the full content rendered as Markdown. This replaces inconsistent truncation behavior across tables and keeps the interface clean without hiding information.
* **AI metric saving fix:** An issue that prevented non-admin users from saving [AI-generated metric](/concepts/metric/ai-generation) candidates has been resolved. All team members can now review and accept metric suggestions without encountering permission errors.
* **Evaluation link fix:** The evaluation link generated in the SDK code snippets now navigates to the correct tab, ensuring you land on the evaluations view as expected.
## AI-Powered Metric Creation
You can now generate evaluation [metrics](/concepts/metric) directly from your product's [specifications](/concepts/product/specification). Select one or more policy specifications, and the AI analyzes your product description and specification rules to produce ready-to-use metric candidates — complete with judge prompt, evaluation parameters, tags, and evaluator model. You review each candidate, edit anything that needs adjusting, and save the ones you want. Saved metrics are automatically linked back to their source specification.
Learn more about the full workflow in the [AI Metric Generation](/concepts/metric/ai-generation) documentation.
## Data Catalogs for Behavior Tests
[Behavior tests](/concepts/product/test/behavior-tests) can now be grounded in real data. When creating a generated behavior test, you can upload a **Data Catalog** — a file containing real values like names, IDs, amounts, and dates. The scenario generator injects catalog values into the generated test cases, producing scenarios that reflect your actual data distribution instead of relying on fictional placeholders.
## Simplified Test Creation
The [test](/concepts/product/test) creation form has been reorganized to reduce visual complexity. Optional fields — Custom User Focus, Data Catalog, and Language — are now grouped behind an **Advanced Options** toggle, keeping the main form focused on required fields and essential choices.
## Test Your Endpoint Connection
You can now test your [endpoint connection](/concepts/product/endpoint-connection) directly from the creation and edit forms. A dedicated test panel sends a sample request to your endpoint and displays the raw response, so you can verify connectivity and response format before saving.
## Run Evaluations from Any Level
The [evaluation](/concepts/product/version/session/evaluation) run dialog now supports two new entry points: you can trigger a run from a specific test or from an individual test case, in addition to the existing version, session, and inference result levels. The dialog title, description, and available options adapt to the context you start from. As part of this change, the action has been renamed from "Run Inference" to "Run Evaluation" across the dashboard to better reflect what it does.
## SDK: Full Conversation Access in Custom Metrics
The `CustomScoreEvaluationMetric.measure()` method now receives an optional `inference_results` parameter containing all conversation turns for the current session. This lets custom metrics reason over the full conversation history — not just the latest turn — when computing scores. This also fixes an issue where evaluating sessions with custom metrics only fetched the most recent turn, making multi-turn custom metric evaluation unreliable. See the [custom metrics tutorial](/sdk/tutorials/create-evaluation-with-custom-scores) for usage details.
## Platform Improvements
* **Failed turn tracking:** The evaluation engine now detects which conversation turns failed during a [simulated conversation](/concepts/product/test/case/conversation-simulator) and records them in a new `failedTurns` field on each [evaluation](/concepts/product/version/session/evaluation). Failed turns are visible in the evaluation detail view and included in CSV exports, making it easier to pinpoint where multi-turn conversations break down.
* **Clearer metric validation errors:** When a metric is missing required evaluation parameters, the error message now lists exactly which parameters are absent — instead of the generic "at least one is missing" message.
* **Inference error visibility:** When a direct inference call results in an error, the error text is now displayed directly in the inference result view.
* **Unified form components:** AI-assisted creation forms and table selection dialogs across the dashboard now share a consistent design and interaction model. This also fixes an issue where discarding all AI-generated metric candidates left the review step in a dead end with no navigation options.
## Human Evaluators
You can now bring humans into the evaluation loop. [Human evaluation](/sdk/tutorials/human-evaluation) lets your team members review and annotate inference results directly from the Galtea platform — alongside the automated LLM-as-a-judge scores you already rely on.
To get started, create a [metric](/concepts/metric) with the evaluation type set to **Human**, assign it to the relevant [user group](/concepts/user-group), and invite your evaluators. Each evaluator sees only the inference results assigned to their group and can submit scores through a dedicated annotation interface.
### User Groups
[User Groups](/concepts/user-group) are the organizational layer behind human evaluation. A user group links a set of team members to specific metrics, so you can control who evaluates what. You can manage user groups from the dashboard or programmatically through the [User Group Service](/sdk/api/user-group/service) in the SDK.
## AI-Assisted Specification Configuration
[Specification](/concepts/product/specification) creation is now smarter. AI-assisted configuration has been extended to all specification types — not just policies. When you create a new specification, the system analyzes your description and suggests the test type, test variant, and specification type automatically. You can accept the suggestions with a single click or adjust them before saving.
For **Policy** specifications specifically, the AI also suggests which test type best validates the rule you are defining — helping you choose between `ACCURACY`, `SECURITY`, and `BEHAVIOR` based on the policy's intent.
You can manage specifications from the dashboard or via the [Specification Service](/sdk/api/specification/service) in the SDK.
## Inference Monitoring and Tracing
Direct inference sessions now surface more visibility into what is happening during evaluation runs:
* **Session status:** Each inference session displays a status indicator — `COMPLETED`, `PENDING`, or `FAILED` — so you can quickly identify which sessions need attention.
* **Failed inference visibility:** When a direct inference call fails, the error is now surfaced directly in the session view, rather than being silently logged.
* **Distributed trace collection:** Remote agents can now send traces back to Galtea automatically. Pass the `AgentInput` object to your agent handler, and all `@trace`-decorated calls are linked to the correct session. See [Tracing Agent Operations](/sdk/tutorials/tracing-agent-operations) for the full setup guide.
## Metrics and Evaluation
* **Flexible text metric validation:** Text-based metrics no longer require a strict `answer` key in the response payload, making it easier to integrate custom output formats.
* **Version endpoint validation:** Starting an [evaluation](/concepts/product/version/session/evaluation) now checks that the selected version has a valid endpoint connection configured, catching misconfigurations before the run begins.
* **SDK fix — session custom metrics:** An issue where `actual_output` was always `None` when evaluating [sessions](/concepts/product/version/session) with custom metrics has been resolved.
* **Filter tests by version:** You can now filter the test list by the selected version, making it faster to find the tests relevant to your current evaluation run.
* **Specification-aware validation:** Products are now enriched with their linked specifications before evaluation validation runs, ensuring the evaluation engine has the full context it needs.
## Platform Improvements
* **Report terminology:** Automated reports now use clearer dimension names — **Accuracy** instead of "Quality" and **Security & Safety** instead of "Red Teaming" — matching the terminology you see in the dashboard.
* **Report visualization:** Metric comparison charts now support multi-page groups, so products with more than six metrics per group render correctly without truncation.
* **Evaluation run form:** The evaluation run form layout has been reorganized for a cleaner workflow when configuring new runs.
* **Improved augmentation:** The [data augmentation](/concepts/product/test/data-augmentation) pipeline has received quality improvements to produce more diverse and representative test case variations.
* **SDK improvements:** The SDK now correctly supports JSON request bodies in HTTP DELETE operations and properly escapes JSON string values in inference input templates.
## Documentation
A new [Human Evaluation tutorial](/sdk/tutorials/human-evaluation) walks you through the full workflow — from creating a human metric and setting up user groups to annotating results in the platform. The tutorial includes step-by-step screenshots showing the metric creation form, the evaluator sidebar, and the annotation interface.
## Specifications
[Specifications](/concepts/product/specification) are now live — a structured way to define and test the behavioral expectations of your product.
A Specification represents a single, testable claim about what your product should or should not do. Each specification has a **type** that classifies the expectation:
* **Capability** — a core function the product can perform
* **Inability** — something the product fundamentally cannot do, regardless of user input
* **Policy** — a rule the product must follow, such as refusing certain requests or always adding a disclaimer
You also assign a **test type** (`ACCURACY`, `SECURITY`, or `BEHAVIOR`) that determines how the specification is evaluated.
Specifications replace the legacy free-text Capabilities, Inabilities, and Policies fields that used to live on the Product form. Your behavioral expectations are now discrete, individually traceable, and can be linked directly to the [metrics](/concepts/metric) that validate them.
You can manage specifications from the dashboard or via the [Specification Service](/sdk/api/specification/service) in the SDK.
### AI-Assisted Configuration
Specification creation is AI-assisted. With a single click, the system suggests the specification type, test type, and test variant based on your description — so you can focus on defining what your product should do, not on configuring how it gets tested.
## Improved Report Generation
The automated report generation pipeline has received a set of quality upgrades:
* **Realism analysis:** Before generating narrative content, the engine now runs a realism check on the underlying data to ensure summaries reflect actual conditions rather than statistical artifacts.
* **Pattern analysis:** A new pattern analysis step examines the diversity and distribution of your test results, informing the structure of each report section. You can toggle this per generation run.
## Better Data Augmentation
[Data Augmentation](/concepts/product/test/data-augmentation) now uses a dedicated **diversity model** alongside the main generation pass. Instead of producing variations that converge on the same patterns, augmented test cases are explicitly steered toward different scenarios, phrasings, and edge cases — giving you broader coverage from the same seed data.
A demo video has also been added to the [Data Augmentation documentation](/concepts/product/test/data-augmentation) to walk you through the full workflow.
## Simplified Metrics
The [Metric](/concepts/metric) entity has been streamlined. Legacy fields — including `criteria`, `evaluation_steps`, `test_type`, `user_persona`, and `stopping_criterias` — have been removed from the metric creation form. These were carry-overs from an older architecture that are no longer needed. If you were passing any of these fields through the SDK, you can safely remove them.
## Platform Improvements
* **403 Forbidden page:** Accessing a route without the required permissions now shows a clear, dedicated error page with an explanatory message, rather than a silent redirect or blank screen.
* **Sorting fixes:** Table columns that do not support server-side sorting no longer display a sort indicator, removing a common source of confusion in large result sets.
* **Evaluation prompts:** User prompts in evaluations are now rendered through Jinja2, enabling richer template-based customization.
## Trace Collection During Platform Simulations
When running evaluations via [Direct Inference](/sdk/tutorials/direct-inferences-and-evaluations-from-platform), you can now collect traces from your endpoint and link them back to Galtea automatically.
Add the new `{{ inference_result_id }}` placeholder to your [Endpoint Connection](/concepts/product/endpoint-connection) input template. Your handler receives the ID, passes it to the SDK's `set_context`, and all `@trace`-decorated calls made during that request are automatically linked to the correct inference result in Galtea. See [Collecting Traces During Direct Inference](/sdk/tutorials/direct-inferences-and-evaluations-from-platform#collecting-traces-during-direct-inference) for the full walkthrough.
## Documentation and Examples
Demo videos are now embedded throughout the documentation — covering [test generation](/concepts/product/test), [Accuracy tests](/concepts/product/test/accuracy-tests), [Security & Safety tests](/concepts/product/test/security-tests), [Behavior tests](/concepts/product/test/behavior-tests), [endpoint connection setup](/concepts/product/endpoint-connection), and [data augmentation](/concepts/product/test/data-augmentation) — so you can see each workflow in action without leaving the docs.
## Automated Report Generation
You can now export your analytics data as a comprehensive PDF report directly from the [Galtea dashboard](https://platform.galtea.ai). The report is generated automatically and includes AI-written summaries for each section — covering scope, methodology, product evaluation, and conclusions — alongside dashboard-style visualizations.
## One-Click Test Case Augmentation
Scaling your [Test Cases](/concepts/product/test/case) just got much simpler. If you have a small base of known-good test cases, you can now augment them with a single click directly from the dashboard. Galtea will automatically generate additional test cases based on your existing ones, letting you quickly expand coverage without manually writing each entry.
## New Quality Test Tasks
We have expanded the [Quality Test](/concepts/product/test/quality-tests) configuration with two new task types: **Correction** and **Other**. When selecting **Other**, you can provide a custom task description to better classify your tests. This makes it easier to define quality evaluations that go beyond the predefined categories.
## New Tutorial: Direct Inferences and Evaluations from the Platform
A new step-by-step guide walks you through the full workflow of running inferences and evaluations [directly from the Galtea dashboard](/sdk/tutorials/direct-inferences-and-evaluations-from-platform) — no SDK code required. It covers creating an [Endpoint Connection](/concepts/product/endpoint-connection), attaching it to a Version, running tests, and reviewing results.
## Performance and Reliability
* **Faster Evaluations:**
The [evaluation](/concepts/product/version/session/evaluation) pipeline has been optimized to reduce redundant database lookups. Pre-fetched entities are reused across batch operations, validation order has been improved, and retry logic now groups evaluations by session for significantly faster batch processing.
* **Improved Error Messages:**
The [IOU](/concepts/metric/iou) and [Spatial Match](/concepts/metric/spatial-match) metrics now return clearer error messages specifying the expected JSON formats for bounding box inputs.
* **Jinja2 Template Validation Fix:**
The [conversation simulator](/concepts/product/test/case/conversation-simulator) template validator no longer raises false positives for valid Jinja2 for-loop patterns in JSON templates, such as the common OpenAI-compatible message format.
* **Dashboard Polish:**
Several UI improvements including better dark mode contrast, refined Pill component styling, and a smarter toast notification system that reduces interruptions for frequent users.
## SDK & Simulation Upgrades
We have significantly improved the flexibility of the Galtea SDK to better fit your existing workflows.
* **Unified Simulation Method:**
The [`simulator.simulate`](/sdk/api/simulator/simulate) method has been upgraded to handle both single-turn and multi-turn simulations. You no longer need to switch methods based on the test case complexity.
* **Simplified Agent Integration:**
We've made the `agent` property much more flexible. You can now pass [methods with various signatures](/quickstart#6-run-the-evaluation) directly (including async generators), removing the strict requirement to wrap everything in a specific Agent class.
## New Platform Features
### AI-Powered Product Definition
Bootstrapping a new product context is now faster than ever. You can upload a set of files (documentation, knowledge base, etc.), and Galtea will use AI to automatically [generate a comprehensive Product Definition](https://platform.galtea.ai/product-new) for you.
### Advanced Endpoint Connections
For more complex conversational agents, we have refined how connections are established. You can now [configure distinct behaviors](/concepts/product/endpoint-connection#param-type) for:
1. The **Initialization** of the conversation.
2. The **Conversation** messages.
3. The **Finalization** of the conversation.
### Single Inference Evaluation
You can now evaluate a single inference result directly from the platform UI. This is perfect for spot-checking model behavior without running a full simulation suite.
## Reliability and UX Improvements
* **Graceful Metric Handling:**
Previously, using a metric incompatible with a specific Test Type would throw an error. Now, the system intelligently marks these metrics as **"skipped"** in the results, allowing the rest of your evaluation to proceed uninterrupted.
* **Enhanced Custom Metrics:**
We have deprecated all DeepEval metrics in favor of our own [**Custom Metrics** architecture](/concepts/metric/evaluation-types#ai-evaluation). This transition significantly enhances the reliability, speed, and flexibility of your evaluations.
* **Test Creation Experience:**
The Test creation form has been polished for better usability, making it easier to define and organize your test cases.
* **Platform Stability:**
General improvements to platform stability and reliability to ensure a smoother experience during heavy load.
## Evaluate Single-Turn Interactions
You can now [run evaluations on specific single turns](/sdk/api/evaluation/create#evaluating-a-single-inference-result) (even when directly [creating an Inference Result](/sdk/api/inference-result/create-and-evaluate)), rather than being restricted to evaluating full threads or sessions. This granularity allows for more precise analysis, enabling you to pinpoint and score individual exchanges within a larger conversation context without the noise of the surrounding dialogue.
## Bulk Actions in Tables
We've enhanced table functionality across the dashboard to improve your productivity. You can now select multiple rows to perform actions in bulk, such as deleting multiple [Test Cases](/concepts/product/test/case) or cleaning up old [Sessions](/concepts/product/version/session) simultaneously.
## Stability & Performance
* **Metric Stability:** We've deployed updates to improve the consistency and reliability of several core metrics.
* **Platform Resilience:** Significant backend optimizations have been implemented to ensure platform stability and maintain low latency, even under periods of high load.
## Run Inferences and Evaluations from the Platform
You can now execute inferences and run evaluations directly from the Galtea dashboard. This new capability allows you to quickly test your [Product](/concepts/product) [Versions](/concepts/product/version) and validate performance without writing a single line of code or switching to your IDE.
This streamlined workflow is perfect for:
* Quick sanity checks on new model versions.
* Running specific test cases ad-hoc.
* Validating changes instantly before full-scale testing.
## General Improvements
* **Bug Fixes & UX Polish:** We've addressed various minor bugs and refined the user interface to provide a smoother, more stable experience across the platform.
## Standardized Tracing with OpenTelemetry
Galtea Traces have been upgraded to use the **OpenTelemetry (OTel)** standard. This major infrastructure update aligns our tracing capabilities with industry standards, paving the way for future seamless integrations with your existing observability tools and monitoring stacks. [Read more about traces](/concepts/product/version/session/trace).
## New Models Available
We have expanded the selection of models available for evaluating your products. You can now leverage the latest capabilities of:
* **Claude-Sonnet-4.5**
* **GPT-5.2**, **GPT-5.1**, **GPT-5**, and **GPT-5-mini**
These models can be selected when configuring your [AI Evaluation metrics](/concepts/metric/evaluation-types#ai-evaluation).
## Revamped Onboarding & Development Modes
We've improved the onboarding experience to better match your workflow:
* **Use-Case Based Onboarding:** The setup process is now split between two clear paths: evaluating a product during **Development** or monitoring a product in **Production** with real users.
* **Platform View Modes:** You can now select your preferred view mode in the platform: **Development** or **Production**. This automatically filters the UI to remove unnecessary options, keeping your workspace clean and focused on the task at hand.
## General Improvements
* **Enhanced Forms:** We've standardized and improved forms across the platform for better usability and a more consistent experience.
## AI-Assisted Product Onboarding
Getting started with Galtea got even simpler. We've improved the [Product](/concepts/product) creation flow by simplifying the integration of AI assistance directly into the product creation form. The system helps you draft comprehensive product descriptions, capabilities, and security boundaries, ensuring your product context is perfectly optimized for our test generation engine from day one.
## Enhanced Inference Result Visualization
We have revamped how [Inference Result](/concepts/product/version/session/inference-result) metrics are visualized. The new layout provides a clearer breakdown of performance data, making it easier to correlate specific input/output pairs with their respective scores. This improvement allows for quicker diagnosis of issues within specific conversation turns.
## Documentation Refresh
We have reorganized our documentation to improve discoverability and ease of use. The structure is now streamlined into clearer categories, making it easier to find SDK references, conceptual guides, and tutorials. Check out here the new docs structure [here](/introduction).
## General Improvements & Bug Fixes
* **General Bug Fixes:** Addressed various minor issues to improve platform stability and performance.
* **UI Polish:** Minor visual updates across the dashboard for a more consistent user experience.
## Enhanced Product Creation Experience
We have completely revamped the onboarding flow for new products. A new, intuitive form is now available to help you create AI-based products faster and more efficiently. This update streamlines the initial setup configuration, improving the overall user experience when onboarding new agents or LLM apps.
## Evaluating Tool Usage
As agents become more autonomous, validating *how* they use tools is just as important as the final answer. We have introduced new capabilities to strictly evaluate tool calls:
### New Property in Test Cases
Test cases now accept a new property: `expected_tools`. This allows you to define exactly which tools an agent should invoke during a specific test scenario. [Read more about Test Case structure](/concepts/product/test/case).
### New Evaluation Parameters
To support this validation, the evaluation process now accepts two specific parameters regarding tool usage:
* `tools_used`: The actual list of tools invoked by the model.
* `expected_tools`: The ground truth list of tools that should have been used.
## New Metric: Tool Correctness
We have added a specialized metric to our library: **Tool Correctness**.
This metric automatically compares the `tools_used` against the `expected_tools` to determine if the agent selected the right functions to solve the user's problem. This is critical for ensuring reliability in agentic workflows. [See metric details](/concepts/metric/tool-correctness).
## Full Visibility with Agent Tracing
You can now add **[Traces](/concepts/product/version/session/trace)** to your [Sessions](/concepts/product/version/session) and [Inference Results](/concepts/product/version/session/inference-result). This unlocks deep observability into your AI agents, allowing you to understand exactly what they are doing at each step of execution—not just the final output.
Traces capture internal operations such as:
* **Tool Calls:** API requests, calculations, or data fetching.
* **Retrieval Steps:** Vector database searches and RAG context retrieval.
* **Chain Orchestration:** Internal routing and decision-making logic.
* **LLM Invocations:** Prompts sent to underlying models.
With the new **[Trace Service](/sdk/api/trace/service)** in the SDK, you can visualize the full execution tree, debug errors, analyze latency for specific operations, and audit agent behavior with granular detail.
We've also added a comprehensive guide on how to instrument your code. [Read the Tracing Tutorial](/sdk/tutorials/tracing-agent-operations) to get started.
## Granular Credit Consumption and Transparency
We've updated our credit system to provide more granularity and transparency. You can now see exactly where your credits are being consumed, giving you detailed visibility into your usage patterns and helping you manage resources more effectively.
## Platform Robustness and Security
We've improved the overall robustness of the platform to ensure higher uptime and reliability. Additionally, we have increased safety by implementing rate limiting on our authentication endpoints, providing better protection against abuse and unauthorized access attempts.
## Improved SDK File Validation
We've enhanced file validation within the SDK. When uploading files—for example, when creating a [Test](/sdk/api/test/create) using `test_file_path` or `ground_truth_file_path`—you will now receive stricter validation and clearer, more descriptive error messages to help you debug issues faster.
## Simplified and More Powerful Metric Creation with Partial Prompts
We've upgraded how custom judge [Metrics](/concepts/metric) are created by introducing a new **Partial Prompt** method. This approach simplifies the process by letting you focus on the core evaluation logic while Galtea handles the final prompt construction. This not only makes creating custom metrics faster but also significantly increases the quality and consistency of the evaluation results. [Learn more about the new method](/concepts/metric#creating-custom-metrics-via-ai-evaluation).
## Clearer Pass/Fail with Binarized Metric Scores
To provide more decisive validation, we've updated three of our key metrics to return a clear binary score (0 for fail, 1 for pass). This change eliminates ambiguous 0.5 scores, making it easier to determine success or failure for critical test cases. The updated metrics are:
* [Factual Accuracy](/concepts/metric/factual-accuracy)
* [Jailbreak Resilience](/concepts/metric/jailbreak-resilience)
* [Data Leakage](/concepts/metric/data-leakage)
## More Robust Quality Test Generation
When creating [Quality Tests](/concepts/product/test/quality-tests) from the dashboard, we now validate your uploaded files to ensure they are in the expected format. This proactive check helps prevent errors during test generation, leading to a smoother and more reliable workflow.
## Platform Enhancements
We're always working to improve the core platform experience. This week's updates include:
* **Improved Performance:** We've optimized our underlying LLM usage for better concurrency, scalability, and robustness, resulting in a faster and more reliable platform.
* **Enhanced Email Validation:** The process for validating user emails has been made more robust to ensure better security and deliverability.
## Flexible Sign-In with Google, GitHub, and GitLab
You can now sign in to Galtea using your Google, GitHub, or GitLab accounts! This makes accessing your workspace faster and more secure, providing a seamless single sign-on (SSO) experience alongside our traditional email and password login.
## Simplified Metric Versioning with "Legacy" Tag
We've introduced a new way to handle updated [Metrics](/concepts/metric). Galtea metrics might be marked as "Legacy," allowing us to improve them over time with the same name. This approach simplifies versioning by removing the need to embed version numbers in metric names, ensuring that your historical evaluations remain linked to the correct metric version.
## Test Generation Transparency and Enhancements
We're bringing more transparency and efficiency to our test generation process:
* **See What's Under the Hood:** The models used to generate a [Test](/concepts/product/test) are now listed in the test's details section, providing clearer insight into how your test cases were created.
* **Faster Test Generation:** We've significantly sped up the processing of multiple files within ZIP archives when generating tests from a knowledge base.
* **Smarter Test Case Handling:** The engine for generating [Quality Tests](/concepts/product/test/quality-tests) now handles the `max_test_cases` parameter more effectively.
## Fixes & Improvements
* Fixed a bug where file names from ZIP archives were not displayed correctly in the UI.
* Resolved an issue that prevented the processing of ZIP files containing two files with the same name (e.g., `folder1/file.txt` and `folder2/file.txt`).
## A Fresh New Look for the Dashboard
We've polished the dashboard with a subtle but significant UI refresh. Expect updated colors, clearer icons, and more consistent dialogs designed to create a smoother, more intuitive, and visually consistent workflow.
## Enhanced Security and Performance
We've bolstered the platform's security with improved user authentication mechanisms to better protect your data. Alongside this, we've enhanced how we handle concurrent requests, leading to better performance and reliability, especially during high-traffic periods.
## Smarter & Faster Red Teaming Generation
Creating [Red Teaming tests](/concepts/product/test/red-teaming-tests) is now better than ever. We've improved language control during generation for more robust and relevant tests, even when no specific language is provided. Plus, we've optimized the generation process for significantly faster test creation, allowing you to build robust security evaluations more efficiently.
## Introducing Partial Prompts: Create Custom Metrics Faster
We're making it easier than ever to create custom judge metrics with our new **Partial Prompt** evaluation type. Now, you can focus solely on defining your evaluation criteria and simply select the data parameters you need—like `input` or `retrieval_context`. Galtea automatically constructs the full, correctly formatted judge prompt for you, saving you time and reducing complexity.
This new approach streamlines the creation of powerful, tailored [Metrics](/concepts/metric), allowing you to concentrate on what matters most: the evaluation logic itself.
## Quality Generator with Improved Language Handling
The LLM language confusion is now patched for the quality generator. Languages are formally detected for the whole document, and test cases are ensured to be generated with the correct language. This ensures more accurate and contextually appropriate test case generation across different language inputs.
## Improved Long Context Processing for Quality Test Cases
Our quality generator now intelligently processes long documents, ensuring optimal context is provided for the best quality generation of test cases. This enhancement streamlines the evaluation process and improves the reliability of generated test cases for comprehensive quality assessments.
## Platform Experience Enhancements
We've also rolled out several quality-of-life improvements and bug fixes for a smoother experience. A notable update is that tables across the platform now refresh instantly after you delete an item with a right-click, making data management more fluid and intuitive.
## Define Product Behavior with Policies
[Products](/concepts/product) now accept a set of policies describing how they should behave in specific situations. This allows you to set explicit rules, such as requiring a disclaimer for financial advice or defining standard responses for out-of-scope questions, to ensure consistent and safe model behavior.
## Advanced Red Teaming Strategies
We've added five new, sophisticated [red teaming strategies](/concepts/product/test/red-teaming-strategies) to help you uncover more complex vulnerabilities. These strategies use your **Product Description** to create highly contextual and evasive prompts:
* **Persuasive Content:** Disguises malicious requests as legitimate business tasks.
* **Creative Writing:** Reframes harmful prompts as creative exercises.
* **Data Analysis:** Hides malicious intent within analytical or data-generation tasks.
* **Bait and Switch:** Lowers model defenses with benign queries before introducing the adversarial prompt.
* **Empathetic Framing:** Uses emotional manipulation to pressure the model into unsafe compliance.
## Smarter Test Generation Engine
Our test generation capabilities have been significantly upgraded for both Quality and Red Teaming tests:
* **New Quality Test Engine:** The engine for generating [Quality Tests](/concepts/product/test/quality-tests) has been rebuilt to produce higher-quality test cases. It can now also incorporate and validate the new product policies you define.
* **Faster Red Teaming Generation:** The generation process for [Red Teaming Tests](/concepts/product/test/red-teaming-tests) is now more reliable and significantly faster, allowing you to build robust security evaluations more efficiently.
## Platform Usability and Performance
We've rolled out several enhancements to improve your workflow and the platform's reliability:
* **Deletion Confirmations:** To prevent accidental data loss, the platform will now ask for confirmation before deleting key entities like Test Cases, Evaluations, and Versions.
* **Improved Table Experience:** The performance and user experience of data tables and their filters have been optimized for a smoother, more responsive interface.
## Platform Simplification and SDK v3.0
We've undertaken a major simplification of our core concepts to make the platform more intuitive. `Evaluation Tasks` are now simply **[Evaluations](/concepts/product/version/session/evaluation)**, and `Metric Types` have been renamed to **[Metrics](/concepts/metric)**. The old parent `Evaluations` entity has been removed entirely. These changes streamline the workflow and clarify the relationship between different parts of Galtea.
To support this, we've released a new major version of our SDK.
## New Conversational Metrics
We're excited to introduce two new metrics designed specifically for evaluating conversational AI:
* **[User Objective Accomplished](/concepts/metric/user-objective-accomplished):** Evaluates whether the user's stated goal was successfully and correctly achieved during the conversation.
* **[User Satisfaction](/concepts/metric/user-satisfaction):** Assesses the user's overall experience, focusing on efficiency and sentiment, to gauge their satisfaction with the interaction.
## Enhanced Test Case Feedback and Management
Improving test quality is now a more collaborative process. When upvoting or downvoting a **[Test Case](/concepts/product/test/case)**, you can now add a `user_score_reason` to provide valuable context for your feedback.
Additionally, you can now filter test cases by their score directly via the SDK using the `user_score` parameter in the **[`test_cases.list()`](/sdk/api/test-case/list)** method.
## Dashboard and SDK Usability Improvements
We've rolled out several updates to make your workflow smoother and more efficient:
* **Improved Dashboard Navigation:** Navigating between related entities like Tests, Test Cases, and Evaluations is now more intuitive. We've also adjusted table interactions—you can now single-click a row to select and copy text without navigating away. To see an entity's details, simply right-click the row.
* **Efficient Batch Fetching in SDK:** The SDK now allows you to fetch objects by providing a list of IDs (e.g., fetching all [Test Cases](/sdk/api/test-case/list) for a list of Test IDs at once), significantly improving usability for batch operations.
## General Improvements
This release also includes numerous performance optimizations and minor UI/UX enhancements across the entire platform to provide a faster and more polished experience.
## Create Conversational Scenarios from Your Quality Tests
You can now generate comprehensive **[Scenario Based Tests](/concepts/product/test/scenario-tests)** directly from your existing **[Quality Tests](/concepts/product/test/quality-tests)**. This powerful feature allows you to transform your single-turn, gold-standard test cases into realistic, multi-turn conversational scenarios, significantly accelerating the process of evaluating your AI's dialogue capabilities.
## New Deterministic Metric: URL Validation
We've added the **[URL Validation](/concepts/metric/url-validation)** metric to our deterministic evaluation suite. It ensures that all URLs in your model’s output are safe, properly formatted with HTTPS, and resolvable. It includes strict validation and SSRF protection, making it essential for any application that generates external links.
## Platform Enhancements and Performance Boost
We've rolled out major performance improvements across the platform, with a special focus on the analytics views, making them faster and more responsive. This update also includes several minor visual fixes, such as ensuring icons render correctly in different themes, to provide a smoother and more polished user experience.
## Consistent Sorting and Enhanced Navigation
We've made significant usability improvements to the platform. Table sorting is now more consistent and persistent; you can navigate away, refresh the page, and your chosen sort order will remain, ensuring a smoother workflow. Additionally, navigating from an [Evaluation](/concepts/product/version/session/evaluation) is easier than ever. You can now right-click on any evaluation in the table or use the dropdown menu in the details page to jump directly to the related [Test Case](/concepts/product/test/case) or [Session](/concepts/product/version/session).
## Upvote and Downvote Test Cases
To help teams better curate their test suites, we've introduced a voting system for [Test Cases](/concepts/product/test/case). You can now upvote or downvote test cases directly from the Dashboard, providing a quick feedback loop on test case quality and relevance.
## SAML SSO Authentication
Organizations can now enhance their security by configuring SAML SSO for authentication. This allows for seamless and secure access to the Galtea platform through your existing identity provider.
## Improved Misuse Resilience Metric
The **[Misuse Resilience](/concepts/metric/misuse-resilience)** metric has been enhanced to accept the full product context. This allows for a more accurate and comprehensive evaluation of your model's ability to resist misuse by leveraging a deeper understanding of your product's intended capabilities and boundaries.
## New Analytics Filter
The analytics page now includes a filter for Test Case language. This allows you to narrow down your analysis and gain more precise insights into the performance of your multilingual models.
## Confidence Scores for Generated Test Cases
We're introducing confidence scores for all generated [Test Cases](/concepts/product/test/case). This new feature provides a clear indicator of the quality and reliability of each test case, helping you better understand your test suites and prioritize human review efforts. Higher scores indicate greater confidence in the test case's relevance and accuracy.
## Simplified and More Flexible Metric Creation
Creating [Metrics](/concepts/metric) is now more intuitive and flexible. Metrics are now directly linked to a specific test category (`QUALITY`, `RED_TEAMING`, or `SCENARIOS`), which simplifies the creation process by tailoring the available parameters to the relevant test category. This change makes it easier to define metrics that are perfectly aligned with your evaluation goals. See the updated [creation guide](/sdk/api/metric/create) for more details.
## Custom Judges for Conversational Evaluation
You can now create your own **[Custom Judges](/concepts/metric/evaluation-types#ai-evaluation)** specifically for evaluating multi-turn conversations. This powerful feature allows you to define complex, stateful evaluation logic that assesses the entire dialogue, enabling you to measure nuanced aspects like task completion, context retention, and persona adherence across multiple turns. Learn more in our guide to [evaluating conversations](/sdk/tutorials/evaluating-conversations).
## Conversation Simulator Enhancements
We've added more control and realism to the [Conversation Simulator](/concepts/product/test/case/conversation-simulator):
* **Agent-First Interactions**: The `simulate` method now includes an `agent_goes_first` parameter, allowing you to test scenarios where the agent initiates the conversation. [See the SDK docs](/sdk/api/simulator/simulate).
* **Selectable Conversation Styles**: When creating a [Scenario Based Test](/concepts/product/test/scenario-tests), you can now choose a conversation style (`written` or `spoken`). This influences the tone and formality of the simulated user's dialogue, enabling more realistic testing. This is available under the `strategies` parameter in the [test creation method](/sdk/api/test/create).
## More Realistic Conversation Simulations
Our **[Conversation Simulator](/concepts/product/test/case/conversation-simulator)** has been significantly upgraded to generate more realistic and human-like interactions. The simulator's evaluation of stopping criteria is now more precise, ensuring that your multi-turn dialogue tests conclude under the correct conditions and provide more accurate insights into your AI's conversational abilities.
Additionally, conversations generated during simulations are now linked to the specific [product version](/concepts/product/version) they were created with, allowing for better tracking and traceability.
To enable version tracking for simulated conversations, please upgrade to Galtea SDK version `1.8.1` or higher.
## Bulk Document Uploads for Test Generation
You can now upload multiple files at once using **ZIP archives** to generate comprehensive [test suites](/concepts/product/test). This feature streamlines the process of creating tests from large document collections, saving you time and effort when building out your evaluation scenarios. For more details, see the [test creation documentation](/sdk/api/test/create).
## Streamlined Onboarding Experience
Getting started with a new [product](/concepts/product) is now faster than ever. When you register a new product in the Galtea dashboard, an initial [version](/concepts/product/version) is now created automatically. This simplification streamlines the setup process and helps you get to your first evaluation more quickly.
## Enhanced Document Processing
We've improved our backend for processing documents used in test generation. This enhancement leads to more accurate and relevant test cases being created from your knowledge bases, improving the overall quality of your evaluations.
## Fresh New Look: Galtea Rebranding is Live!
We're excited to unveil Galtea's complete visual transformation! Our new branding includes a refreshed logo, updated slogan, and a modern look across the entire platform. Experience the new design on our [website](https://galtea.ai/) and explore the updated [dashboard interface](https://platform.galtea.ai/). This rebrand reflects our continued commitment to providing a more intuitive and visually appealing user experience.
## Enhanced Analytics with Improved Version Comparisons
The Analytics views have received significant improvements to make data analysis more efficient and user-friendly. Key enhancements include:
* **Streamlined Chart Comparisons:** Charts now support easier side-by-side version comparisons, helping you quickly identify performance differences across iterations.
* **Optimized Filter Layout:** Filters no longer occupy valuable screen real estate, giving you more space to focus on your analytics data and insights.
* **Improved Visual Clarity:** Enhanced data presentation makes it easier to interpret results and make informed decisions about your AI systems.
These improvements build upon our existing [analytics capabilities](/sdk/tutorials/create-evaluation) to provide a more seamless evaluation experience.
## Strengthened Authentication and Security
We've implemented more secure and robust authentication mechanisms across the platform. These behind-the-scenes security enhancements provide better protection for your data and ensure reliable access to your Galtea workspace, giving you peace of mind when working with sensitive AI evaluation data.
## Advanced PDF Knowledge Extraction
Our document processing capabilities have been significantly improved for PDF files. The enhanced extraction algorithms now provide more accurate and comprehensive knowledge retrieval from PDF documents, making it easier to create relevant [test cases](/concepts/product/test) and evaluation scenarios from your existing documentation and knowledge bases.
## TestCase Scenarios Creation and Edition via Dashboard
You can now create and edit **TestCase Scenarios** directly through the Dashboard interface. This streamlined workflow allows you to define complex multi-turn conversation scenarios without leaving the platform, making it easier to set up comprehensive testing workflows for your conversational AI systems.
## Enhanced TestCase Dashboard with Test-Specific Columns
We've improved the TestCase Dashboard with smarter table displays that now only show columns relevant to each specific test. This reduces visual clutter and makes it easier to focus on the information that matters most for your particular testing scenario, whether you're working with single-turn evaluations, conversation scenarios, or other tests.
## New Judge Template Selector for Judge Metrics
When creating Judge metrics, you can now select from pre-built judge templates to accelerate your metric setup process. This feature provides a starting point for common evaluation patterns while still allowing full customization of your evaluation prompts and scoring logic.
## Infrastructure Improvements for Event Resilience
We've enhanced our event handling infrastructure to provide better resilience against unexpected system events. These improvements help ensure that your tests and evaluations are preserved and continue running smoothly, even during system maintenance or unexpected interruptions.
## Create Custom Judges via Your Own Prompts
You can now define **Custom Judge** metrics by crafting your own evaluation prompts during metric creation. This allows you to encode your domain-specific rubrics, product constraints, or behavioral guidelines directly into the metric—giving you precise control over how LLM outputs are assessed. Simply write your prompt, specify the scoring logic, and Galtea will leverage LLM-as-a-judge techniques to evaluate outputs according to your standards.
## New Deterministic Metrics
Four new deterministic metrics are now available:
* **Text Similarity**: Quantifies how closely two texts resemble each other using character-level fuzzy matching.
* **Text Match**: Checks if generated text is sufficiently similar to a reference string, returning a pass/fail based on a threshold.
* **Spatial Match**: Verifies if a predicted box aligns with a reference box using IoU scoring, producing a pass/fail result.
* **IoU (Intersection over Union)**: Computes the overlap ratio between predicted and reference boxes for alignment and detection tasks.
These new metrics complement our existing evaluation suite and are perfect for scenarios requiring deterministic, rule-based scoring.
## Dashboard Redesign: Second Iteration
We've launched the second iteration of our redesigned dashboard with a refreshed visual language focused on clarity and usability. Key improvements include:
* **Modern Forms**: Forms have been modernized to provide a more intuitive and visually appealing user experience, as well as giving a more professional look to the Dashboard.
This redesign represents our commitment to delivering a more intuitive and visually appealing user experience while maintaining the powerful functionality you rely on.
## Expanded Evaluator Model Support
We've added support for more evaluator models to enhance your evaluation capabilities:
* **Gemini-2.5-Flash**: Google's latest high-performance model optimized for speed and accuracy
* **Gemini-2.5-Flash-Lite**: A lightweight variant offering faster processing with efficient resource usage
* **Gemini-2.0-Flash**: Google's established model providing reliable evaluation performance
You can now [link these models to specific Metrics](/concepts/metric) for consistent evaluation results and leverage specialized models for different evaluation scenarios. Learn more about configuring evaluator models in our [SDK documentation](/sdk/api/metric/service).
## Enhanced Conversation Simulation
Testing conversational AI just got more powerful with two major improvements:
* **Visible Stopping Reasons**: You can now see exactly why simulated conversations ended in the dashboard, providing crucial insights into dialogue flow and helping you identify areas for improvement.
* **Custom User Persona Definitions**: Create highly specific user personas when generating [Scenario Based Tests](/concepts/product/test/scenario-tests). Define detailed user backgrounds, goals, and behaviors to test how your AI handles diverse user interactions more effectively.
These enhancements work seamlessly with our [Conversation Simulator](/concepts/product/test/case/conversation-simulator) to deliver more realistic and insightful testing scenarios.
## Classic NLP Metrics Now Available
We've expanded our metric library with three essential deterministic metrics for precise text evaluation:
* **[BLEU](/concepts/metric/bleu)**: Measures n-gram overlap between generated and reference text, ideal for machine translation and constrained generation tasks.
* **[ROUGE](/concepts/metric/rouge)**: Evaluates summarization quality by measuring the longest common subsequence between candidate and reference summaries.
* **[METEOR](/concepts/metric/meteor)**: Assesses translation and paraphrasing by aligning words using exact matches, stems, and synonyms for more nuanced evaluation.
These classic metrics complement our existing evaluation suite and are perfect for scenarios requiring deterministic, rule-based scoring.
## Enhanced Red Teaming with Jailbreak Resilience v2
Security testing gets an upgrade with **[Jailbreak Resilience v2](/concepts/metric/jailbreak-resilience-v2)**, an improved version of our jailbreak resistance metric. This enhanced evaluation provides more comprehensive assessment of your model's ability to resist adversarial prompts and maintain safety boundaries across various attack vectors.
## Dashboard Redesign: First Iteration
We've launched the first iteration of our redesigned dashboard with a refreshed visual language focused on clarity and usability. Key improvements include:
* **Modern Typography**: Cleaner, more readable text throughout the platform
* **Refined UI Elements**: Updated buttons, cards, and form elements with reduced rounded corners for a more contemporary look
* **Streamlined Tables**: Enhanced data presentation with improved content layout
* **Updated Login Experience**: A more polished and user-friendly authentication flow
This redesign represents our commitment to delivering a more intuitive and visually appealing user experience while maintaining the powerful functionality you rely on.
## Improved SDK Documentation
We've enhanced our [SDK documentation](/sdk/api/metric/create#param-evaluator-model-name) with clearer guidance on defining evaluator models for metrics, making it easier to configure and customize your evaluation workflows.
## Test Your Chatbots with Simulated Conversations
It is now possible to generate tests that simulate realistic, multi-turn user interactions. Our new **[Scenario Based Tests](/concepts/product/test/scenario-tests)** allow you to define user personas and goals to evaluate how well your conversational AI handles complex dialogues.
This feature is powered by the **[Conversation Simulator](/concepts/product/test/case/conversation-simulator)**, which programmatically runs these scenarios to test dialogue flow, context handling, and task completion. Get started with our new **[Simulating User Conversations tutorial](/sdk/tutorials/simulating-conversations)**.
## New Red Teaming Metric: Data Leakage
We've added the **[Data Leakage](/concepts/metric/data-leakage)** metric to our suite of Red Teaming evaluations. This metric assesses whether your LLM returns content that could contain sensitive information, such as PII, financial data, or proprietary business data. It is crucial for ensuring your applications are secure and privacy-compliant.
## Enhanced Metric Management
We've rolled out several improvements to make metric creation and management more powerful and intuitive:
* **Link Metrics to Specific Models**: You can now associate a [Metric](/concepts/metric) with a specific evaluator model (e.g., "GPT-4.1"). This ensures consistency across evaluation runs and allows you to use specialized models for certain metrics.
* **Simplified Custom Scoring**: We've introduced a more streamlined method for defining and calculating scores for your own deterministic metrics using the `CustomScoreEvaluationMetric` class. This makes it easier to integrate your custom, rule-based logic directly into the Galtea workflow. Learn more in our [tutorial on evaluating with custom scores](/sdk/tutorials/create-evaluation-with-custom-scores).
## Support for Larger Inputs and Outputs
To better support applications that handle large documents or complex queries, we have increased the maximum character size for evaluation inputs and outputs to 250,000 characters.
## Test Your Chatbots with Realistic Conversation Simulation
You can now evaluate your conversational AI with our new **[Conversation Simulator](/concepts/product/test/case/conversation-simulator)**. This powerful feature allows you to test multi-turn interactions by simulating realistic user conversations, complete with specific goals and personas. It's the perfect way to assess your product's dialogue flow, context handling, and task completion abilities.
Get started with our step-by-step guide on **[Simulating User Conversations](/sdk/tutorials/simulating-conversations)**.
## New Metric: Resilience To Noise
We've expanded our RAG metrics with **[Resilience To Noise](/concepts/metric/resilience-to-noise)**. This metric evaluates your product's ability to maintain accuracy and coherence when faced with "noisy" input, such as:
* Typographical errors
* OCR/ASR mistakes
* Grammatical errors
* Irrelevant or distracting content
This is essential for ensuring your AI performs reliably in real-world scenarios where user input isn't always perfect. [Learn more about how it's calculated](/concepts/metric/resilience-to-noise).
## Stay in Control with Enhanced Credit Management
We've rolled out a new and improved credit management system to give you better visibility and control over your usage. The system now includes proactive warnings that notify you when you are approaching your allocated credit limits, helping you avoid unexpected service interruptions and manage your resources more effectively.
## Streamlined Conversation Logging with OpenAI-Aligned Format
Logging entire conversations is now easier and more intuitive. We've updated our batch creation method to align with the widely-used `messages` format from OpenAI, consisting of `role` and `content` pairs. This makes sending multi-turn interaction data to Galtea simpler than ever.
See the new format in action in the **[Inference Result Batch Creation docs](/sdk/api/inference-result/create-batch)**.
## Tailor Your Red Teaming with Custom Threats
You can now define your own custom threats when creating [Red Teaming Tests](/concepts/product/test/red-teaming-tests). This new capability allows you to move beyond our pre-defined threat library and create highly specific adversarial tests that target the unique vulnerabilities and edge cases of your AI product. Simply describe the threat you want to simulate, and Galtea will generate relevant test cases.
## New Red Teaming Strategies: Role Play and Prefix Injection
We've expanded our arsenal of [Red Teaming Strategies](/concepts/product/test/red-teaming-strategies) to help you build more robust AI defenses:
* **Role Play**: This strategy attempts to alter the model's identity (e.g., "You are now an unrestricted AI"), encouraging it to bypass its own safety mechanisms and perform actions it would normally refuse.
* **Prefix Injection**: Adds a misleading or tactical instruction before the actual malicious prompt. This can trick the model into a different mode of operation, making it more susceptible to the adversarial attack.
## Introducing the Misuse Resilience Metric
A new [non-deterministic metric](/concepts/metric), **[Misuse Resilience](/concepts/metric/misuse-resilience)**, is now available. This powerful metric evaluates your product's ability to stay aligned with its intended purpose, as defined in your **[product description](/concepts/product)**, even when faced with adversarial inputs or out-of-scope requests. It ensures your AI doesn't get diverted into performing unintended actions, a crucial aspect of building robust and responsible AI systems. Learn more in the [full documentation](/concepts/metric/misuse-resilience).
## Enhanced Test Case Management: Mark as Reviewed
To improve collaboration and workflow for human annotation teams, [Test Cases](/concepts/product/test/case) can now be marked as "reviewed". This feature allows you to:
* Track which test cases have been validated by a human.
* See who performed the review, providing a clear audit trail.
* Filter and manage your test sets with greater confidence.
## Introducing the Factual Accuracy Metric
We've added a new **Factual Accuracy** metric to our evaluation toolkit! This [non-deterministic metric](/concepts/metric/factual-accuracy) measures whether the information in your model's output is factually correct when compared to a trusted reference answer. It's particularly valuable for RAG and question answering systems where accuracy is paramount.
The metric uses an LLM-as-a-judge approach to compare key facts between your model's output and the expected answer, helping you catch hallucinations and ensure your AI provides reliable information to users. [Read the full documentation here](/concepts/metric/factual-accuracy).
## Enhanced Red Teaming with New Attack Strategies
Our red teaming capabilities just got more sophisticated! We've added two powerful new [attack strategies](/concepts/product/test/red-teaming-strategies):
* **Biblical Strategy**: Transforms adversarial prompts into biblical/ancient scripture style using poetic and symbolic language to disguise malicious intent while preserving meaning.
* **Math Prompt Strategy**: Encodes harmful requests into formal mathematical notation using group theory concepts to obscure the intent from standard text analysis.
These strategies join our existing arsenal to help you test your AI's defenses against increasingly creative attack vectors that real-world adversaries might use. [See all available red teaming strategies](/concepts/product/test/red-teaming-strategies).
## Smarter Red Teaming Test Generation
We've significantly improved how [red teaming tests](/concepts/product/test/red-teaming-tests) are generated. Our system now takes even more factors into account when creating adversarial test cases:
* **Product-Aware Generation**: Tests are now more precisely tailored to your specific product's strengths, weaknesses, and operational boundaries.
* **Context-Sensitive Attacks**: The generation process better understands your product's intended use cases to craft more relevant and challenging scenarios.
* **Enhanced Threat Modeling**: Our algorithms now consider a broader range of factors when determining the most effective attack vectors for your particular AI system.
This means your [red teaming tests](/concepts/product/test/red-teaming-tests) will be more effective at uncovering real vulnerabilities and edge cases specific to your product.
## Better Metric Source Visibility and Management
Understanding where your metrics come from is now easier than ever! We've enhanced the platform to provide clearer visibility into metric sources:
* **Source Classification**: All metrics are now clearly labeled with their source - whether they're from established frameworks, custom Galtea implementations, or other origins.
* **Enhanced Filtering**: You can now filter metrics by their source to quickly find the evaluation criteria that best fit your needs.
* **Improved Descriptions**: Metric descriptions now include more detailed information about their origins and implementation, with links to relevant documentation.
The three main metric sources you'll see are:
* **Galtea**: Custom metrics designed specifically for your needs, like our new Factual Accuracy metric
* **G-Eval**: Framework-based metrics that use evaluation criteria or steps for assessment
* **Established Frameworks**: Metrics adapted from proven evaluation libraries and methodologies
This transparency helps you make more informed decisions about which metrics to use for your specific evaluation needs.
## Generate Quality Tests from Examples
You can now create Quality Tests directly from your own examples using the new **Few Shots** parameter. This makes it easier to tailor tests to your specific use cases and ensure your models are evaluated on the scenarios that matter most. [Learn more about test creation](/sdk/api/test/create).
## Metric Tags
[Metrics](/concepts/metric) now support **tags** for easier classification and discovery. Quickly find and organize metrics relevant to your projects. [See all metrics](/concepts/metric).
## Enhanced Product Details
[Products](/concepts/product) now include new detail fields:
* **Capabilities**: What your product can do.
* **Inabilities**: Known limitations.
* **Security Boundaries**: Define the security scope and constraints.
These additions help you document and communicate your product's strengths and boundaries more clearly. [Read about product details](/concepts/product).
## Improved Q\&A Generation
Question-answer pairs are now generated with improved accuracy and clarity, thanks to better text filtering and processing.
## New Guide: Setting Up Your Product Description
We've created a comprehensive guide to help you set up your product descriptions effectively. This guide covers best practices, examples, and tips to ensure your product is presented in the best light.
Check it out [here](https://docs.galtea.ai/sdk/tutorials/writing-specifications)
## General Improvements
We've made various bug fixes and UX/UI improvements across the Dashboard, SDK, and more, making your experience smoother and more reliable.
## Major Platform Overhaul
We've been hard at work reorganizing and expanding the Galtea platform to handle more use cases and prepare for exciting future features. This release brings significant improvements to the [dashboard](https://platform.galtea.ai/), [SDK](/sdk/installation), and test generation.
## Dashboard Enhancements
* **Reorganized [Version](/concepts/product/version), [Test](/concepts/product/test), and [Evaluation](/concepts/product/version/session/evaluation) Views:**\
Detailed views have been streamlined and improved for clearer insights.
* **New [Sessions](/concepts/product/version/session) Visualizations:**\
Easily organize and navigate conversations through our new Sessions feature.
* **Evaluations Visualization Removed:**\
The dashboard now focuses on **Sessions** and **Evaluations** as the primary elements.
* **Better Filters Across Tables:**\
Quickly find what you need with improved filtering capabilities on the dashboard.
* **General Bug Fixes & UX Improvements:**\
Enjoy smoother interactions, clearer tooltips, and more intuitive code snippets.
## SDK v2 Released
The new Galtea SDK v2 is here! It includes breaking changes to simplify workflows and add session support.
* **Implicit Session Creation:**\
Sessions are created automatically when needed for evaluations.
* **Repurposed [evaluations.create()](/sdk/api/evaluation/create):**\
The old method is replaced by `create_single_turn()` for test-based evaluations, while `create()` now exclusively handles [session-based](/sdk/api/session/service) evaluations.
* **New [evaluations.create\_single\_turn()](/sdk/api/evaluation/create-single-turn) Method:**\
Use this for single-turn [test cases](/sdk/api/test-case/service). It now requires `version_id` instead of `evaluation_id`.
* **Simplified [Version Creation](/sdk/api/version/create):**\
The `galtea.versions.create()` method now accepts all properties directly, no need for an `optional_props` dictionary.
* **[Sessions Support](/sdk/api/session/service):**\
Group multiple inference results under a single session for better multi-turn tracking using `galtea.sessions.create()`.
## Improved Test Case Generation
* **Smarter Test Coverage:**\
[Test cases](/concepts/product/test/case) are now distributed more intelligently across your documents for better coverage based on the number of questions you choose to generate.
* **Single Threat per Red Teaming Test:**\
Red Teaming Tests now only allow a single threat per test, ensuring clearer results.
## Improved Test Generation
Our test generation capabilities have been significantly upgraded:
* **Versatile Red Teaming:** Red Teaming tests are now more powerful, allowing you to employ multiple [attack strategies](/concepts/product/test/red-teaming-strategies) to thoroughly probe your AI's defenses.
* **Better Synthetic Data:** We've made general improvements to the quality of synthetic data generation, ensuring your tests are more effective and realistic.
## Code Snippets Now Available on the Dashboard
We're making it easier than ever to integrate Galtea into your development process!
* **Simplified Evaluation Setup:** The "Create Evaluation" form on the dashboard has been replaced with a convenient code snippet. Simply copy and paste it directly into your project to get started.
* **Streamlined Creation:** Similarly, a new code snippet for "Create Evaluation" is now available on the dashboard, simplifying how you send evaluation data to Galtea. You can easily copy and paste this into your project.
## Usability Improvements
We've also rolled out several usability enhancements based on your feedback:
* **Enhanced Readability in Tables:** Table cells now correctly render line breaks, making it easier to view multi-line content and detailed information at a glance.
* **Controlled Test Case Generation:** To ensure optimal performance and manageability, the maximum number of test cases automatically generated for a single test from a knowledge base is now capped at 1000.
## Streamlined Onboarding and Quicker Starts
We've revamped the platform onboarding! It's now more visually intuitive and to help new users get evaluating in no time, we now provide a default [Metric](/concepts/metric) and a default [Test](/concepts/product/test). This makes it easier than ever to get started with Galtea and run your first evaluation quickly.
## Deeper Insights with Visible Conversation Turns
Understanding the full context of interactions is key. You can now view the complete [conversation turns](/sdk/tutorials/evaluating-conversations) associated with your [test cases](/concepts/product/test/case) directly within the dashboard. This offers richer context, aiding in more thorough analysis and debugging of your conversational AI products.
## Dashboard Usability Boost
We're continually refining the Galtea experience. This update brings several UI enhancements across the dashboard, designed to improve overall usability and make your workflow smoother and more intuitive.
## Tailor Your Test Generation: Selectable Test Case Counts
Gain more control over your testing process! When [generating tests](/sdk/api/test/create), you can now specify the exact number of [test cases](/concepts/product/test/case) you want Galtea to create. This allows you to fine-tune the scope and depth of your tests according to your needs.
## Track Your Team's Work: Creator Attribution Displayed
Clarity in collaboration is important. Now, the user who created a [Product](/concepts/product), [Test](/concepts/product/test), [Version](/concepts/product/version), or other key assets will be clearly displayed on their respective details pages. This helps in tracking ownership and contributions within your team.
## Enhanced Table Functionality for Easier Data Navigation
Working with data tables in the dashboard is now more efficient:
* **Clear Filter Indicators:** Easily see which filters are currently applied to any table.
* **Quick Filter Reset:** A new "Clear All Filters" button allows you to reset your view with a single click.
## New Conversation Evaluation Metrics
You can now [evaluate conversations](/sdk/tutorials/monitor-production-responses-to-user-queries#evaluating-conversations) using these new metrics:
* **Role Adherence** - Assess how well an AI stays within its defined role
* **Knowledge Retention** - Measure how effectively information is remembered throughout a conversation
* **Conversation Completeness** - Evaluate whether all user queries were fully addressed
* **Conversation Relevancy** - Determine if responses remain on-topic and purposeful
## Enhanced Security Framework
We've significantly improved user access management by implementing an **Attribute-Based Access Control (ABAC)** strategy, providing more granular control over who can access what within your organization.
## Extended Data Generation Capabilities
Our data generation tools have been expanded with:
* **Catalan Language Support** - Create synthetic data in Catalan to enhance your multilingual applications
* **Added support for text-based files** - Upload your knowledge base in virtually any text-based format including JSON, HTML, Markdown, and more
## Improved Test Creation Experience
We've enhanced the clarity of threat selection in the Test Creation form. The selection now displays both the threat and which security frameworks that threat covers, making it easier to align your testing with specific security standards.
## Analytics & Navigation Enhancements
* **Reduced Clutter in Analytics Filters** - Tests and Versions filtering now only display elements that have been used in an evaluation
* **Streamlined Navigation** - Clicking the "input" cell in the evaluations table now navigates directly to the associated Test Case
## Bug Fixes & Improvements
We've resolved several issues to ensure a smoother experience:
* Fixed a bug that could trigger an infinite loop in the Test Cases List of the dashboard
* Addressed multiple small UI glitches and errors throughout the platform
## Improvements in Red Teaming Tests
* **New "misuse" threat implemented**\
Now red teaming incorporates a new threat, misuse, which are queries that not necessaryly malicious however out-of-scope for you specific product. You can now test whether your product can successfully block these queries by marking "Mitre Atlas: Ambiguous prompts" in the threat list.
* **Better "data leakage" and "toxicity" tests**\
The red teaming tests incorporate better your product meta data, to generate the most adequate test cases for "data leakage" and "toxicity".
## Analytics Page Upgrades
We're continuing to expand the power of the Analytics page! This update introduces:
* **Radar View for Version Comparison**\
You can now visualize performance across multiple metrics for a single version using the brand-new radar view. It provides a quick way to understand strengths and weaknesses at a glance.
* **Smarter Metric Filters**\
Filters now only show metrics that have actually been used in evaluations—removing unnecessary clutter and making it easier to find relevant data.
* **Graph Tooltips**\
Hovering over truncated names now reveals full labels with tooltips, helping you understand graph contents more clearly.
## SDK Safeguards
We've added protections to ensure your SDK integration is as smooth and reliable as possible:
* **Version Compatibility Checks**\
If the SDK version you're using is not compatible with the current API, it will now throw a clear error to prevent unexpected behavior.
* **Update Notifications**\
When a new SDK version is available, you'll get a console message with update information—keeping you in the loop without being intrusive.
## Bug Fixes
* **Metric Range Calculation**\
Some default metrics were previously displaying inverted scoring scales (e.g., treating 0% as best and 100% as worst). This is now resolved for accurate interpretation.
* **Test Creation Not Possible Through `.txt` Knowledge Base Files**\
Due to a recent refactor, the creation of tests using knowledge base files with `.txt` extensions was not possible. This has been fixed and you can now create tests using `.txt` files as the knowledge base again.
## Monitoring Is Live!
Real-world user interactions with your products can now be fully monitored and analyzed. Using the Galtea SDK, you can trigger evaluations in a production environment and view how different versions perform with real users. [Read more here](/sdk/tutorials/monitor-production-responses-to-user-queries).
## Improved Galtea Red Teaming Tests
Our simulation-generated tests have been upgraded—delivering higher-quality outcomes. Red teaming tests can now be directed to validate even more specific aspects of various security standards, such as OWASP, MITRE ATLAS, and NIST. Specifically, we have improved **jailbreak** attacks, in addition to new **financial attacks** and **toxicity** prompts.
## New Analytics Page
A completely redesigned analytics page is now available! It features:
* **Enhanced Filtering Capabilities.**
* **Improved Data Clarity and Layout.**
The new design not only raises the clarity and density of data presentation but also improves your overall user experience.
> And with monitoring active, you can see production evaluation results in real time on this page!
## User Experience Enhancements
We're continuously refining the platform based on your feedback. This week's improvements include:
* **Customizable Evaluations List:**\
You can now select which metrics you are interested in, so the evaluations list only shows the ones you need.
* **Enhanced Evaluation List Filtering:**\
Easily filter evaluations by versions, evaluations, tests and test groups.
* **Enhanced Test List Filtering:**\
Easily filter tests by its group.
* **Smart Table Sorting:**\
When you apply a custom sort, the default (usually creation date) is automatically disabled.
