Changelog - Galtea Docs

2026-07-03

Test Case Revisions, Version Coverage in Analytics, and More

Track Changes to Your Test Cases

Test cases now keep a revision history, like metric revisions. When you edit a test case’s evaluation content, the platform creates a new revision instead of overwriting the old one. Annotation-only edits, such as reviewer or score, still edit in place. The list shows only the latest revision by default, with an opt-in filter for older ones, and each revision links back to the one it came from. Evaluations that ran against an old test case now show an Outdated Test Case badge, and the SDK warns you when this happens.

Compare Version Coverage in Analytics

The analytics page now shows shared test-case coverage so you can compare version scores fairly. Two versions can have the same average score even if one was evaluated against many more test cases, and shared coverage makes that gap visible. Each version shows how many test cases it was evaluated against, out of all test cases used across the shown versions, as a count and a percentage, with a Total / Used / Ignored summary. The selected-version area of each analytics tab is now two cards: radial gauges for the selected version, and a scores radar across all shown versions. Both stay visible and follow one grouping toggle (Specifications / Metrics). All numbers respect the active filters.

Platform Improvements

Iterate a version from an existing one: You can now create a version from an existing one to track how it evolved. Each version records its parent as an Evolved from link, available in the dashboard, API, and SDK.
Persona gender on test cases: Test cases can now store and show the persona’s gender.
Attach session metadata at creation: The SDK can now add metadata to a session when it is created.
Cancelled evaluation status in the SDK: The SDK EvaluationStatus enum now includes CANCELLED.
Development routes hidden in production: Dev and demo routes are no longer shown in the production dashboard.
Clearer report export errors: Exporting a report for a product with no data now returns a clear client error instead of a server error.
Faster permission checks: Permission checks now run in parallel, so permission-heavy requests are faster.
Sturdier SDK networking: The SDK now reuses one HTTP session with retries and a timeout, and only prints colored output to a terminal.
Dashboard polish: Reduced oversized text, fixed clickable empty space in truncated table cells, restored non-admin assistant routes to the catalog, and refreshed the sidebar create-product button.

2026-06-26

Run Voice Simulations, Update Specifications from a Document, and More

Run Voice Simulations on Your Phone Agents

You can now run voice simulations against AI agents that answer the phone. A new phone connection tells Galtea how to reach your agent: you set the phone number once, and Galtea dials your agent and runs the conversation as a real voice call. This is Direct Inference over the phone.A few things that make it powerful:

The caller sounds human, in your agent’s language. Set a test case’s language and the synthetic caller speaks it with a natively-accented voice (a Spanish test case is answered in Spanish), so one phone connection covers many languages with no extra setup.
Voice agents are scored like text agents. Galtea transcribes each agent reply with speech-to-text automatically, so every metric you already use works over the phone, including security tests that speak adversarial prompts out loud.
You can replay the call however you like. The entire conversation is captured as a continuous call recording that you can play back in the dashboard, either as a single clip or turn by turn.

Update Specifications from a Document

As your product evolves, its documentation (PRDs, design docs, spec sheets) often changes before its specifications do. A new Update from document action on the product detail page bridges that gap.Upload a document and the platform compares its content against your product’s existing specifications. Each suggestion is classified as one of:

New — a behavioral expectation found in the document that has no matching specification yet.
Updated — an existing specification that the document refines or extends.
Contradictory — an existing specification that conflicts with something in the document.

You review all suggestions and choose which to apply. Accepted new suggestions create a new specification. Accepted updated and contradictory suggestions replace the referenced one. Anything you leave unselected stays untouched. Nothing changes without your explicit confirmation.

Product Cards Show Version Scores

Product cards now tell you more at a glance. Each card shows two version rows:

Best — the version with the highest average evaluation score, shown with a percentage (e.g., v2 (55%)).
Newest — the most recently created version.

A version with no evaluations shows (-) instead of a percentage. The Evaluations counter and Total Cost row have been removed from cards to keep the view focused.

Cross-Product Ranking in Val

Val (the Galtea AI assistant) can now compare your products against each other on a shared metric. Ask Val something like “how does the whole product family perform on Faithfulness?” and it returns a worst-first ranking with the evaluation count, mean, and min/max score for each product, plus an overall evaluation-weighted score across all of them.If you do not specify a metric, Val first lists the metrics shared across your selected products and waits for you to pick one before returning numbers.

Platform Improvements

Try a demo product during onboarding: New users can now clone a ready-made Galtea demo during the onboarding flow.
Faster audio loading in inference results: Audio files in inference result sessions now load lazily and in a single batched request as turns scroll into view, rather than fetching all URLs and preloading all audio on render. The audio player loads only the duration header for visible turns, so the seek bar shows the correct length without downloading the full file. Play latency is unchanged.
Val assistant fixes: Several usability issues in the Val widget are resolved. The ⌘K / Ctrl+K shortcut no longer fires while you are typing in another field. Pressing Enter mid-stream inserts a newline as expected. Citation chips no longer appear for links inside code blocks. The clipboard copy button falls back gracefully on plain HTTP. The route catalog no longer suggests admin-only pages that regular members cannot access.
Tooltips close when the pointer moves onto their content: Tooltips now close as soon as the pointer leaves the trigger, like any standard hover element. Previously, moving the pointer from the trigger onto the tooltip kept it open and could block clicks on elements beneath it.
Long entity names stay inside their column: Long names and identifiers in link cells (Test, Version, and similar columns in data tables) now wrap and clamp to two lines inside their column instead of overflowing into the next one.

2026-06-19

Usability improvements and bug fixes ahead of bigger features

This release focuses on usability, bug fixes, and small improvements while we work on bigger features coming in the next weeks.

Platform Improvements

Create specifications during onboarding: You can now create your custom specifications during the onboarding flow to cover gaps you would like to evaluate that our AI system did not detect or did not expect to be a primary focus.
Filter sessions by status: The session list now has a status filter — select PENDING, COMPLETED, or FAILED (or any combination) to focus on the runs that matter. Available in the dashboard, the API (statuses query parameter on GET /sessions), and the SDK via sessions.list(status=…).
See direct inference progress right away: When you start a direct inference run, every session it will produce now appears in the list as PENDING immediately — instead of trickling in as each one is picked up — so you can track progress from the moment the run starts.
Security metrics now evaluate correctly: Data Leakage, Jailbreak Resilience, and Misuse Resilience evaluations were being silently marked SKIPPED because they depended on a field the product form no longer surfaces. These metrics now draw context from product_capabilities and product_inabilities (populated from your specifications) and run as expected.
“New Metric from template” fixed: Clicking From template in the New Metric screen was throwing a routing error and closing the dialog immediately. The dialog now opens and works correctly.

Documentation and Examples

SDK evaluation examples now show the specification-first approach — passing specification_ids= instead of metric names — side by side with the metric-name approach, in tabbed code blocks, on every relevant tutorial page.

2026-06-15

Smoother Onboarding, Endpoint Pre-flight Checks, and Faster Evaluation Runs

Smoother Onboarding

Getting your first product evaluated now takes less waiting and less typing.Generation runs in the background. When the onboarding tutorial creates the tests and metrics for your selected specifications, it no longer holds you on the review step until everything finishes. Generation continues while you keep moving through setup — only the final Launch Evaluation button waits for it to be ready, and if anything fails to generate you get a clear retry instead of a stuck screen.Your first product is named for you. The onboarding form no longer asks you to name your product. The platform derives a concise name from the description you provide as part of the same step, so there’s one less field between you and your first evaluation.

Platform Improvements

Endpoint connections are checked before any work is queued: When you start an evaluation, the platform now runs a quick pre-flight probe against the version’s endpoint connection first. If the endpoint is unreachable or returns an error, the request is rejected immediately with a clear message — instead of queuing inferences that were always going to fail.
Faster large evaluation runs: Evaluation runs now process their sessions in parallel rather than one after another, so large runs finish sooner and a single big run no longer blocks others behind it. Each run also respects a per-endpoint rate limit, so your endpoint is never flooded with requests.

Documentation and Examples

A new Improve Your Product with Evaluations and an AI Agent playbook walks through the full evaluate → diagnose → improve → compare loop: hand your failing evaluations to an AI coding agent using the Galtea Agent Skill and CLI, let it propose a fix, then re-run the evaluations to prove it helped — illustrated end to end on a RAG support chatbot.
The Direct Inferences & Evaluations from Platform tutorial now documents the full round-trip for stateful multi-turn conversations: how to extract a value from one response into session metadata, and then send it back on the next turn.

2026-06-10

Analytics Insights, AI-Assisted Metric Creation, Specification-led Evaluation Runs, and More

Analytics Insights

The Analytics Overview tab now includes an Insights card that automatically scans your evaluation data and surfaces what needs your attention. The card applies deterministic rules across three categories:

Score health — flags metrics that fall below the warning (85) or critical (60) thresholds, and links each flag directly to the relevant evaluations with the judge’s reasoning so you can dig in immediately.
Specification coverage — identifies specifications that have no linked tests yet, so coverage gaps don’t stay invisible.
Version regressions — detects metric scores that dropped 10 or more points from one version to another, distinguishing warnings from critical degradations.

Flags are sorted by severity. Each carries a named action — clicking it navigates you to the right place in the platform rather than leaving you to find it yourself. When nothing is flagged, the card shows an explicit all-good state so you know the check ran. A plain-language narrative at the top summarizes the findings.Val (the platform assistant) now answers evaluation-quality questions using the same backend rules as this card, so the assistant and the dashboard always agree on what counts as a red flag.

Smarter Run Evaluation Flow

Running an evaluation from a version or product now opens a Specifications tab by default. It pre-selects the specifications that have runnable tests and submits via the existing batch endpoint — the platform resolves linked tests and metrics server-side, so you don’t have to select them manually. If a product has no specifications, the dialog falls back to the familiar manual test and metric selector.In the manual selector, the metric picker now ranks and pre-selects metrics based on relevance:

Linked — metrics directly linked to the specification of your selected test are pre-selected and appear first.
Recommended — metrics linked to any specification sharing the same test type appear next with a badge.

Incompatible metrics are still listed at the bottom so you can override, but the common case no longer requires manual searching.

Complete Metric with AI

Two places in the dashboard now offer a Complete With AI button on metrics: the manual create-metric form and the candidate edit dialog inside the AI metric generation review step.Fill in the metric name, description, and evaluation type — and let the AI generate the judge prompt and evaluation parameters for you. Your name, description, and tags are never overwritten. If you already have a judge prompt, the platform prompts for confirmation before replacing it. AI-generated fields display a badge so you always see what was filled by the model.This works for AI Evaluation and Human Evaluation metrics. Self-Hosted metrics are excluded (they have no judge prompt).

AI-Assisted Specification Creation

Two improvements to the specification creation flow:Fill with AI (the dialog on the create and update pages) now rewrites your rough note into a properly written specification description and classifies the type in one step, returning description, type, testType, and testVariant together. Previously it only filled the category selectors and never touched the description.Complete with AI (the inline button) sends the description you already wrote, applies only type, testType, and testVariant, and never touches your text. The button was previously labeled “Categorize with AI” — the new name reflects that it completes the fields you left empty without altering the ones you filled.In Detailed generation mode, the AI now also considers a built-in catalog of safety and quality specifications that apply to most AI products: jailbreak resistance, illegal-activity refusal, system-prompt non-disclosure, cross-user data leakage, AI-identity disclosure, and others. Specifications that don’t fit (for example, safety specs for backend-only products with no human users) are excluded automatically. You review and edit everything before any of it is saved.

Import Endpoint Connection from cURL

Creating an endpoint connection now offers an Import from cURL option. Paste a cURL command — from your API docs, a Postman collection, or a terminal history — and the platform extracts the URL, headers, and request body, then maps the body to a Jinja2 input template with {{ input.user_message }} pre-wired. For conversation-type connections, a lightweight AI pass enriches the template further. The New Endpoint Connection button is now a From cURL / Blank dropdown.In the onboarding flow, the cURL paste step is the default view; the traditional form is available via Skip — fill in manually. Fields that couldn’t be recovered from the cURL command are flagged inline, and a mandatory probe test confirms the connection works before saving.

Platform Improvements

Row click navigates to entity details: Every table row in the dashboard now navigates to the entity’s detail page on click. The primary cell renders a real link, so Ctrl/Cmd/middle-click opens it in a new tab. Clicking on a text cell opens a full-text dialog with a copy button instead of navigating. Drag-to-select on table text still works.
Assistant renamed to Val: The Galtea platform assistant is now named Val across the dashboard panel and the public docs site. The panel header, launcher pill (“Ask Val”), and the agent’s own persona are all updated.
Breadcrumb navigation restores your position: Clicking a breadcrumb link now returns you to the exact tab and pagination page you were on when you drilled into the current entity. For a fresh deep link, it falls back to the tab that lists the entity you were viewing.
Analytics score charts use a fixed 0–100 axis: Score charts in the Analytics section are anchored to a 0–100 scale so visual proportions are consistent across products and time windows. Previously the axis auto-scaled to the data range, making small absolute differences appear large.
List endpoints default to your organization: Every GET list endpoint now scopes to the caller’s organization automatically, so you no longer need to pass an organization filter to list your own data. Admins can opt into a cross-organization view with the allOrganizations flag.

2026-06-08

Galtea Assistant Redesign, Onboarding Experience, Job Cancellation, and More

Galtea Assistant Redesign

The Galtea AI assistant is now a non-modal docked side panel that sits alongside the dashboard instead of covering it. When open, the page content reflows into a true split view — the panel holds a fixed width on the right while the rest of the dashboard fills the remaining space. You can resize the split by dragging the divider. Open and close the panel from anywhere in the dashboard with ⌘K (Mac) or Ctrl+K (Windows/Linux).The assistant now shows its work while it processes your message. A live step trace displays each phase of the agent’s reasoning — searching the docs, calling API tools, writing the answer — with the active step pulsing in real time. When the response arrives, the trace collapses to a compact summary (for example, “Worked for 2.3s · 4 steps”) that you can expand at any time to review the full trajectory.For write operations, the assistant proposes an action and you approve it before anything changes. Each proposal shows a preview of what will happen, and destructive actions display how many items would be affected before you confirm. When you approve, the dashboard navigates you directly to the affected product, version, or evaluation. The assistant can now reach any allow-listed resource in the platform — reads are driven by the live API catalog, and writes cover 28 curated operations — so it adapts as new endpoints are added without requiring new tool code.

Redesigned Onboarding Experience

New users now flow through a focused, full-page onboarding shell that adapts to how you want to run evaluation — import traces, connect your endpoint, or use the SDK — and pre-selects the specifications matching the goal you gave at signup, so first evaluation results arrive faster.

Word Document Upload for Product Setup

When setting up your product with AI assistance, you can now upload Word documents (.docx) alongside PDFs and other formats. The platform extracts the content and uses it to generate specifications, so teams that maintain documentation in Word can feed it directly into Galtea without converting files first.

Job Cancellation

You can now cancel in-progress async jobs from both the API and the SDK. When you start an inference batch — for example, via evaluations.run() in endpoint-connection mode — and need to stop it early, call galtea.jobs.cancel(job_id). Queued jobs are removed from the queue; jobs already running receive a cooperative cancellation signal and stop at their next checkpoint. Work already dispatched when the signal is observed is not rolled back. Cancellation is idempotent — calling cancel() on an already-cancelled job returns state='cancelled' rather than raising an error. Use galtea.jobs.get_status(job_id) to check the current state before deciding whether to cancel. See the Jobs Service reference for the full API.

Platform Improvements

PASS/FAIL verdict shown in the annotation dialog: When annotating evaluations, the score pill now displays the binary PASS/FAIL outcome alongside the numeric score so you can see the verdict at a glance.
“See details” button on every table row: All entity tables — sessions, evaluations, tests, and more — now show a direct-navigation icon button to the right of the row actions menu. Click to open the detail page; hold Cmd/Ctrl to open it in a new tab.
Unified test creation form: Creating a test from a single specification now uses the same creation form as batch creation. The previous dedicated single-spec flow has been retired.
Accuracy-requirement specifications correctly classified: Specifications of the accuracy-requirement type are now saved as POLICY + QUALITY. Previously, creating one via the API produced an incorrect type.
Coverage radar chart renders correctly in edge cases: The coverage radar chart now renders when two specifications share the same name or when a version has no scores for one of the axes. Previously, either condition caused the chart to be skipped.
Clearer SDK error messages for agent integrations: When an agent skill invocation fails due to an argument shape mismatch, the SDK now surfaces a clear description of the issue instead of a cryptic internal error.
LLM configuration errors surface immediately: Misconfigured API keys, unsupported model names, and other non-retryable LLM errors now cause evaluations to fail on the first attempt rather than consuming the full retry budget before reporting the problem.
Red-teaming generation handles content filters: When a content filter blocks a generation step, the red-teaming generator now retries. If the seed prompt itself keeps triggering the filter after retries, that seed is skipped and generation continues with the remaining prompts.
Specification linking documented in CLI and SDK: The galtea spec link-tests and galtea spec unlink-tests commands and their SDK equivalents — galtea.specifications.link_tests() and galtea.specifications.unlink_tests() — are now documented in the CLI --help output and SDK docstrings, making them easier to discover.

2026-06-01

Metric Optimization Lifecycle and Platform Improvements

Metric Optimization Lifecycle

When you optimize an AI Evaluation metric, the result now behaves like any other revision of that metric. The optimized version joins the same revision lineage as a manual “Create new revision” — so you can trace the full optimization history in the metric’s revision panel and see how the judge prompt has evolved across generations.The Optimize button now shows the correct annotation threshold. It counts only the human annotations where your score and the AI’s score fall on opposite sides of the PASS/FAIL boundary — evaluations where your annotation genuinely overrides the AI’s verdict. Previously, any numeric difference between scores counted, even when both scores agreed on the outcome.

Platform Improvements

Incompatible evaluations now show SKIPPED instead of FAILED: Evaluations that cannot run — such as sessions where the agent produced no output, or deterministic metrics applied to multi-turn sessions — are now caught at pre-flight and marked SKIPPED immediately, before being dispatched to the evaluator. Previously these were dispatched and returned FAILED with no clear explanation. The evaluation result now includes the specific reason the evaluation was skipped.

Documentation and Examples

Direct Inference OTel collector: authentication guide updated: The Direct Inference OTel exporter tutorial now covers the required Authorization: Bearer <Galtea-API-key> header with four supply patterns — SDK environment variables over HTTP, SDK environment variables over gRPC, programmatic Python, and OTel Collector YAML relay. The section also documents shell-expansion pitfalls with OTEL_EXPORTER_OTLP_TRACES_HEADERS, a curl smoke-test command for self-validation, and a note on the current gRPC TLS handshake issue with HTTP/4318 as the validated path.

2026-05-26

Langfuse Trace Import, SDK Models, and More

Import Langfuse Traces

If you already export traces from Langfuse, you can now bring them into Galtea without any code changes. Open New Session → Import traces (Beta), upload your Langfuse export file — JSON, JSONL, or CSV are all accepted, and both the Traces and Observations export tabs from Langfuse work — select the version you want to link them to, and click Import. Each trace becomes a session with its observations mapped to inference results, ready to evaluate immediately.The import runs as a background job, so you can navigate away while large files are processed. The dialog stays open and shows progress; once complete, the sessions table refreshes automatically.This is the recommended path for teams already using Langfuse’s LangChain or LangGraph integration and wanting to run Galtea evaluations on their existing trace history without modifying their agent code.

SDK: Model Management

The SDK now includes a ModelService, accessible as galtea.models. It supports the full lifecycle — list, get, get_by_name, create, update, and delete — bringing programmatic parity with the dashboard’s model management for cost tracking and estimation. list() accepts both a partial-match name filter and an exact-match names list, alongside the standard ids, pagination, and date-range parameters.See the Model Service reference for the full API.

Platform Improvements

FULL_PROMPT metric creation is now blocked: POST /metrics and galtea.metrics.create(...) reject requests that set source=FULL_PROMPT or that supply a judgePrompt without evaluationParams. Use PARTIAL_PROMPT for new AI Evaluation metrics — it covers the same use cases with explicit, user-configurable evaluation parameters. Existing FULL_PROMPT metrics are unaffected and continue to be evaluated end to end.
SDK generate() reflects live agent status: inference_results.generate() now creates the record as PENDING while your agent runs and only transitions to GENERATED on a successful return. Previously the record was immediately marked GENERATED with a null output, which broke the contract that GENERATED means the agent completed successfully.
Onboarding video on the welcome screen and Quickstart: The product welcome screen and the Quickstart page now include an embedded walkthrough video so new users can see the end-to-end setup flow before writing any code.
More accurate AI evaluation scores: Judge prompts now produce reason before score. This chain-of-thought ordering consistently improves score quality by forcing the model to articulate its reasoning before committing to a number.
Table header overflow fixed: Long column headers — such as version hashes in the analytics comparison table — are now clipped at the cell boundary instead of visually bleeding into adjacent columns.

2026-05-18

Metric Optimization, Galtea Assistant, and More

Metric Optimization

When you create a custom AI Evaluation metric, the judge prompt rarely scores responses exactly the way your team would on day one. The new Optimize action closes that gap: it takes a metric’s current judge prompt, learns from the human annotations your team has already submitted, and rewrites the prompt so the AI score lines up more closely with the human score. The result is saved as a sibling metric named <original name> (Optimized), so you can compare both versions side by side before adopting the new one.The Optimize button appears on the details page of any AI Evaluation metric created by your organization. It becomes available once enough annotations have been collected, with a particular focus on cases where the annotator disagreed with the AI score: those disagreements are the signal the optimizer learns from.

Galtea Assistant (Beta)

A new in-product AI assistant is now available. The floating launcher in the bottom-right corner of the dashboard opens a chat panel that can answer questions about the platform, point you to the right page, and walk you through common workflows. The same assistant is also embedded in the docs site.

Form Improvements

Submitting a new product now uses AI to validate that the description matches what Galtea can test. Submissions that look voice, image, or multimodal, or that read more like a system prompt or build instructions than a product description, open a confirmation dialog before saving. The check is advisory and never blocks creation, so you can review the warning and proceed or go back and edit.Navigating away from any creation or edit form with unsaved changes now opens a confirmation dialog with “Discard” and “Keep editing” choices, so accidental clicks no longer lose your work. Forms protected include metric and product creation and edit, test edit, endpoint connection edit, and dialog-level flows such as Add Specification.

Platform Improvements

Full customization when creating tests from Behavior specs: Creating a test from a Behavior specification now gives you the complete test creation form, with every customization option available, including data-catalog file uploads.
Faster AI-assisted product setup: Generating a product description and specifications during product creation is now noticeably faster. Specification file uploads are also accepted in both quick and detailed generation modes.
Clearer status for empty-parameter evaluations: Evaluations whose required fields (such as retrieval_context) are empty or whitespace-only strings now resolve as SKIPPED upfront with a clear “missing required parameters” reason, instead of dispatching and landing as FAILED.
CLI: background spec refresh and safer plaintext login: The Galtea CLI now refreshes its OpenAPI spec cache in a detached background process, so commands no longer pay the refresh latency. Logging in to a non-localhost HTTP host now requires --insecure to prevent accidental credential exposure over plaintext; the consent is sticky on that host so re-logins do not re-prompt.

2026-05-08

Specifications View Redesign, Two-Factor Authentication, and More

Specifications View Redesign

The specifications list is now split into two tables: Policies and Capabilities & Inabilities. Each row shows separate Metrics and Tests columns — a green check with a count when the resource exists, or a “Missing” pill that routes you to the test creation form when it does not.

Two-Factor Authentication

Email two-factor authentication is now available under Settings → Security, and organization admins can enforce MFA for all members from the organization settings page. See Registration and Sign-In for details.

Documentation and Examples

Galtea Agent Skill: The Galtea Agent Skill page has been refreshed with updated install instructions and the Claude Code plugin install path.

Platform Improvements

Fraction input for annotation scores: The human evaluation score field now accepts fractions (e.g., 7 out of 10) alongside percentages.
Max iterations for behavior test generation: Behavior test AI generation now exposes a max_iterations setting, with the default lowered from 20 to 10.
Language field promoted in test creation: The Language dropdown is now visible directly in the test creation form, not only under Advanced Options.
CLI via Homebrew: The Galtea CLI is now installable via Homebrew. See the CLI installation page for all options.
Same-name metric revisions: Creating a metric revision that reuses the parent’s name no longer fails.

2026-05-04

Galtea CLI Launch, Product Tags, and Smarter Pre-Evaluation Guidance

Galtea CLI

The Galtea CLI is now generally available. Install it via apt, dnf, or pip, then authenticate with galtea login or a GALTEA_API_KEY environment variable. The CLI auto-generates a galtea <resource> <verb> command for every public API endpoint, so new endpoints appear as new commands without a CLI release. See the new Installation and Usage pages to get started.The Galtea Agent Skill now drives Galtea through the CLI, which improves how reliably coding assistants like Claude Code and Cursor can manage Galtea on your behalf.

Product Tags

Products now support tags. You can add and edit tags from the create and update forms, filter the product list by one or more tags, and see tags as chips on product cards and the details page.

Smarter Pre-Evaluation Guidance

Before launching an evaluation, the run dialog now checks whether each selected metric can be satisfied by the current product and endpoint connection configuration, and flags any that cannot with a tooltip describing what is missing.When an evaluation is still SKIPPED at runtime, the error message now explains what is missing and where to fix it, grouped by source, with actionable guidance included.

Structured Inputs for AI-Generated Tests

Structured JSON inputs were previously limited to uploaded or manually created test cases. AI-generated test cases — scenarios, red-teaming, and gold-standard — now also receive a structured input populated from the endpoint connection’s JSON Schema.AI-assisted product and specification generation also now respects the language of the source material.

Documentation and Examples

New CLI Installation and CLI Usage pages.
New AgentInput reference page with all fields, the ConversationMessage structure, and updated SDK code examples.

Platform Improvements

Endpoint connection picker in test creation: Test creation and the AI candidate review screen now offer an explicit endpoint connection picker under Advanced Options.
Filter test cases by input substring: The Test Cases list has a new debounced text filter, including matches inside structured JSON inputs.
Metric lineage on the details page: Revised metrics display an “Evolved from” link to the parent metric, and the revision flow now shows a warning before marking the source metric as legacy.
Inference Results in the session dropdown: The session row dropdown now includes an “Inference Results” option alongside “Evaluations” and “Traces”.
Sidebar cleanup: The Models entry has been removed from the sidebar. Models remain reachable from the organization dropdown and the version forms.
Optional secret fields on endpoint connection updates: Credential fields are now optional when editing an existing endpoint connection — leave them empty to keep the stored value.
SDK @observe fix for async agents: The observe decorator now correctly preserves Galtea context across async coroutines.
Clearer max-iterations stopping reason: Conversations that hit the iteration cap now record a “Max iterations reached” stopping reason instead of a generic error.
Updated wording: Product creation now reads “Register product” throughout the platform.

2026-04-27

Metric Versioning with Evaluation Replay, Galtea Agent Skill, and More

Metric Versioning with Evaluation Replay

Metrics now support versioning. The new “New Revision” action on any metric opens the creation form pre-filled with the source metric’s name, description, tags, evaluation type, judge prompt, evaluation parameters, evaluator model, and user groups. Revised metrics stay linked to their predecessor through a parent-child lineage, so you can trace the evolution of a metric over time. When a revision is created, only the direct parent is marked as legacy, allowing parallel branches from the same source.After saving a revision, you are prompted to replay historical evaluations from the metric family against the new version. The replay dialog lists your products, shows how many previous evaluations are eligible per product, and lets you select which ones to re-run. Evaluations that have already been replayed are automatically deduplicated.The metric creation flow now surfaces all three modes upfront: Blank, From Template, and Metric Revision, each with a description of what it does. “From Template” (previously called “Duplicate”) creates an independent copy with no lineage link, while “Metric Revision” creates a linked successor.

Galtea Agent Skill

The Galtea Agent Skill is now available as an open-source Agent Skill for AI coding assistants. Once installed, tools like Claude Code, Cursor, and Windsurf can authenticate against Galtea, manage products, run evaluations, inspect sessions and traces, and follow validated workflows without you having to explain Galtea first.Install it by pointing your coding agent to the Galtea-AI/skills repository, or use the skills CLI: npx skills add Galtea-AI/skills --skill "galtea".

Run Evaluations from Specifications

You can now run evaluations directly from a specification page. The action is available both on the specification detail page header and as a row action on the specification list. The dialog shows a version dropdown (filtered to versions with an endpoint connection, latest pre-selected) and displays the specification’s linked tests and metrics as read-only badges, so you can verify the evaluation scope at a glance before launching.The action is gated on the specification having at least one linked metric and one linked test. When either is missing, the button is disabled with a tooltip explaining the requirement.

Language Variables in Input Templates

Two new built-in template variables are available in endpoint connection input templates: {{ language_code }} resolves to the test case’s ISO 639-1 code (e.g. es), and {{ language_name }} resolves to the full language name (e.g. Spanish). Both resolve to an empty string when the test case has no language set, keeping templates compatible with language-agnostic tests. The placeholder toolbar in the dashboard includes both variables for easy insertion. For the full template syntax reference, see Structured Input Template Syntax.

Reliable Trace-Dependent Evaluations

Evaluations that depend on trace data (TRACES or TOOLS_USED parameters) now wait for traces to be ingested before running. Previously, these evaluations could execute against empty trace data when traces had not yet arrived. The platform now defers them with automatic retries, and if traces do not arrive within the retry window, the evaluations land as SKIPPED with an actionable reason explaining what was missing. Sessions already marked as completed are detected immediately, avoiding unnecessary waiting. No credits are consumed for skipped evaluations.

2026-04-20

CSV Export for Evaluations, Inference Result Error Tracking, and More

Export Evaluation Results as CSV

You can now export evaluation results directly from the dashboard as a CSV file. The export action appears in the row selection toolbar on the evaluation list page. With no rows selected, all evaluations matching the current filters are exported. When specific rows are selected, only those are included.The CSV includes columns for Score, Metric, Input, Output, Expected Output, Retrieval Context, Version, and Test, along with export-only identifier columns (Evaluation ID, Session ID, Test Case ID) that do not appear in the table UI but provide full traceability for downstream analysis.

Inference Result Error Tracking

When an agent crashes or returns an empty response during simulation, the corresponding inference result is now properly marked as FAILED with the error details attached, instead of remaining stuck in a PENDING state. Each inference result carries its own error information, so you can see exactly what went wrong at the individual turn level.The evaluation pipeline also respects these statuses: evaluations are automatically skipped for non-ready inference results, with a clear reason recorded explaining which turns were not ready and why. In the dashboard, failed turns display a red error badge with the error message visible inline.

One-Click Metric Duplication

You can now duplicate an existing metric as a starting point for a new one. The “Use as Template” action in the metric dropdown navigates to the creation form pre-filled with the source metric’s name, description, tags, evaluation type, judge prompt, evaluation parameters, evaluator model, and user groups. The duplicated metric is independent from the original, so you can adjust any field before saving.

Platform Improvements

Test case source file tracking: Test cases generated from uploaded documents now display the name of the original source file, improving traceability for file-based test generation.
Annotate action on evaluation details: The Annotate (or Evaluate, for pending human review) action is now accessible directly from the evaluation detail page header dropdown, in addition to the list view context menu.
Clearer error messages for test generation: When QUALITY test generation fails, the error message now includes the specific reason returned by the generator, replacing generic HTTP error responses with actionable guidance.
Analytics radar chart fix: The radar chart in the Analytics tab now works correctly when a specification section has fewer than 3 scored items. The view automatically switches to radial value cards with a tooltip explaining the minimum requirement.
Smoother table text filters: Text filter inputs across all dashboard tables no longer flash or revert while typing. Filter changes are debounced before syncing to the URL, reducing unnecessary API calls and eliminating the visible blink caused by intermediate re-renders.

2026-04-13

Human Annotation for All Evaluations, Structured JSON Inputs, SDK Endpoint Connections, and More

Human Annotation for All Evaluations

Human annotation is no longer limited to human evaluation metrics. You can now annotate any completed evaluation with a human score and reasoning, regardless of its metric source. This means AI-evaluated results from full-prompt, partial-prompt, deterministic, or self-hosted metrics can all receive a parallel human assessment. Unlike human evaluation metrics (which contribute to the evaluation score and analytics), human annotations are stored as supplementary feedback alongside the original result.

SDK: Endpoint Connection Management

The SDK now provides a full CRUD service for endpoint connections, so you can manage your agent connections entirely from code. Available operations include create, get, get_by_name, list, update, delete, and test.The service supports all authentication types (Bearer, API Key, Basic), retry configuration with customizable backoff strategies, and partial updates where you can modify individual fields without resending the entire configuration.

SDK: Job-Based `wait_for`

The evaluations.wait_for() method now accepts a job_id parameter in addition to evaluation_ids. When you run an evaluation against an endpoint connection without a local agent, run() returns a jobId. You can now pass that ID directly to wait_for(job_id=...) and let the SDK handle the entire lifecycle: polling the job status, discovering the evaluations it produced, and waiting for them to complete.

SDK: Langfuse `CallbackHandler`

Building on the Langfuse integration from the previous release, the SDK now includes a CallbackHandler for LangChain users. Instead of decorating individual functions with @observe, you can pass the handler directly to chain.invoke(config={"callbacks": [handler]}) and every LLM call, tool invocation, and retriever query in the chain will be traced automatically.The handler manages Galtea context through depth tracking, so nested callbacks are handled correctly without manual cleanup. It works as a singleton (set the inference result ID per request) or as a per-request instance. See the CallbackHandler integration guide for setup instructions.

Structured JSON Inputs

This release extends JSON-structured inputs across the platform.SDK: input_data and context_data on AgentInput. When using the structured agent signature, your agent function now receives context_data alongside the conversation messages. If your test cases include structured context fields (e.g., customer_tier, language, or any custom key-value pairs), they flow automatically into AgentInput.context_data during evaluation and simulation. The generate() method also accepts structured input as a dictionary with multiple fields.CSV import with JSON values. When importing test cases from CSV, the platform now detects and preserves JSON objects in the input and context columns. You no longer need to flatten structured data before uploading. For the time being, structured JSON inputs are only available for uploaded or manually created tests.

2026-04-07

Langfuse Integration, Public REST API Reference, and Platform Improvements

Langfuse Integration

If you already use Langfuse to observe your LLM application, you can now send the same traces to Galtea for evaluation by swapping a single import. Replace from langfuse import observe with from galtea.integrations.langfuse import observe, and every span will flow to both Langfuse and Galtea simultaneously.The integration is fully transparent to Langfuse: your Langfuse dashboard, trace IDs, and alerts remain completely unaffected. Galtea only exports traces when you explicitly link an inference_result_id, so there is no overhead for unrelated function calls.Two wrappers are available:

observe decorator for the standard @observe pattern
start_as_current_observation context manager for manual span control

Get started with the Langfuse Integration overview, or check the Integration Guide for a detailed walkthrough of the code changes needed to add Galtea evaluation alongside Langfuse.

Public REST API Reference

The platform now exposes a public OpenAPI specification covering approximately 81 endpoints across products, versions, specifications, tests, sessions, evaluations, metrics, traces, and more. Internal-only endpoints (authentication flows, billing, user administration) are excluded automatically.You can import the spec into Postman, Cursor, Claude, or any OpenAPI-compatible tool to explore and call the API directly. Every endpoint description now includes a link back to the relevant documentation page, making it easier to navigate between the spec and the guides.

Platform Improvements

Improved code and JSON display: Code and JSON data across evaluation details, trace details, session details, and endpoint connection test results now render with proper formatting and syntax highlighting instead of plain preformatted text.
Flexible behavior test case editing: The input field is no longer required when editing behavior test cases, since behavior-type tests do not always need explicit input values.
Specification type tooltips: Hovering over the specification type pill (Capability, Inability, Policy) in the dashboard now shows a short description of what each type means, linking to the Specifications documentation for more detail.

2026-03-30

Zero-Code Trace Ingestion, Structured JSON Inputs, and SDK Improvements

Zero-Code Trace Ingestion via OpenTelemetry

If your service is already instrumented with OpenTelemetry, you can now send traces to Galtea without writing any Galtea-specific code. The platform exposes an OTLP-compatible trace ingestion endpoint that accepts standard OTLP/JSON payloads from your existing OTel Collector.When you run a direct inference through Galtea, the platform automatically injects a W3C traceparent header into the request to your endpoint. If your service propagates that header internally (as most OTel-instrumented services do), all resulting spans are automatically correlated with the corresponding Galtea inference result. No manual ID passing, no SDK calls for trace creation, just your existing instrumentation working end to end.For services that are not OTel-instrumented, the Galtea SDK’s @trace decorator and start_trace context manager continue to work as before. A step-by-step guide covering all tracing methods, including W3C trace context propagation, is available in the documentation.

Structured JSON Inputs

Test case inputs and session contexts now support structured JSON objects in addition to plain text strings. This means you can store complex, multi-field payloads directly in the platform database. Existing string-based inputs continue to work unchanged.In upcoming releases, structured JSON inputs will be more broadly supported across the platform, including endpoint connections and SDK methods.

SDK: Synchronous Evaluation Polling

The Python SDK now includes evaluations.wait_for(), which polls one or more evaluations until they leave PENDING status and raises a TimeoutError if the deadline is exceeded. This simplifies scripts and CI pipelines that need to block until results are ready.

Normalized JSON Field Match Metric

The JSON Field Match metric now supports a normalized comparison mode that performs case-insensitive and accent-insensitive matching within field values. For example, "Sí" and "SI" are treated as equal. This is useful when evaluating outputs in multilingual contexts or when casing differences are not meaningful.

Editable Test Metadata

Tests now include an editable metadata field where you can store free-form notes or custom tracking information. The field is available in the dashboard, the API, and the SDK, giving you a lightweight way to annotate tests without modifying their structure.

Platform Improvements

Time-of-day filters: Date filters throughout the dashboard now support hour, minute, and second precision, so you can narrow results to a specific time window rather than a full day.
Augmented column: The test cases table now includes an Augmented column indicating which test cases were generated through AI augmentation.
Double-click prevention: The Save Accepted button during metric generation no longer triggers duplicate submissions if clicked rapidly.
Pagination stability: Table pagination no longer resets unexpectedly when navigating between pages via URL.
Show Details navigation: The Show Details action in entity tables now opens directly to the details tab.

2026-03-23

Specification Analytics, CSV Export, and Expanded Documentation

Specifications in Analytics

The analytics dashboard now includes a dedicated Specifications section that aggregates evaluation scores per specification, giving you a clear picture of how each policy area is performing across versions. Specification-level radar charts let you compare coverage at a glance, and clicking any specification label filters the entire dashboard to show only metrics linked to that specification.Version comparison charts have also been improved with click-to-highlight and double-click-to-navigate directly to the evaluations list.

Test Case CSV Export

You can now export your test cases as CSV files directly from the dashboard. Select specific test cases or export all visible ones — the platform generates a CSV with 22 columns covering input, expected output, strategy, language, scores, and more, then provides a direct download link. Use exported data for offline review, spreadsheet analysis, or integration with external workflows.

Documentation and Video Guides

The documentation has been significantly expanded with new video walkthroughs and reference pages:

Video guides now cover specifications, versions, traces, human evaluation, endpoint connection setup, and AI-powered specification generation — each embedded directly in the relevant docs page
A new Writing Specifications guide covers how to define specifications for your product — behavioral expectations, policies, and rules that drive metric generation and test creation
New reference pages for evaluation types, evaluation parameters, and endpoint connection configuration provide deeper detail on evaluation setup
A runnable evaluation demo covers endpoint-based, specification-filtered, and local-agent evaluation modes

Platform Improvements

JSON field matching metric: A new deterministic metric that compares top-level JSON fields between actual and expected output, returning the ratio of matching fields as the score. Supports lenient parsing of actual output wrapped in markdown code fences or surrounding text.
Endpoint connection validation: Before running an evaluation, the platform now validates that all required endpoint connections on the version still exist — preventing runtime failures from deleted connections.
Credit warning badge: A visual indicator in the sidebar shows your organization’s credit status — amber when credits are running low, red when exhausted — so you always know where you stand before launching evaluations.
Retry defaults: New endpoint connections default to retries enabled with 3 attempts and exponential backoff, improving reliability out of the box.
Sidebar overlay: The navigation sidebar now overlays content on hover instead of pushing the page layout, reducing visual disruption while browsing.
Form and table polish: Clearer validation errors when switching to AI generation mode, a clear button for optional form fields, improved column width handling in tables, better form field reset behavior, and more consistent UI across the dashboard.

2026-03-16

Smarter Data Generation and Platform Refinements

Improved Golden Dataset Generation

The golden dataset generation pipeline has been redesigned. You can expect more diverse questions, more accurate source attribution, and fewer duplicated or low-quality entries in your generated test cases.

Platform Improvements

Automatic text truncation: Long text in table cells across the dashboard is now automatically clamped to two rows. Click any truncated cell to expand and view the full content rendered as Markdown. This replaces inconsistent truncation behavior across tables and keeps the interface clean without hiding information.
AI metric saving fix: An issue that prevented non-admin users from saving AI-generated metric candidates has been resolved. All team members can now review and accept metric suggestions without encountering permission errors.
Evaluation link fix: The evaluation link generated in the SDK code snippets now navigates to the correct tab, ensuring you land on the evaluations view as expected.

2026-03-09

AI-Powered Metrics, Data Catalogs, and Streamlined Evaluations

AI-Powered Metric Creation

You can now generate evaluation metrics directly from your product’s specifications. Select one or more policy specifications, and the AI analyzes your product description and specification rules to produce ready-to-use metric candidates — complete with judge prompt, evaluation parameters, tags, and evaluator model. You review each candidate, edit anything that needs adjusting, and save the ones you want. Saved metrics are automatically linked back to their source specification.Learn more about the full workflow in the AI Metric Generation documentation.

Data Catalogs for Behavior Tests

Behavior tests can now be grounded in real data. When creating a generated behavior test, you can upload a Data Catalog — a file containing real values like names, IDs, amounts, and dates. The scenario generator injects catalog values into the generated test cases, producing scenarios that reflect your actual data distribution instead of relying on fictional placeholders.

Simplified Test Creation

The test creation form has been reorganized to reduce visual complexity. Optional fields — Custom User Focus, Data Catalog, and Language — are now grouped behind an Advanced Options toggle, keeping the main form focused on required fields and essential choices.

Test Your Endpoint Connection

You can now test your endpoint connection directly from the creation and edit forms. A dedicated test panel sends a sample request to your endpoint and displays the raw response, so you can verify connectivity and response format before saving.

Run Evaluations from Any Level

The evaluation run dialog now supports two new entry points: you can trigger a run from a specific test or from an individual test case, in addition to the existing version, session, and inference result levels. The dialog title, description, and available options adapt to the context you start from. As part of this change, the action has been renamed from “Run Inference” to “Run Evaluation” across the dashboard to better reflect what it does.

SDK: Full Conversation Access in Custom Metrics

The CustomScoreEvaluationMetric.measure() method now receives an optional inference_results parameter containing all conversation turns for the current session. This lets custom metrics reason over the full conversation history — not just the latest turn — when computing scores. This also fixes an issue where evaluating sessions with custom metrics only fetched the most recent turn, making multi-turn custom metric evaluation unreliable. See the custom metrics tutorial for usage details.

Platform Improvements

Failed turn tracking: The evaluation engine now detects which conversation turns failed during a simulated conversation and records them in a new failedTurns field on each evaluation. Failed turns are visible in the evaluation detail view and included in CSV exports, making it easier to pinpoint where multi-turn conversations break down.
Clearer metric validation errors: When a metric is missing required evaluation parameters, the error message now lists exactly which parameters are absent — instead of the generic “at least one is missing” message.
Inference error visibility: When a direct inference call results in an error, the error text is now displayed directly in the inference result view.
Unified form components: AI-assisted creation forms and table selection dialogs across the dashboard now share a consistent design and interaction model. This also fixes an issue where discarding all AI-generated metric candidates left the review step in a dead end with no navigation options.

2026-03-02

Human Evaluators, Smarter Specifications, and Inference Improvements

Human Evaluators

You can now bring humans into the evaluation loop. Human evaluation lets your team members review and annotate inference results directly from the Galtea platform — alongside the automated LLM-as-a-judge scores you already rely on.To get started, create a metric with the evaluation type set to Human, assign it to the relevant user group, and invite your evaluators. Each evaluator sees only the inference results assigned to their group and can submit scores through a dedicated annotation interface.

User Groups

User Groups are the organizational layer behind human evaluation. A user group links a set of team members to specific metrics, so you can control who evaluates what. You can manage user groups from the dashboard or programmatically through the User Group Service in the SDK.

AI-Assisted Specification Configuration

Specification creation is now smarter. AI-assisted configuration has been extended to all specification types — not just policies. When you create a new specification, the system analyzes your description and suggests the test type, test variant, and specification type automatically. You can accept the suggestions with a single click or adjust them before saving.For Policy specifications specifically, the AI also suggests which test type best validates the rule you are defining — helping you choose between ACCURACY, SECURITY, and BEHAVIOR based on the policy’s intent.You can manage specifications from the dashboard or via the Specification Service in the SDK.

Inference Monitoring and Tracing

Direct inference sessions now surface more visibility into what is happening during evaluation runs:

Session status: Each inference session displays a status indicator — COMPLETED, PENDING, or FAILED — so you can quickly identify which sessions need attention.
Failed inference visibility: When a direct inference call fails, the error is now surfaced directly in the session view, rather than being silently logged.
Distributed trace collection: Remote agents can now send traces back to Galtea automatically. Pass the AgentInput object to your agent handler, and all @trace-decorated calls are linked to the correct session. See Tracing Agent Operations for the full setup guide.

Metrics and Evaluation

Flexible text metric validation: Text-based metrics no longer require a strict answer key in the response payload, making it easier to integrate custom output formats.
Version endpoint validation: Starting an evaluation now checks that the selected version has a valid endpoint connection configured, catching misconfigurations before the run begins.
SDK fix — session custom metrics: An issue where actual_output was always None when evaluating sessions with custom metrics has been resolved.
Filter tests by version: You can now filter the test list by the selected version, making it faster to find the tests relevant to your current evaluation run.
Specification-aware validation: Products are now enriched with their linked specifications before evaluation validation runs, ensuring the evaluation engine has the full context it needs.

Platform Improvements

Report terminology: Automated reports now use clearer dimension names — Accuracy instead of “Quality” and Security & Safety instead of “Red Teaming” — matching the terminology you see in the dashboard.
Report visualization: Metric comparison charts now support multi-page groups, so products with more than six metrics per group render correctly without truncation.
Evaluation run form: The evaluation run form layout has been reorganized for a cleaner workflow when configuring new runs.
Improved augmentation: The data augmentation pipeline has received quality improvements to produce more diverse and representative test case variations.
SDK improvements: The SDK now correctly supports JSON request bodies in HTTP DELETE operations and properly escapes JSON string values in inference input templates.

Documentation

A new Human Evaluation tutorial walks you through the full workflow — from creating a human metric and setting up user groups to annotating results in the platform. The tutorial includes step-by-step screenshots showing the metric creation form, the evaluator sidebar, and the annotation interface.

2026-02-23

Specifications, Smarter Reports, and Platform Polish

Specifications

Specifications are now live — a structured way to define and test the behavioral expectations of your product.A Specification represents a single, testable claim about what your product should or should not do. Each specification has a type that classifies the expectation:

Capability — a core function the product can perform
Inability — something the product fundamentally cannot do, regardless of user input
Policy — a rule the product must follow, such as refusing certain requests or always adding a disclaimer

You also assign a test type (ACCURACY, SECURITY, or BEHAVIOR) that determines how the specification is evaluated.Specifications replace the legacy free-text Capabilities, Inabilities, and Policies fields that used to live on the Product form. Your behavioral expectations are now discrete, individually traceable, and can be linked directly to the metrics that validate them.You can manage specifications from the dashboard or via the Specification Service in the SDK.

AI-Assisted Configuration

Specification creation is AI-assisted. With a single click, the system suggests the specification type, test type, and test variant based on your description — so you can focus on defining what your product should do, not on configuring how it gets tested.

Improved Report Generation

The automated report generation pipeline has received a set of quality upgrades:

Realism analysis: Before generating narrative content, the engine now runs a realism check on the underlying data to ensure summaries reflect actual conditions rather than statistical artifacts.
Pattern analysis: A new pattern analysis step examines the diversity and distribution of your test results, informing the structure of each report section. You can toggle this per generation run.

Better Data Augmentation

Data Augmentation now uses a dedicated diversity model alongside the main generation pass. Instead of producing variations that converge on the same patterns, augmented test cases are explicitly steered toward different scenarios, phrasings, and edge cases — giving you broader coverage from the same seed data.A demo video has also been added to the Data Augmentation documentation to walk you through the full workflow.

Simplified Metrics

The Metric entity has been streamlined. Legacy fields — including criteria, evaluation_steps, test_type, user_persona, and stopping_criterias — have been removed from the metric creation form. These were carry-overs from an older architecture that are no longer needed. If you were passing any of these fields through the SDK, you can safely remove them.

Platform Improvements

403 Forbidden page: Accessing a route without the required permissions now shows a clear, dedicated error page with an explanatory message, rather than a silent redirect or blank screen.
Sorting fixes: Table columns that do not support server-side sorting no longer display a sort indicator, removing a common source of confusion in large result sets.
Evaluation prompts: User prompts in evaluations are now rendered through Jinja2, enabling richer template-based customization.

Trace Collection During Platform Simulations

When running evaluations via Direct Inference, you can now collect traces from your endpoint and link them back to Galtea automatically.Add the new {{ inference_result_id }} placeholder to your Endpoint Connection input template. Your handler receives the ID, passes it to the SDK’s set_context, and all @trace-decorated calls made during that request are automatically linked to the correct inference result in Galtea. See Collecting Traces During Direct Inference for the full walkthrough.

Documentation and Examples

Demo videos are now embedded throughout the documentation — covering test generation, Accuracy tests, Security & Safety tests, Behavior tests, endpoint connection setup, and data augmentation — so you can see each workflow in action without leaving the docs.

2026-02-16

Reports, Augmentation, and Performance

Automated Report Generation

You can now export your analytics data as a comprehensive PDF report directly from the Galtea dashboard. The report is generated automatically and includes AI-written summaries for each section — covering scope, methodology, product evaluation, and conclusions — alongside dashboard-style visualizations.

One-Click Test Case Augmentation

Scaling your Test Cases just got much simpler. If you have a small base of known-good test cases, you can now augment them with a single click directly from the dashboard. Galtea will automatically generate additional test cases based on your existing ones, letting you quickly expand coverage without manually writing each entry.

New Quality Test Tasks

We have expanded the Quality Test configuration with two new task types: Correction and Other. When selecting Other, you can provide a custom task description to better classify your tests. This makes it easier to define quality evaluations that go beyond the predefined categories.

New Tutorial: Direct Inferences and Evaluations from the Platform

A new step-by-step guide walks you through the full workflow of running inferences and evaluations directly from the Galtea dashboard — no SDK code required. It covers creating an Endpoint Connection, attaching it to a Version, running tests, and reviewing results.

Performance and Reliability

Faster Evaluations: The evaluation pipeline has been optimized to reduce redundant database lookups. Pre-fetched entities are reused across batch operations, validation order has been improved, and retry logic now groups evaluations by session for significantly faster batch processing.
Improved Error Messages: The IOU and Spatial Match metrics now return clearer error messages specifying the expected JSON formats for bounding box inputs.
Jinja2 Template Validation Fix: The conversation simulator template validator no longer raises false positives for valid Jinja2 for-loop patterns in JSON templates, such as the common OpenAI-compatible message format.
Dashboard Polish: Several UI improvements including better dark mode contrast, refined Pill component styling, and a smarter toast notification system that reduces interruptions for frequent users.

2026-02-09

Enhanced Simulation and UI Improvements

SDK & Simulation Upgrades

We have significantly improved the flexibility of the Galtea SDK to better fit your existing workflows.

Unified Simulation Method: The simulator.simulate method has been upgraded to handle both single-turn and multi-turn simulations. You no longer need to switch methods based on the test case complexity.
Simplified Agent Integration: We’ve made the agent property much more flexible. You can now pass methods with various signatures directly (including async generators), removing the strict requirement to wrap everything in a specific Agent class.

New Platform Features

AI-Powered Product Definition

Bootstrapping a new product context is now faster than ever. You can upload a set of files (documentation, knowledge base, etc.), and Galtea will use AI to automatically generate a comprehensive Product Definition for you.

Advanced Endpoint Connections

For more complex conversational agents, we have refined how connections are established. You can now configure distinct behaviors for:

The Initialization of the conversation.
The Conversation messages.
The Finalization of the conversation.

Single Inference Evaluation

You can now evaluate a single inference result directly from the platform UI. This is perfect for spot-checking model behavior without running a full simulation suite.

Reliability and UX Improvements

Graceful Metric Handling: Previously, using a metric incompatible with a specific Test Type would throw an error. Now, the system intelligently marks these metrics as “skipped” in the results, allowing the rest of your evaluation to proceed uninterrupted.
Enhanced Custom Metrics: We have deprecated all DeepEval metrics in favor of our own Custom Metrics architecture. This transition significantly enhances the reliability, speed, and flexibility of your evaluations.
Test Creation Experience: The Test creation form has been polished for better usability, making it easier to define and organize your test cases.
Platform Stability: General improvements to platform stability and reliability to ensure a smoother experience during heavy load.

2026-02-02

Single-Turn Evaluations & Bulk Actions

Evaluate Single-Turn Interactions

You can now run evaluations on specific single turns (even when directly creating an Inference Result), rather than being restricted to evaluating full threads or sessions. This granularity allows for more precise analysis, enabling you to pinpoint and score individual exchanges within a larger conversation context without the noise of the surrounding dialogue.

Bulk Actions in Tables

We’ve enhanced table functionality across the dashboard to improve your productivity. You can now select multiple rows to perform actions in bulk, such as deleting multiple Test Cases or cleaning up old Sessions simultaneously.

Stability & Performance

Metric Stability: We’ve deployed updates to improve the consistency and reliability of several core metrics.
Platform Resilience: Significant backend optimizations have been implemented to ensure platform stability and maintain low latency, even under periods of high load.

2026-01-26

Run Inferences from Dashboard & UX Improvements

Run Inferences and Evaluations from the Platform

You can now execute inferences and run evaluations directly from the Galtea dashboard. This new capability allows you to quickly test your Product Versions and validate performance without writing a single line of code or switching to your IDE.This streamlined workflow is perfect for:

Quick sanity checks on new model versions.
Running specific test cases ad-hoc.
Validating changes instantly before full-scale testing.

General Improvements

Bug Fixes & UX Polish: We’ve addressed various minor bugs and refined the user interface to provide a smoother, more stable experience across the platform.

2026-01-19

OTel Tracing, New Models, and UX Improvements

Standardized Tracing with OpenTelemetry

Galtea Traces have been upgraded to use the OpenTelemetry (OTel) standard. This major infrastructure update aligns our tracing capabilities with industry standards, paving the way for future seamless integrations with your existing observability tools and monitoring stacks. Read more about traces.

New Models Available

We have expanded the selection of models available for evaluating your products. You can now leverage the latest capabilities of:

Claude-Sonnet-4.5
GPT-5.2, GPT-5.1, GPT-5, and GPT-5-mini

These models can be selected when configuring your AI Evaluation metrics.

Revamped Onboarding & Development Modes

We’ve improved the onboarding experience to better match your workflow:

Use-Case Based Onboarding: The setup process is now split between two clear paths: evaluating a product during Development or monitoring a product in Production with real users.
Platform View Modes: You can now select your preferred view mode in the platform: Development or Production. This automatically filters the UI to remove unnecessary options, keeping your workspace clean and focused on the task at hand.

General Improvements

Enhanced Forms: We’ve standardized and improved forms across the platform for better usability and a more consistent experience.

2026-01-12

AI-Assisted Onboarding, Visual Enhancements, and Docs Reorg

AI-Assisted Product Onboarding

Getting started with Galtea got even simpler. We’ve improved the Product creation flow by simplifying the integration of AI assistance directly into the product creation form. The system helps you draft comprehensive product descriptions, capabilities, and security boundaries, ensuring your product context is perfectly optimized for our test generation engine from day one.

Enhanced Inference Result Visualization

We have revamped how Inference Result metrics are visualized. The new layout provides a clearer breakdown of performance data, making it easier to correlate specific input/output pairs with their respective scores. This improvement allows for quicker diagnosis of issues within specific conversation turns.

Documentation Refresh

We have reorganized our documentation to improve discoverability and ease of use. The structure is now streamlined into clearer categories, making it easier to find SDK references, conceptual guides, and tutorials. Check out here the new docs structure here.

General Improvements & Bug Fixes

General Bug Fixes: Addressed various minor issues to improve platform stability and performance.
UI Polish: Minor visual updates across the dashboard for a more consistent user experience.

2026-01-05

Tool Correctness Metric & New Product UX

Enhanced Product Creation Experience

We have completely revamped the onboarding flow for new products. A new, intuitive form is now available to help you create AI-based products faster and more efficiently. This update streamlines the initial setup configuration, improving the overall user experience when onboarding new agents or LLM apps.

Evaluating Tool Usage

As agents become more autonomous, validating how they use tools is just as important as the final answer. We have introduced new capabilities to strictly evaluate tool calls:

New Property in Test Cases

Test cases now accept a new property: expected_tools. This allows you to define exactly which tools an agent should invoke during a specific test scenario. Read more about Test Case structure.

New Evaluation Parameters

To support this validation, the evaluation process now accepts two specific parameters regarding tool usage:

tools_used: The actual list of tools invoked by the model.
expected_tools: The ground truth list of tools that should have been used.

New Metric: Tool Correctness

We have added a specialized metric to our library: Tool Correctness.This metric automatically compares the tools_used against the expected_tools to determine if the agent selected the right functions to solve the user’s problem. This is critical for ensuring reliability in agentic workflows. See metric details.

2025-12-18

Agent Tracing & Observability

Full Visibility with Agent Tracing

You can now add Traces to your Sessions and Inference Results. This unlocks deep observability into your AI agents, allowing you to understand exactly what they are doing at each step of execution—not just the final output.Traces capture internal operations such as:

Tool Calls: API requests, calculations, or data fetching.
Retrieval Steps: Vector database searches and RAG context retrieval.
Chain Orchestration: Internal routing and decision-making logic.
LLM Invocations: Prompts sent to underlying models.

With the new Trace Service in the SDK, you can visualize the full execution tree, debug errors, analyze latency for specific operations, and audit agent behavior with granular detail.

We’ve also added a comprehensive guide on how to instrument your code. Read the Tracing Tutorial to get started.

2025-11-24

Granular credit tracking, security improvements, and better SDK validation

Granular Credit Consumption and Transparency

We’ve updated our credit system to provide more granularity and transparency. You can now see exactly where your credits are being consumed, giving you detailed visibility into your usage patterns and helping you manage resources more effectively.

Platform Robustness and Security

We’ve improved the overall robustness of the platform to ensure higher uptime and reliability. Additionally, we have increased safety by implementing rate limiting on our authentication endpoints, providing better protection against abuse and unauthorized access attempts.

Improved SDK File Validation

We’ve enhanced file validation within the SDK. When uploading files—for example, when creating a Test using test_file_path or ground_truth_file_path—you will now receive stricter validation and clearer, more descriptive error messages to help you debug issues faster.

2025-11-17

Simplified Metric Creation, Binarized Scores, and Performance Boosts

Simplified and More Powerful Metric Creation with Partial Prompts

We’ve upgraded how custom judge Metrics are created by introducing a new Partial Prompt method. This approach simplifies the process by letting you focus on the core evaluation logic while Galtea handles the final prompt construction. This not only makes creating custom metrics faster but also significantly increases the quality and consistency of the evaluation results. Learn more about the new method.

Clearer Pass/Fail with Binarized Metric Scores

To provide more decisive validation, we’ve updated three of our key metrics to return a clear binary score (0 for fail, 1 for pass). This change eliminates ambiguous 0.5 scores, making it easier to determine success or failure for critical test cases. The updated metrics are:

More Robust Quality Test Generation

When creating Quality Tests from the dashboard, we now validate your uploaded files to ensure they are in the expected format. This proactive check helps prevent errors during test generation, leading to a smoother and more reliable workflow.

Platform Enhancements

We’re always working to improve the core platform experience. This week’s updates include:

Improved Performance: We’ve optimized our underlying LLM usage for better concurrency, scalability, and robustness, resulting in a faster and more reliable platform.
Enhanced Email Validation: The process for validating user emails has been made more robust to ensure better security and deliverability.

2025-11-10

Social Logins, Metric Versioning, and Test Generation Improvements

You can now sign in to Galtea using your Google, GitHub, or GitLab accounts! This makes accessing your workspace faster and more secure, providing a seamless single sign-on (SSO) experience alongside our traditional email and password login.

New social login options for Google, GitHub, and GitLab

Simplified Metric Versioning with “Legacy” Tag

We’ve introduced a new way to handle updated Metrics. Galtea metrics might be marked as “Legacy,” allowing us to improve them over time with the same name. This approach simplifies versioning by removing the need to embed version numbers in metric names, ensuring that your historical evaluations remain linked to the correct metric version.

Test Generation Transparency and Enhancements

We’re bringing more transparency and efficiency to our test generation process:

See What’s Under the Hood: The models used to generate a Test are now listed in the test’s details section, providing clearer insight into how your test cases were created.
Faster Test Generation: We’ve significantly sped up the processing of multiple files within ZIP archives when generating tests from a knowledge base.
Smarter Test Case Handling: The engine for generating Quality Tests now handles the max_test_cases parameter more effectively.

Fixes & Improvements

Fixed a bug where file names from ZIP archives were not displayed correctly in the UI.
Resolved an issue that prevented the processing of ZIP files containing two files with the same name (e.g., folder1/file.txt and folder2/file.txt).

2025-11-03

UI Refresh, Enhanced Security, and Faster Red Teaming

A Fresh New Look for the Dashboard

We’ve polished the dashboard with a subtle but significant UI refresh. Expect updated colors, clearer icons, and more consistent dialogs designed to create a smoother, more intuitive, and visually consistent workflow.

Enhanced Security and Performance

We’ve bolstered the platform’s security with improved user authentication mechanisms to better protect your data. Alongside this, we’ve enhanced how we handle concurrent requests, leading to better performance and reliability, especially during high-traffic periods.

Smarter & Faster Red Teaming Generation

Creating Red Teaming tests is now better than ever. We’ve improved language control during generation for more robust and relevant tests, even when no specific language is provided. Plus, we’ve optimized the generation process for significantly faster test creation, allowing you to build robust security evaluations more efficiently.

2025-10-27

Simplified Metric Creation, Smarter Simulations, and UX Polish

Introducing Partial Prompts: Create Custom Metrics Faster

We’re making it easier than ever to create custom judge metrics with our new Partial Prompt evaluation type. Now, you can focus solely on defining your evaluation criteria and simply select the data parameters you need—like input or retrieval_context. Galtea automatically constructs the full, correctly formatted judge prompt for you, saving you time and reducing complexity.This new approach streamlines the creation of powerful, tailored Metrics, allowing you to concentrate on what matters most: the evaluation logic itself.

Creating a metric with the new Partial Prompt method

Quality Generator with Improved Language Handling

The LLM language confusion is now patched for the quality generator. Languages are formally detected for the whole document, and test cases are ensured to be generated with the correct language. This ensures more accurate and contextually appropriate test case generation across different language inputs.

Improved Long Context Processing for Quality Test Cases

Our quality generator now intelligently processes long documents, ensuring optimal context is provided for the best quality generation of test cases. This enhancement streamlines the evaluation process and improves the reliability of generated test cases for comprehensive quality assessments.

Platform Experience Enhancements

We’ve also rolled out several quality-of-life improvements and bug fixes for a smoother experience. A notable update is that tables across the platform now refresh instantly after you delete an item with a right-click, making data management more fluid and intuitive.

2025-10-15

Product Policies, New Red Teaming Strategies, and Smarter Test Generation

Define Product Behavior with Policies

Products now accept a set of policies describing how they should behave in specific situations. This allows you to set explicit rules, such as requiring a disclaimer for financial advice or defining standard responses for out-of-scope questions, to ensure consistent and safe model behavior.

Advanced Red Teaming Strategies

We’ve added five new, sophisticated red teaming strategies to help you uncover more complex vulnerabilities. These strategies use your Product Description to create highly contextual and evasive prompts:

Persuasive Content: Disguises malicious requests as legitimate business tasks.
Creative Writing: Reframes harmful prompts as creative exercises.
Data Analysis: Hides malicious intent within analytical or data-generation tasks.
Bait and Switch: Lowers model defenses with benign queries before introducing the adversarial prompt.
Empathetic Framing: Uses emotional manipulation to pressure the model into unsafe compliance.

Smarter Test Generation Engine

Our test generation capabilities have been significantly upgraded for both Quality and Red Teaming tests:

New Quality Test Engine: The engine for generating Quality Tests has been rebuilt to produce higher-quality test cases. It can now also incorporate and validate the new product policies you define.
Faster Red Teaming Generation: The generation process for Red Teaming Tests is now more reliable and significantly faster, allowing you to build robust security evaluations more efficiently.

Platform Usability and Performance

We’ve rolled out several enhancements to improve your workflow and the platform’s reliability:

Deletion Confirmations: To prevent accidental data loss, the platform will now ask for confirmation before deleting key entities like Test Cases, Evaluations, and Versions.
Improved Table Experience: The performance and user experience of data tables and their filters have been optimized for a smoother, more responsive interface.

2025-10-06

Platform Simplification, New Conversational Metrics, and Major Usability Upgrades

Platform Simplification and SDK v3.0

We’ve undertaken a major simplification of our core concepts to make the platform more intuitive. Evaluation Tasks are now simply Evaluations, and Metric Types have been renamed to Metrics. The old parent Evaluations entity has been removed entirely. These changes streamline the workflow and clarify the relationship between different parts of Galtea.To support this, we’ve released a new major version of our SDK.

New Conversational Metrics

We’re excited to introduce two new metrics designed specifically for evaluating conversational AI:

User Objective Accomplished: Evaluates whether the user’s stated goal was successfully and correctly achieved during the conversation.
User Satisfaction: Assesses the user’s overall experience, focusing on efficiency and sentiment, to gauge their satisfaction with the interaction.

Enhanced Test Case Feedback and Management

Improving test quality is now a more collaborative process. When upvoting or downvoting a Test Case, you can now add a user_score_reason to provide valuable context for your feedback.Additionally, you can now filter test cases by their score directly via the SDK using the user_score parameter in the test_cases.list() method.

Dashboard and SDK Usability Improvements

We’ve rolled out several updates to make your workflow smoother and more efficient:

Improved Dashboard Navigation: Navigating between related entities like Tests, Test Cases, and Evaluations is now more intuitive. We’ve also adjusted table interactions—you can now single-click a row to select and copy text without navigating away. To see an entity’s details, simply right-click the row.
Efficient Batch Fetching in SDK: The SDK now allows you to fetch objects by providing a list of IDs (e.g., fetching all Test Cases for a list of Test IDs at once), significantly improving usability for batch operations.

General Improvements

This release also includes numerous performance optimizations and minor UI/UX enhancements across the entire platform to provide a faster and more polished experience.

2025-09-29

Generate Scenarios from Quality Tests, URL Validation Metric, and Platform Boost

Create Conversational Scenarios from Your Quality Tests

You can now generate comprehensive Scenario Based Tests directly from your existing Quality Tests. This powerful feature allows you to transform your single-turn, gold-standard test cases into realistic, multi-turn conversational scenarios, significantly accelerating the process of evaluating your AI’s dialogue capabilities.

New Deterministic Metric: URL Validation

We’ve added the URL Validation metric to our deterministic evaluation suite. It ensures that all URLs in your model’s output are safe, properly formatted with HTTPS, and resolvable. It includes strict validation and SSRF protection, making it essential for any application that generates external links.

Platform Enhancements and Performance Boost

We’ve rolled out major performance improvements across the platform, with a special focus on the analytics views, making them faster and more responsive. This update also includes several minor visual fixes, such as ensuring icons render correctly in different themes, to provide a smoother and more polished user experience.

2025-09-22

Consistent Sorting, Improved Navigation, and SAML SSO

We’ve made significant usability improvements to the platform. Table sorting is now more consistent and persistent; you can navigate away, refresh the page, and your chosen sort order will remain, ensuring a smoother workflow. Additionally, navigating from an Evaluation is easier than ever. You can now right-click on any evaluation in the table or use the dropdown menu in the details page to jump directly to the related Test Case or Session.

Right-click navigation from the Evaluation table

Upvote and Downvote Test Cases

To help teams better curate their test suites, we’ve introduced a voting system for Test Cases. You can now upvote or downvote test cases directly from the Dashboard, providing a quick feedback loop on test case quality and relevance.

Upvote and downvote buttons for Test Cases

SAML SSO Authentication

Organizations can now enhance their security by configuring SAML SSO for authentication. This allows for seamless and secure access to the Galtea platform through your existing identity provider.

Improved Misuse Resilience Metric

The Misuse Resilience metric has been enhanced to accept the full product context. This allows for a more accurate and comprehensive evaluation of your model’s ability to resist misuse by leveraging a deeper understanding of your product’s intended capabilities and boundaries.

New Analytics Filter

The analytics page now includes a filter for Test Case language. This allows you to narrow down your analysis and gain more precise insights into the performance of your multilingual models.

2025-09-15

Test Case Confidence, Custom Judges for Conversations, and More

Confidence Scores for Generated Test Cases

We’re introducing confidence scores for all generated Test Cases. This new feature provides a clear indicator of the quality and reliability of each test case, helping you better understand your test suites and prioritize human review efforts. Higher scores indicate greater confidence in the test case’s relevance and accuracy.

Simplified and More Flexible Metric Creation

Creating Metrics is now more intuitive and flexible. Metrics are now directly linked to a specific test category (QUALITY, RED_TEAMING, or SCENARIOS), which simplifies the creation process by tailoring the available parameters to the relevant test category. This change makes it easier to define metrics that are perfectly aligned with your evaluation goals. See the updated creation guide for more details.

Custom Judges for Conversational Evaluation

You can now create your own Custom Judges specifically for evaluating multi-turn conversations. This powerful feature allows you to define complex, stateful evaluation logic that assesses the entire dialogue, enabling you to measure nuanced aspects like task completion, context retention, and persona adherence across multiple turns. Learn more in our guide to evaluating conversations.

Conversation Simulator Enhancements

We’ve added more control and realism to the Conversation Simulator:

Agent-First Interactions: The simulate method now includes an agent_goes_first parameter, allowing you to test scenarios where the agent initiates the conversation. See the SDK docs.
Selectable Conversation Styles: When creating a Scenario Based Test, you can now choose a conversation style (written or spoken). This influences the tone and formality of the simulated user’s dialogue, enabling more realistic testing. This is available under the strategies parameter in the test creation method.

2025-09-08

Smarter Simulations, Bulk Uploads, and Streamlined Onboarding

More Realistic Conversation Simulations

Our Conversation Simulator has been significantly upgraded to generate more realistic and human-like interactions. The simulator’s evaluation of stopping criteria is now more precise, ensuring that your multi-turn dialogue tests conclude under the correct conditions and provide more accurate insights into your AI’s conversational abilities.Additionally, conversations generated during simulations are now linked to the specific product version they were created with, allowing for better tracking and traceability.

To enable version tracking for simulated conversations, please upgrade to Galtea SDK version 1.8.1 or higher.

Bulk Document Uploads for Test Generation

You can now upload multiple files at once using ZIP archives to generate comprehensive test suites. This feature streamlines the process of creating tests from large document collections, saving you time and effort when building out your evaluation scenarios. For more details, see the test creation documentation.

Streamlined Onboarding Experience

Getting started with a new product is now faster than ever. When you register a new product in the Galtea dashboard, an initial version is now created automatically. This simplification streamlines the setup process and helps you get to your first evaluation more quickly.

Enhanced Document Processing

We’ve improved our backend for processing documents used in test generation. This enhancement leads to more accurate and relevant test cases being created from your knowledge bases, improving the overall quality of your evaluations.

2025-09-01

Galtea Rebranding, Enhanced Analytics, and Platform Security

Fresh New Look: Galtea Rebranding is Live!

We’re excited to unveil Galtea’s complete visual transformation! Our new branding includes a refreshed logo, updated slogan, and a modern look across the entire platform. Experience the new design on our website and explore the updated dashboard interface. This rebrand reflects our continued commitment to providing a more intuitive and visually appealing user experience.

Enhanced Analytics with Improved Version Comparisons

The Analytics views have received significant improvements to make data analysis more efficient and user-friendly. Key enhancements include:

Streamlined Chart Comparisons: Charts now support easier side-by-side version comparisons, helping you quickly identify performance differences across iterations.
Optimized Filter Layout: Filters no longer occupy valuable screen real estate, giving you more space to focus on your analytics data and insights.
Improved Visual Clarity: Enhanced data presentation makes it easier to interpret results and make informed decisions about your AI systems.

These improvements build upon our existing analytics capabilities to provide a more seamless evaluation experience.

Strengthened Authentication and Security

We’ve implemented more secure and robust authentication mechanisms across the platform. These behind-the-scenes security enhancements provide better protection for your data and ensure reliable access to your Galtea workspace, giving you peace of mind when working with sensitive AI evaluation data.

Advanced PDF Knowledge Extraction

Our document processing capabilities have been significantly improved for PDF files. The enhanced extraction algorithms now provide more accurate and comprehensive knowledge retrieval from PDF documents, making it easier to create relevant test cases and evaluation scenarios from your existing documentation and knowledge bases.

2025-08-11

Dashboard Enhancements: TestCase Management, Judge Templates, and System Resilience

TestCase Scenarios Creation and Edition via Dashboard

You can now create and edit TestCase Scenarios directly through the Dashboard interface. This streamlined workflow allows you to define complex multi-turn conversation scenarios without leaving the platform, making it easier to set up comprehensive testing workflows for your conversational AI systems.

Enhanced TestCase Dashboard with Test-Specific Columns

We’ve improved the TestCase Dashboard with smarter table displays that now only show columns relevant to each specific test. This reduces visual clutter and makes it easier to focus on the information that matters most for your particular testing scenario, whether you’re working with single-turn evaluations, conversation scenarios, or other tests.

New Judge Template Selector for Judge Metrics

When creating Judge metrics, you can now select from pre-built judge templates to accelerate your metric setup process. This feature provides a starting point for common evaluation patterns while still allowing full customization of your evaluation prompts and scoring logic.

Infrastructure Improvements for Event Resilience

We’ve enhanced our event handling infrastructure to provide better resilience against unexpected system events. These improvements help ensure that your tests and evaluations are preserved and continue running smoothly, even during system maintenance or unexpected interruptions.

2025-08-04

Create Custom Judges via Own Prompts in Metric Creation, New Deterministic Metrics and UI Improvements

Create Custom Judges via Your Own Prompts

You can now define Custom Judge metrics by crafting your own evaluation prompts during metric creation. This allows you to encode your domain-specific rubrics, product constraints, or behavioral guidelines directly into the metric—giving you precise control over how LLM outputs are assessed. Simply write your prompt, specify the scoring logic, and Galtea will leverage LLM-as-a-judge techniques to evaluate outputs according to your standards.

New Deterministic Metrics

Four new deterministic metrics are now available:

Text Similarity: Quantifies how closely two texts resemble each other using character-level fuzzy matching.
Text Match: Checks if generated text is sufficiently similar to a reference string, returning a pass/fail based on a threshold.
Spatial Match: Verifies if a predicted box aligns with a reference box using IoU scoring, producing a pass/fail result.
IoU (Intersection over Union): Computes the overlap ratio between predicted and reference boxes for alignment and detection tasks.

These new metrics complement our existing evaluation suite and are perfect for scenarios requiring deterministic, rule-based scoring.

Dashboard Redesign: Second Iteration

We’ve launched the second iteration of our redesigned dashboard with a refreshed visual language focused on clarity and usability. Key improvements include:

Modern Forms: Forms have been modernized to provide a more intuitive and visually appealing user experience, as well as giving a more professional look to the Dashboard.

This redesign represents our commitment to delivering a more intuitive and visually appealing user experience while maintaining the powerful functionality you rely on.

2025-07-28

New Evaluator Models, Classic Metrics, and Dashboard Redesign

Expanded Evaluator Model Support

We’ve added support for more evaluator models to enhance your evaluation capabilities:

Gemini-2.5-Flash: Google’s latest high-performance model optimized for speed and accuracy
Gemini-2.5-Flash-Lite: A lightweight variant offering faster processing with efficient resource usage
Gemini-2.0-Flash: Google’s established model providing reliable evaluation performance

You can now link these models to specific Metrics for consistent evaluation results and leverage specialized models for different evaluation scenarios. Learn more about configuring evaluator models in our SDK documentation.

Enhanced Conversation Simulation

Testing conversational AI just got more powerful with two major improvements:

Visible Stopping Reasons: You can now see exactly why simulated conversations ended in the dashboard, providing crucial insights into dialogue flow and helping you identify areas for improvement.
Custom User Persona Definitions: Create highly specific user personas when generating Scenario Based Tests. Define detailed user backgrounds, goals, and behaviors to test how your AI handles diverse user interactions more effectively.

These enhancements work seamlessly with our Conversation Simulator to deliver more realistic and insightful testing scenarios.

Classic NLP Metrics Now Available

We’ve expanded our metric library with three essential deterministic metrics for precise text evaluation:

BLEU: Measures n-gram overlap between generated and reference text, ideal for machine translation and constrained generation tasks.
ROUGE: Evaluates summarization quality by measuring the longest common subsequence between candidate and reference summaries.
METEOR: Assesses translation and paraphrasing by aligning words using exact matches, stems, and synonyms for more nuanced evaluation.

These classic metrics complement our existing evaluation suite and are perfect for scenarios requiring deterministic, rule-based scoring.

Enhanced Red Teaming with Jailbreak Resilience v2

Security testing gets an upgrade with Jailbreak Resilience v2, an improved version of our jailbreak resistance metric. This enhanced evaluation provides more comprehensive assessment of your model’s ability to resist adversarial prompts and maintain safety boundaries across various attack vectors.

Dashboard Redesign: First Iteration

We’ve launched the first iteration of our redesigned dashboard with a refreshed visual language focused on clarity and usability. Key improvements include:

Modern Typography: Cleaner, more readable text throughout the platform
Refined UI Elements: Updated buttons, cards, and form elements with reduced rounded corners for a more contemporary look
Streamlined Tables: Enhanced data presentation with improved content layout
Updated Login Experience: A more polished and user-friendly authentication flow

This redesign represents our commitment to delivering a more intuitive and visually appealing user experience while maintaining the powerful functionality you rely on.

Improved SDK Documentation

We’ve enhanced our SDK documentation with clearer guidance on defining evaluator models for metrics, making it easier to configure and customize your evaluation workflows.

2025-07-21

Conversation Simulation, New Data Leakage Metric, and Platform Enhancements

Test Your Chatbots with Simulated Conversations

It is now possible to generate tests that simulate realistic, multi-turn user interactions. Our new Scenario Based Tests allow you to define user personas and goals to evaluate how well your conversational AI handles complex dialogues.This feature is powered by the Conversation Simulator, which programmatically runs these scenarios to test dialogue flow, context handling, and task completion. Get started with our new Simulating User Conversations tutorial.

New Red Teaming Metric: Data Leakage

We’ve added the Data Leakage metric to our suite of Red Teaming evaluations. This metric assesses whether your LLM returns content that could contain sensitive information, such as PII, financial data, or proprietary business data. It is crucial for ensuring your applications are secure and privacy-compliant.

Enhanced Metric Management

We’ve rolled out several improvements to make metric creation and management more powerful and intuitive:

Link Metrics to Specific Models: You can now associate a Metric with a specific evaluator model (e.g., “GPT-4.1”). This ensures consistency across evaluation runs and allows you to use specialized models for certain metrics.
Simplified Custom Scoring: We’ve introduced a more streamlined method for defining and calculating scores for your own deterministic metrics using the CustomScoreEvaluationMetric class. This makes it easier to integrate your custom, rule-based logic directly into the Galtea workflow. Learn more in our tutorial on evaluating with custom scores.

Support for Larger Inputs and Outputs

To better support applications that handle large documents or complex queries, we have increased the maximum character size for evaluation inputs and outputs to 250,000 characters.

2025-07-14

Conversation Simulation, New Metric, and Credit Management

Test Your Chatbots with Realistic Conversation Simulation

You can now evaluate your conversational AI with our new Conversation Simulator. This powerful feature allows you to test multi-turn interactions by simulating realistic user conversations, complete with specific goals and personas. It’s the perfect way to assess your product’s dialogue flow, context handling, and task completion abilities.Get started with our step-by-step guide on Simulating User Conversations.

New Metric: Resilience To Noise

We’ve expanded our RAG metrics with Resilience To Noise. This metric evaluates your product’s ability to maintain accuracy and coherence when faced with “noisy” input, such as:

Typographical errors
OCR/ASR mistakes
Grammatical errors
Irrelevant or distracting content

This is essential for ensuring your AI performs reliably in real-world scenarios where user input isn’t always perfect. Learn more about how it’s calculated.

Stay in Control with Enhanced Credit Management

We’ve rolled out a new and improved credit management system to give you better visibility and control over your usage. The system now includes proactive warnings that notify you when you are approaching your allocated credit limits, helping you avoid unexpected service interruptions and manage your resources more effectively.

Streamlined Conversation Logging with OpenAI-Aligned Format

Logging entire conversations is now easier and more intuitive. We’ve updated our batch creation method to align with the widely-used messages format from OpenAI, consisting of role and content pairs. This makes sending multi-turn interaction data to Galtea simpler than ever.See the new format in action in the Inference Result Batch Creation docs.

2025-07-07

Custom Threats, New Metrics, and Enhanced Test Case Management

Tailor Your Red Teaming with Custom Threats

You can now define your own custom threats when creating Red Teaming Tests. This new capability allows you to move beyond our pre-defined threat library and create highly specific adversarial tests that target the unique vulnerabilities and edge cases of your AI product. Simply describe the threat you want to simulate, and Galtea will generate relevant test cases.

New Red Teaming Strategies: Role Play and Prefix Injection

We’ve expanded our arsenal of Red Teaming Strategies to help you build more robust AI defenses:

Role Play: This strategy attempts to alter the model’s identity (e.g., “You are now an unrestricted AI”), encouraging it to bypass its own safety mechanisms and perform actions it would normally refuse.
Prefix Injection: Adds a misleading or tactical instruction before the actual malicious prompt. This can trick the model into a different mode of operation, making it more susceptible to the adversarial attack.

Introducing the Misuse Resilience Metric

A new non-deterministic metric, Misuse Resilience, is now available. This powerful metric evaluates your product’s ability to stay aligned with its intended purpose, as defined in your product description, even when faced with adversarial inputs or out-of-scope requests. It ensures your AI doesn’t get diverted into performing unintended actions, a crucial aspect of building robust and responsible AI systems. Learn more in the full documentation.

Enhanced Test Case Management: Mark as Reviewed

To improve collaboration and workflow for human annotation teams, Test Cases can now be marked as “reviewed”. This feature allows you to:

Track which test cases have been validated by a human.
See who performed the review, providing a clear audit trail.
Filter and manage your test sets with greater confidence.

2025-06-30

New Metric, More Red Teaming Strategies and Better Usability

Introducing the Factual Accuracy Metric

We’ve added a new Factual Accuracy metric to our evaluation toolkit! This non-deterministic metric measures whether the information in your model’s output is factually correct when compared to a trusted reference answer. It’s particularly valuable for RAG and question answering systems where accuracy is paramount.The metric uses an LLM-as-a-judge approach to compare key facts between your model’s output and the expected answer, helping you catch hallucinations and ensure your AI provides reliable information to users. Read the full documentation here.

Enhanced Red Teaming with New Attack Strategies

Our red teaming capabilities just got more sophisticated! We’ve added two powerful new attack strategies:

Biblical Strategy: Transforms adversarial prompts into biblical/ancient scripture style using poetic and symbolic language to disguise malicious intent while preserving meaning.
Math Prompt Strategy: Encodes harmful requests into formal mathematical notation using group theory concepts to obscure the intent from standard text analysis.

These strategies join our existing arsenal to help you test your AI’s defenses against increasingly creative attack vectors that real-world adversaries might use. See all available red teaming strategies.

Smarter Red Teaming Test Generation

We’ve significantly improved how red teaming tests are generated. Our system now takes even more factors into account when creating adversarial test cases:

Product-Aware Generation: Tests are now more precisely tailored to your specific product’s strengths, weaknesses, and operational boundaries.
Context-Sensitive Attacks: The generation process better understands your product’s intended use cases to craft more relevant and challenging scenarios.
Enhanced Threat Modeling: Our algorithms now consider a broader range of factors when determining the most effective attack vectors for your particular AI system.

This means your red teaming tests will be more effective at uncovering real vulnerabilities and edge cases specific to your product.

Better Metric Source Visibility and Management

Understanding where your metrics come from is now easier than ever! We’ve enhanced the platform to provide clearer visibility into metric sources:

Source Classification: All metrics are now clearly labeled with their source - whether they’re from established frameworks, custom Galtea implementations, or other origins.
Enhanced Filtering: You can now filter metrics by their source to quickly find the evaluation criteria that best fit your needs.
Improved Descriptions: Metric descriptions now include more detailed information about their origins and implementation, with links to relevant documentation.

Enhanced Metrics Table with Source Visibility

The three main metric sources you’ll see are:

Galtea: Custom metrics designed specifically for your needs, like our new Factual Accuracy metric
G-Eval: Framework-based metrics that use evaluation criteria or steps for assessment
Established Frameworks: Metrics adapted from proven evaluation libraries and methodologies

This transparency helps you make more informed decisions about which metrics to use for your specific evaluation needs.

2025-06-23

Quality Test Generation, Metric Tags, and Product Details

Generate Quality Tests from Examples

You can now create Quality Tests directly from your own examples using the new Few Shots parameter. This makes it easier to tailor tests to your specific use cases and ensure your models are evaluated on the scenarios that matter most. Learn more about test creation.

Metric Tags

Metrics now support tags for easier classification and discovery. Quickly find and organize metrics relevant to your projects. See all metrics.

Enhanced Product Details

Products now include new detail fields:

Capabilities: What your product can do.
Inabilities: Known limitations.
Security Boundaries: Define the security scope and constraints.

These additions help you document and communicate your product’s strengths and boundaries more clearly. Read about product details.

Improved Q&A Generation

Question-answer pairs are now generated with improved accuracy and clarity, thanks to better text filtering and processing.

New Guide: Setting Up Your Product Description

We’ve created a comprehensive guide to help you set up your product descriptions effectively. This guide covers best practices, examples, and tips to ensure your product is presented in the best light. Check it out here

General Improvements

We’ve made various bug fixes and UX/UI improvements across the Dashboard, SDK, and more, making your experience smoother and more reliable.

2025-06-16

Major Platform Overhaul & SDK v2

Major Platform Overhaul

We’ve been hard at work reorganizing and expanding the Galtea platform to handle more use cases and prepare for exciting future features. This release brings significant improvements to the dashboard, SDK, and test generation.

Dashboard Enhancements

Reorganized Version, Test, and Evaluation Views:
Detailed views have been streamlined and improved for clearer insights.
New Sessions Visualizations:
Easily organize and navigate conversations through our new Sessions feature.
Evaluations Visualization Removed:
The dashboard now focuses on Sessions and Evaluations as the primary elements.
Better Filters Across Tables:
Quickly find what you need with improved filtering capabilities on the dashboard.
General Bug Fixes & UX Improvements:
Enjoy smoother interactions, clearer tooltips, and more intuitive code snippets.

SDK v2 Released

The new Galtea SDK v2 is here! It includes breaking changes to simplify workflows and add session support.

Implicit Session Creation:
Sessions are created automatically when needed for evaluations.
Repurposed evaluations.create():
The old method is replaced by create_single_turn() for test-based evaluations, while create() now exclusively handles session-based evaluations.
New evaluations.create_single_turn() Method:
Use this for single-turn test cases. It now requires version_id instead of evaluation_id.
Simplified Version Creation:
The galtea.versions.create() method now accepts all properties directly, no need for an optional_props dictionary.
Sessions Support:
Group multiple inference results under a single session for better multi-turn tracking using galtea.sessions.create().

Improved Test Case Generation

Smarter Test Coverage:
Test cases are now distributed more intelligently across your documents for better coverage based on the number of questions you choose to generate.
Single Threat per Red Teaming Test:
Red Teaming Tests now only allow a single threat per test, ensuring clearer results.

2025-05-26

Enhanced Test Generation & Streamlined Workflow with Code Snippets

Improved Test Generation

Our test generation capabilities have been significantly upgraded:

Versatile Red Teaming: Red Teaming tests are now more powerful, allowing you to employ multiple attack strategies to thoroughly probe your AI’s defenses.
Better Synthetic Data: We’ve made general improvements to the quality of synthetic data generation, ensuring your tests are more effective and realistic.

Code Snippets Now Available on the Dashboard

We’re making it easier than ever to integrate Galtea into your development process!

Simplified Evaluation Setup: The “Create Evaluation” form on the dashboard has been replaced with a convenient code snippet. Simply copy and paste it directly into your project to get started.
Streamlined Creation: Similarly, a new code snippet for “Create Evaluation” is now available on the dashboard, simplifying how you send evaluation data to Galtea. You can easily copy and paste this into your project.

Usability Improvements

We’ve also rolled out several usability enhancements based on your feedback:

Enhanced Readability in Tables: Table cells now correctly render line breaks, making it easier to view multi-line content and detailed information at a glance.
Controlled Test Case Generation: To ensure optimal performance and manageability, the maximum number of test cases automatically generated for a single test from a knowledge base is now capped at 1000.

2025-05-19

Platform Upgrades: Easier Onboarding, Improved UI & Finer Control

Streamlined Onboarding and Quicker Starts

We’ve revamped the platform onboarding! It’s now more visually intuitive and to help new users get evaluating in no time, we now provide a default Metric and a default Test. This makes it easier than ever to get started with Galtea and run your first evaluation quickly.

Deeper Insights with Visible Conversation Turns

Understanding the full context of interactions is key. You can now view the complete conversation turns associated with your test cases directly within the dashboard. This offers richer context, aiding in more thorough analysis and debugging of your conversational AI products.

Dashboard Usability Boost

We’re continually refining the Galtea experience. This update brings several UI enhancements across the dashboard, designed to improve overall usability and make your workflow smoother and more intuitive.

Tailor Your Test Generation: Selectable Test Case Counts

Gain more control over your testing process! When generating tests, you can now specify the exact number of test cases you want Galtea to create. This allows you to fine-tune the scope and depth of your tests according to your needs.

Track Your Team’s Work: Creator Attribution Displayed

Clarity in collaboration is important. Now, the user who created a Product, Test, Version, or other key assets will be clearly displayed on their respective details pages. This helps in tracking ownership and contributions within your team.Working with data tables in the dashboard is now more efficient:

Clear Filter Indicators: Easily see which filters are currently applied to any table.
Quick Filter Reset: A new “Clear All Filters” button allows you to reset your view with a single click.

2025-05-12

New Conversation Evaluation and Extended Data Generation Capabilities

New Conversation Evaluation Metrics

You can now evaluate conversations using these new metrics:

Role Adherence - Assess how well an AI stays within its defined role
Knowledge Retention - Measure how effectively information is remembered throughout a conversation
Conversation Completeness - Evaluate whether all user queries were fully addressed
Conversation Relevancy - Determine if responses remain on-topic and purposeful

Enhanced Security Framework

We’ve significantly improved user access management by implementing an Attribute-Based Access Control (ABAC) strategy, providing more granular control over who can access what within your organization.

Extended Data Generation Capabilities

Our data generation tools have been expanded with:

Catalan Language Support - Create synthetic data in Catalan to enhance your multilingual applications
Added support for text-based files - Upload your knowledge base in virtually any text-based format including JSON, HTML, Markdown, and more

Improved Test Creation Experience

We’ve enhanced the clarity of threat selection in the Test Creation form. The selection now displays both the threat and which security frameworks that threat covers, making it easier to align your testing with specific security standards.

Reduced Clutter in Analytics Filters - Tests and Versions filtering now only display elements that have been used in an evaluation
Streamlined Navigation - Clicking the “input” cell in the evaluations table now navigates directly to the associated Test Case

Bug Fixes & Improvements

We’ve resolved several issues to ensure a smoother experience:

Fixed a bug that could trigger an infinite loop in the Test Cases List of the dashboard
Addressed multiple small UI glitches and errors throughout the platform

2025-05-05

Analytics Upgrades and Red Teaming Test Improvements

Improvements in Red Teaming Tests

New “misuse” threat implemented
Now red teaming incorporates a new threat, misuse, which are queries that not necessaryly malicious however out-of-scope for you specific product. You can now test whether your product can successfully block these queries by marking “Mitre Atlas: Ambiguous prompts” in the threat list.
Better “data leakage” and “toxicity” tests
The red teaming tests incorporate better your product meta data, to generate the most adequate test cases for “data leakage” and “toxicity”.

Analytics Page Upgrades

We’re continuing to expand the power of the Analytics page! This update introduces:

Radar View for Version Comparison
You can now visualize performance across multiple metrics for a single version using the brand-new radar view. It provides a quick way to understand strengths and weaknesses at a glance.
Smarter Metric Filters
Filters now only show metrics that have actually been used in evaluations—removing unnecessary clutter and making it easier to find relevant data.
Graph Tooltips
Hovering over truncated names now reveals full labels with tooltips, helping you understand graph contents more clearly.

SDK Safeguards

We’ve added protections to ensure your SDK integration is as smooth and reliable as possible:

Version Compatibility Checks
If the SDK version you’re using is not compatible with the current API, it will now throw a clear error to prevent unexpected behavior.
Update Notifications
When a new SDK version is available, you’ll get a console message with update information—keeping you in the loop without being intrusive.

Bug Fixes

Metric Range Calculation
Some default metrics were previously displaying inverted scoring scales (e.g., treating 0% as best and 100% as worst). This is now resolved for accurate interpretation.
Test Creation Not Possible Through .txt Knowledge Base Files
Due to a recent refactor, the creation of tests using knowledge base files with .txt extensions was not possible. This has been fixed and you can now create tests using .txt files as the knowledge base again.

2025-04-28

Monitorization and UI improvements

Monitoring Is Live!

Real-world user interactions with your products can now be fully monitored and analyzed. Using the Galtea SDK, you can trigger evaluations in a production environment and view how different versions perform with real users. Read more here.

Improved Galtea Red Teaming Tests

Our simulation-generated tests have been upgraded—delivering higher-quality outcomes. Red teaming tests can now be directed to validate even more specific aspects of various security standards, such as OWASP, MITRE ATLAS, and NIST. Specifically, we have improved jailbreak attacks, in addition to new financial attacks and toxicity prompts.

New Analytics Page

A completely redesigned analytics page is now available! It features:

Enhanced Filtering Capabilities.
Improved Data Clarity and Layout.

The new design not only raises the clarity and density of data presentation but also improves your overall user experience.

And with monitoring active, you can see production evaluation results in real time on this page!

User Experience Enhancements

We’re continuously refining the platform based on your feedback. This week’s improvements include:

Customizable Evaluations List:
You can now select which metrics you are interested in, so the evaluations list only shows the ones you need.
Enhanced Evaluation List Filtering:
Easily filter evaluations by versions, evaluations, tests and test groups.
Enhanced Test List Filtering:
Easily filter tests by its group.
Smart Table Sorting:
When you apply a custom sort, the default (usually creation date) is automatically disabled.

​Track Changes to Your Test Cases

​Compare Version Coverage in Analytics

​Platform Improvements

​Run Voice Simulations on Your Phone Agents

​Update Specifications from a Document

​Product Cards Show Version Scores

​Cross-Product Ranking in Val

​Platform Improvements

​Platform Improvements

​Documentation and Examples

​Smoother Onboarding

​Platform Improvements

​Documentation and Examples

​Analytics Insights

​Smarter Run Evaluation Flow

​Complete Metric with AI

​AI-Assisted Specification Creation

​Import Endpoint Connection from cURL

​Platform Improvements

​Galtea Assistant Redesign

​Redesigned Onboarding Experience

​Word Document Upload for Product Setup

​Job Cancellation

​Platform Improvements

​Metric Optimization Lifecycle

​Platform Improvements

​Documentation and Examples

​Import Langfuse Traces

​SDK: Model Management

​Platform Improvements

​Metric Optimization

​Galtea Assistant (Beta)

​Form Improvements

​Platform Improvements

​Specifications View Redesign

​Two-Factor Authentication

​Documentation and Examples

​Platform Improvements

​Galtea CLI

​Product Tags

​Smarter Pre-Evaluation Guidance

​Structured Inputs for AI-Generated Tests

​Documentation and Examples

​Platform Improvements

​Metric Versioning with Evaluation Replay

​Galtea Agent Skill

​Run Evaluations from Specifications

​Language Variables in Input Templates

​Reliable Trace-Dependent Evaluations

​Export Evaluation Results as CSV

​Inference Result Error Tracking

​One-Click Metric Duplication

​Platform Improvements

​Human Annotation for All Evaluations

​SDK: Endpoint Connection Management

​SDK: Job-Based wait_for

​SDK: Langfuse CallbackHandler

​Structured JSON Inputs

​Langfuse Integration

​Public REST API Reference

​Platform Improvements

​Zero-Code Trace Ingestion via OpenTelemetry

​Structured JSON Inputs

​SDK: Synchronous Evaluation Polling

​Normalized JSON Field Match Metric

​Editable Test Metadata

​Platform Improvements

​Specifications in Analytics

​Test Case CSV Export

​Documentation and Video Guides

​Platform Improvements

​Improved Golden Dataset Generation

​Platform Improvements

​AI-Powered Metric Creation

​Data Catalogs for Behavior Tests

​Simplified Test Creation

​Test Your Endpoint Connection

​Run Evaluations from Any Level

​SDK: Full Conversation Access in Custom Metrics

​Platform Improvements

Track Changes to Your Test Cases

Compare Version Coverage in Analytics

Platform Improvements

Run Voice Simulations on Your Phone Agents

Update Specifications from a Document

Product Cards Show Version Scores

Cross-Product Ranking in Val

Platform Improvements

Platform Improvements

Documentation and Examples

Smoother Onboarding

Platform Improvements

Documentation and Examples

Analytics Insights

Smarter Run Evaluation Flow

Complete Metric with AI

AI-Assisted Specification Creation

Import Endpoint Connection from cURL

Platform Improvements

Galtea Assistant Redesign

Redesigned Onboarding Experience

Word Document Upload for Product Setup

Job Cancellation

Platform Improvements

Metric Optimization Lifecycle

Platform Improvements

Documentation and Examples

Import Langfuse Traces

SDK: Model Management

Platform Improvements

Metric Optimization

Galtea Assistant (Beta)

Form Improvements

Platform Improvements

Specifications View Redesign

Two-Factor Authentication

Documentation and Examples

Platform Improvements

Galtea CLI

Product Tags

Smarter Pre-Evaluation Guidance

Structured Inputs for AI-Generated Tests

Documentation and Examples

Platform Improvements

Metric Versioning with Evaluation Replay

Galtea Agent Skill

Run Evaluations from Specifications

Language Variables in Input Templates

Reliable Trace-Dependent Evaluations

Export Evaluation Results as CSV

Inference Result Error Tracking

One-Click Metric Duplication

Platform Improvements

Human Annotation for All Evaluations

SDK: Endpoint Connection Management

SDK: Job-Based `wait_for`

SDK: Langfuse `CallbackHandler`

Structured JSON Inputs

Langfuse Integration

Public REST API Reference

Platform Improvements

Zero-Code Trace Ingestion via OpenTelemetry

Structured JSON Inputs

SDK: Synchronous Evaluation Polling

Normalized JSON Field Match Metric

Editable Test Metadata

Platform Improvements

Specifications in Analytics

Test Case CSV Export

Documentation and Video Guides

Platform Improvements

Improved Golden Dataset Generation

Platform Improvements

AI-Powered Metric Creation

Data Catalogs for Behavior Tests

Simplified Test Creation

Test Your Endpoint Connection

Run Evaluations from Any Level

SDK: Full Conversation Access in Custom Metrics

Platform Improvements