Skip to main content

​
What is a Specification?

A Specification in Galtea represents a single, testable behavioral expectation for a product. Examples include “Must decline political questions”, “Can answer questions about phone specifications”, or “Always includes a disclaimer when giving financial advice”. Each specification has a type that classifies the kind of behavioral expectation:
  • Capability: A core function the product can perform — what the product is designed to accomplish or deliver. For example: “Can classify images as containing cats or dogs”, “Can retrieve and display the user’s account balance” (conversational), or “Can generate personalized product recommendations based on browsing history”.
  • Inability: An action or task the product is fundamentally unable to do even if a malicious actor gained full control over it. These represent hard technical constraints where the infrastructure, system access, or architecture simply does not exist. For example: “Cannot execute wire transfers” (no payment gateway connected) or “Cannot access other customers’ data” (no system integration exists).
  • Policy: A mandatory rule or guideline the product must follow — from restrictions to behavioral guidelines. This includes refusal rules (what it must decline), interaction patterns (how it responds), communication style, and mandatory disclaimers. For example: “Must include a disclaimer when discussing medical topics” or “Refuses requests to share confidential business data”.
Specifications also have a test type that determines the evaluation approach: ACCURACY, SECURITY, or BEHAVIOR.
Specifications replace the legacy free-text fields (Capabilities, Inabilities, Policies) on Products with structured, individually testable expectations linked to specific metrics.

​
The Specification-Driven Workflow

Specifications are at the center of Galtea’s recommended evaluation flow:
  1. Define specifications for your product (Capability, Inability, Policy)
  2. Generate metrics from specifications using AI Metric Generation
  3. Create tests from specifications — the test type is auto-derived
  4. Run evaluations with evaluations.run() — resolves specs, tests, and metrics automatically
When creating or editing a specification in the dashboard, you have two AI assistance options. Fill with AI opens a dialog where you describe the behavior in a rough note and the AI rewrites it into a properly written specification description, then auto-fills the type, test type, and variant. Complete with AI classifies a description you have already written, suggesting the type, test type, and variant without changing the text. Review and edit the AI suggestions before saving — nothing is persisted until you submit the form.

Specification-Driven Evaluations Tutorial

End-to-end guide: define specs, generate metrics, create tests, and run evaluations — all from specifications.

​
Keeping Specifications Up to Date

As your product evolves, its documentation — PRDs, design docs, spec sheets — often changes before its specifications do. The POST /products/{id}/regenerate-config endpoint lets you cross-check an uploaded document against a product’s existing specifications and receive classified suggestions without persisting anything. Each suggestion carries a status:
  • new — a candidate specification extracted from the document that has no match among the product’s existing specifications.
  • updated — a candidate that refines or extends an existing specification (includes a reference to the existing spec).
  • contradictory — a candidate that conflicts with an existing specification (includes a reference to the existing spec).
The endpoint persists nothing. Applying a suggestion — creating a new specification, updating an existing one, or removing a contradicted one — is a separate, explicit step using the standard specification endpoints in the REST API. This means nothing changes without your confirmation and unaffected specifications remain untouched.
The dashboard flow for reviewing and applying suggestions is tracked in a follow-up. In the meantime the endpoint is available via the REST API and the CLI (galtea products regenerate-config).

​
SDK Integration

The SDK allows you to create, list, retrieve, and delete specifications, as well as link and unlink metrics. See the Specification Service API documentation for more details.

Specification Service

Manage specifications programmatically

​
Specification Properties

​
id
string
Unique identifier of the specification.
​
product_id
string
required
The ID of the product this specification belongs to.
​
description
string
required
A description of the testable behavioral expectation. Example: “Must decline political questions and redirect the user to authoritative sources.”
​
type
Enum
required
The type of specification. Possible values: CAPABILITY, INABILITY, POLICY.
​
test_type
Enum
The type of test for this specification. Possible values: ACCURACY, SECURITY, BEHAVIOR.
Required for POLICY specifications. Must not be provided for CAPABILITY or INABILITY specifications.
​
test_variant
string
Variant of the test type. Applicable for ACCURACY and SECURITY test types.
​
metric_ids
list[string]
List of metric IDs linked to this specification.
​
created_at
string
Timestamp of when the specification was created (ISO 8601 format).

Concepts overview

How Galtea’s concepts connect — diagram + per-entity quick reference.

Product

A functionality or service being evaluated

Metric

Ways to evaluate and score product performance