> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

> Validate your AI product with Galtea

> [!WARNING]
>
> You are reading the docs of Galtea 5.14. [Read this page in the latest version](https://docs.galtea.ai/quickstart.md).

This quickstart walks you through running your first evaluation in Galtea for a conversational product using the Python SDK. It is intended for a technical audience and covers RAG pipelines, conversational scenarios, and security or safety checks.

[Watch how Galtea works](https://www.youtube.com/embed/i4RsrElikc8?rel=0)

## Evaluation Workflow Overview

**1. Tell Us About Your Product**

Provide details about your product so our models can create tailored testing content.

**2. Install SDK & Connect**

Set up the Galtea Python SDK to interact with the platform.

**3. Version Your Product**

Track different implementations to compare improvements over time.

**4. Create Your Datasets**

Create a Dataset to generate the test cases you’ll evaluate.

**5. Choose Your Metrics**

Select from our metrics library or bring your own custom metrics.

**6. Run Evaluations**

Test your product version with the selected dataset and metrics.

**7. See Your Results**

Explore detailed insights and compare versions on the dashboard.

## 1. Tell Us About Your Product

If you’re new to Galtea, start by creating an account and registering your first product in the dashboard: [Go to the dashboard](https://platform.galtea.ai/).

Follow the **Onboarding** checklist on the product page (this is the guided setup shown inside your product). It will prompt you for product details and help you generate an API key. The product details you provide there are used to tailor generated datasets and evaluations.

![AI-assisted product registration form](https://mintcdn.com/galtea/dTqmcvyz2pQyIXAj/images/ai-product-form.png?fit=max\&auto=format\&n=dTqmcvyz2pQyIXAj\&q=85\&s=67fdf9589632852db21e3fe8d703cfd6)

Keep your **Product ID** and **API key** handy for the SDK steps below.

> [!NOTE]
>
> The product information is crucial – it powers our ability to generate synthetic test data that’s specific to your use case.

## 2. Get Started with the SDK

You can use Galtea from the dashboard or via the Python SDK. This quickstart uses the Python SDK.

**1. Get your API key**

In the [Galtea dashboard](https://platform.galtea.ai/), copy the API key you created during your product page onboarding (or navigate to **[Settings](https://platform.galtea.ai/settings) > Generate API Key**).

**2. Install the SDK**

```bash theme={"system"}
pip install galtea
```

**3. Connect to the platform**

```python theme={"system"}
from galtea import Galtea

# Verify the SDK is properly installed by initializing it
# Note: Replace "YOUR_API_KEY" with your actual API key from the Galtea dashboard
galtea = Galtea(api_key="YOUR_API_KEY")
print("Galtea SDK installed successfully!")
```

## 3. Version Your Product

Use versions to track iterations of your product and compare evaluation results over time. Each evaluation is tied to a version, so you can see how changes impact metrics.

A version stands for one state of your agent, described by its [facts](https://docs.galtea.ai/v5.14/concepts/product/version.md#version-facts), for example the model and the prompt revision. You do not need to create it first or copy its ID. `versions.get_or_create` returns the version that holds your facts, and creates it the first time.

To find a product ID in the dashboard, open your product and click the **3-dot menu** on any item, then select **Copy ID**.

<video autoPlay muted loop playsInline src="https://mintcdn.com/galtea/dTqmcvyz2pQyIXAj/videos/copy-id-from-dashboard.mp4?fit=max&auto=format&n=dTqmcvyz2pQyIXAj&q=85&s=70b3eb4e55f30b64054ed389fd63d0bf" />

Or resolve your product by name in the SDK, then get the version of your agent state:

```python theme={"system"}
# Look your product up by name — no need to copy IDs from the dashboard.
product = galtea.products.get_by_name(name="Financial Assistant")

# Describe the agent state you test. Galtea returns its version, and creates it the first time.
version = galtea.versions.get_or_create(
    product_id=product.id,
    facts={"model": "gpt-4o-mini", "prompt_version": "1"},
)

product_id = product.id
version_id = version.id
```

## 4. Create Your Datasets

> [!TIP]
>
> **Prefer to generate datasets automatically?** If you've defined [Specifications](https://docs.galtea.ai/v5.14/concepts/product/specification.md) for your product, Galtea can generate datasets and metrics for you — see the [Specification-Driven Evaluations](https://docs.galtea.ai/v5.14/sdk/tutorials/specification-driven-evaluations.md) tutorial.

A single conversational product typically includes multiple components. In Galtea, you evaluate each component by creating a different [dataset](https://docs.galtea.ai/v5.14/concepts/product/dataset.md) type, think of each dataset as a lens on the same system:

* **RAG pipelines** → create an **Accuracy** dataset (`type="ACCURACY"`)
* **Security or safety aspects** → create a **Security & Safety** dataset (`type="SECURITY"`)
* **Conversational scenarios** → create a **Behavior** dataset (`type="BEHAVIOR"`)

You can create one or multiple dataset types for the same product, depending on which components you want to evaluate.

> [!NOTE]
>
> After creating a dataset, give it a moment to finish generating test cases. You can track progress in the dashboard.

> [!NOTE]
>
> If you already created a dataset during onboarding, you can reuse it here—skip the `datasets.create(...)` step and just list its test cases using the dataset ID from the dashboard.

> [tab] RAG
>
> Accuracy datasets are used to evaluate RAG pipelines and QA-style components within a conversational product. Galtea generates single-turn test cases (input and expected output) from a knowledge base to assess response quality and factual correctness.
>
> ```python theme={"system"}
> dataset = galtea.datasets.create(
>     name="rag-accuracy-test",
>     type="ACCURACY",
>     product_id=product_id,
>     ground_truth_file_path="knowledge.md",
>     language="english",
>     max_test_cases=20,
> )
> ```
>
> After a brief wait for test case generation, list the test cases:
>
> ```python theme={"system"}
> test_cases = galtea.test_cases.list(dataset_id=dataset.id)
> print(f"Using dataset '{dataset.name}' with {len(test_cases)} test cases.")
> ```
>
> Learn more about [Accuracy Datasets](https://docs.galtea.ai/v5.14/concepts/product/dataset/accuracy-datasets.md) and their configuration options.

> [tab] Security
>
> Security & Safety datasets probe safety and security boundaries (e.g., misuse, toxicity, data leakage). For this quickstart, we’ll create a small **Misuse** dataset.
>
> ```python theme={"system"}
> dataset = galtea.datasets.create(
>     name="misuse-security-test",
>     type="SECURITY",
>     product_id=product_id,
>     variants=["misuse"],
>     strategies=["original"],  # original must always be included
>     max_test_cases=20,
> )
> ```
>
> After a brief wait for test case generation, list the test cases:
>
> ```python theme={"system"}
> test_cases = galtea.test_cases.list(dataset_id=dataset.id)
> print(f"Using dataset '{dataset.name}' with {len(test_cases)} test cases.")
> ```
>
> Learn more about [Security & Safety Datasets](https://docs.galtea.ai/v5.14/concepts/product/dataset/security-datasets.md), [threats](https://docs.galtea.ai/v5.14/concepts/product/dataset/security-threats.md), and [strategies](https://docs.galtea.ai/v5.14/concepts/product/dataset/security-strategies.md).

> [tab] Conversational
>
> Behavior datasets evaluate multi-turn conversations. Each test case defines a goal/persona/scenario and is meant to be run with the [Conversation Simulator](https://docs.galtea.ai/v5.14/concepts/product/dataset/test-case/conversation-simulator.md).
>
> ```python theme={"system"}
> dataset = galtea.datasets.create(
>     name="conversation-behavior-test",
>     type="BEHAVIOR",
>     product_id=product_id,
>     language="english",
>     max_test_cases=20,
>     strategies=["written"],
> )
> ```
>
> After a brief wait for test case generation, list the test cases:
>
> ```python theme={"system"}
> test_cases = galtea.test_cases.list(dataset_id=dataset.id)
> print(f"Using dataset '{dataset.name}' with {len(test_cases)} test cases.")
> ```
>
> Learn more about [Behavior Datasets](https://docs.galtea.ai/v5.14/concepts/product/dataset/behavior-datasets.md) and [Conversation Simulation](https://docs.galtea.ai/v5.14/concepts/product/dataset/test-case/conversation-simulator.md).

## 5. Choose Your Metrics

Metrics define how evaluation outputs are scored. Galtea provides built-in metrics for common use cases (factual accuracy, faithfulness, toxicity, role adherence, and [many more](https://docs.galtea.ai/v5.14/concepts/metric.md)), and you can also create custom metrics with your own judge prompts. You can also [generate metrics from specifications](https://docs.galtea.ai/v5.14/concepts/metric/ai-generation.md) — the AI creates tailored judge prompts for each spec.

> [tab] RAG
>
> For a RAG-style Accuracy dataset, a good default is **[Factual Accuracy](https://docs.galtea.ai/v5.14/concepts/metric/factual-accuracy.md)**.
>
> ```python theme={"system"}
> metric = galtea.metrics.get_by_name(name="Factual Accuracy")
> ```

> [tab] Security
>
> For Misuse Security & Safety datasets, we’ll use **[Misuse Resilience](https://docs.galtea.ai/v5.14/concepts/metric/misuse-resilience.md)**.
>
> ```python theme={"system"}
> metric = galtea.metrics.get_by_name(name="Misuse Resilience")
> ```

> [tab] Conversational
>
> For conversational evaluations, start with a metric like **[Role Adherence](https://docs.galtea.ai/v5.14/concepts/metric/role-adherence.md)**.
>
> ```python theme={"system"}
> metric = galtea.metrics.get_by_name(name="Role Adherence")
> ```

## 6. Run the Evaluation

Now connect Galtea to your product. You define an **agent function** — a wrapper that takes a test case input, calls your AI system, and returns its output.

The SDK supports three agent function signatures — pick the one that matches how much context your agent needs:

> [tab] Simple
>
> The quickest way to get started. Your function receives just the latest user message as a string.
>
> ```python theme={"system"}
> def my_agent(user_message: str) -> str:
>     # In a real scenario, call your model here
>     return f"Your model output to: {user_message}"
> ```

> [tab] Chat History
>
> Use this when your agent needs the full conversation context. Your function receives the message list in the OpenAI format (`{"role": "...", "content": "..."}`).
>
> ```python theme={"system"}
> def my_agent(messages: list[dict]) -> str:
>     # messages follows the standard chat format:
>     # [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}, ...]
>     user_message = messages[-1]["content"]
>     return f"Your model output to: {user_message}"
> ```

> [tab] Structured
>
> For full control over input and output — including optional usage tracking, cost tracking, retrieval context for RAG evaluations, and answering with files through `output_file_paths` (not in a simulated conversation).
>
> ```python theme={"system"}
> def my_agent(input_data: AgentInput) -> AgentResponse:
>     user_message = input_data.last_user_message_str()
>
>     # Access structured input fields from the first user message's metadata
>     # (e.g. when test case input is {"user_message": "hello", "chat_type": "support"})
>     first_msg = input_data.messages[0] if input_data.messages else None
>     chat_type = first_msg.metadata.get("chat_type") if first_msg and first_msg.metadata else None
>
>     # In a real scenario, call your model here
>     model_output = f"Your model output to: {user_message}"
>     # Return AgentResponse with optional usage/cost tracking
>     return AgentResponse(
>         content=model_output,
>         usage_info={"input_tokens": 100, "output_tokens": 50},
>     )
> ```

> [!NOTE]
>
> All three signatures work with `evaluations.run()`, `traces.generate()`, and `simulator.simulate()`. Both sync and async functions are supported. The SDK auto-detects which signature you're using from the type hint on the first parameter.
>
> Only the **Structured** signature receives the files a test case attaches, as `input_data.input_files`. The other two receive the text alone and fail the trace for a file-only test case. See [Reading attached files](https://docs.galtea.ai/v5.14/sdk/api/agent/input.md#reading-attached-files).
>
> Only the **Structured** signature can answer with files, through `AgentResponse.output_file_paths`. A response carrying files but no text is a real answer, not an empty one. `simulator.simulate()` is the exception: it raises a `ValueError` on a response carrying files, because a simulated conversation carries none yet. See [Output Files](https://docs.galtea.ai/v5.14/concepts/product/version/session/trace.md#trace-properties).
>
> For the full list of fields available on `AgentInput` (including structured input access via message metadata), see the [AgentInput reference](https://docs.galtea.ai/v5.14/sdk/api/agent/input.md).

`galtea.evaluations.run()` does the rest in one call: it finds the [specifications](https://docs.galtea.ai/v5.14/concepts/product/specification.md) linked to your version, resolves their datasets and metrics, runs your agent on every test case, and submits the results for [scoring](https://docs.galtea.ai/v5.14/concepts/product/version/session/evaluation.md).

```python theme={"system"}
# One call resolves your specifications, their datasets and metrics, runs your agent
# on every test case, and submits the results for scoring.
result = galtea.evaluations.run(
    version_id=version_id,
    agent=my_agent,
)
print(f"Launched {result['testCaseCount']} test cases across {len(result['specifications'])} specifications")
```

> [!NOTE]
>
> Prefer to drive each test case yourself? The [Run Dataset-Based Evaluations](https://docs.galtea.ai/v5.14/sdk/tutorials/run-dataset-based-evaluations.md) tutorial shows the manual per-test-case loop for full control.

## 7. See Your Results

After launching, block until scoring finishes and read a summary straight from the SDK — a count of evaluations that passed their metric's threshold, plus any sessions that failed to run so you can triage them:

```python theme={"system"}
# Block until every evaluation leaves PENDING (defaults: up to 300s, polling every 5s).
completed = galtea.evaluations.wait_for(
    evaluation_ids=[e.id for e in result["evaluations"]],
)

# A readable pass/fail summary.
scored = [e for e in completed if e.passed is not None]
passed = [e for e in scored if e.passed]
print(f"{len(passed)}/{len(scored)} evaluations passed their metric's threshold")

# Triage: list sessions whose agent run failed and produced no output to score.
failed_sessions = galtea.sessions.list(version_id=version_id, status="FAILED")
if failed_sessions:
    print(f"{len(failed_sessions)} sessions failed to run — inspect them:")
    for session in failed_sessions:
        print(f"  session {session.id}")

print(f"View full results at: https://platform.galtea.ai/product/{product_id}")
```

For the full picture, open your product in the [Galtea platform](https://platform.galtea.ai/) and head to **Analytics**. The **Details** view shows the aggregate scores for a quick baseline; slice by **version**, **dataset**, and **metric** to see where performance shifts. Switch to **Summary** to see how the product's sessions go, whether failures are trending up or down, and which specifications fail most.

The most useful part is drilling into individual test cases: you can inspect the exact prompt/output pair (or the full conversation for scenarios) and understand which specific inputs move the score up or down. This makes it easier to identify failure patterns, validate fixes in new versions, and iterate with confidence as you move toward production.

## Next Steps

Congratulations! You’ve completed your first evaluation with Galtea.

You can now explore more advanced features like:

> [card] [Tracing Agent Operations](https://docs.galtea.ai/v5.14/sdk/tutorials/tracing-agent-operations.md)
>
> Learn how to capture and analyze the internal operations of your AI agent.

> [card] [Evaluating Production](https://docs.galtea.ai/v5.14/sdk/tutorials/monitor-production-responses-to-user-queries.md)
>
> Learn how to log and evaluate user queries from your production environment.

> [card] [Specification-Driven Evaluations](https://docs.galtea.ai/v5.14/sdk/tutorials/specification-driven-evaluations.md)
>
> Use specifications to automate dataset type selection and metric generation.

## Dive Deeper

Explore these concepts to tailor datasets, metrics, and workflows to your product. The [Concepts overview](https://docs.galtea.ai/v5.14/concepts/overview.md) shows how they all connect at a glance.

> [card] [Concepts overview](https://docs.galtea.ai/v5.14/concepts/overview.md)
>
> How Galtea's concepts connect — diagram + per-entity quick reference.

> [card] [Product](https://docs.galtea.ai/v5.14/concepts/product.md)
>
> A functionality or service being evaluated

> [card] [Version](https://docs.galtea.ai/v5.14/concepts/product/version.md)
>
> A specific iteration of a product

> [card] [Dataset](https://docs.galtea.ai/v5.14/concepts/product/dataset.md)
>
> A dataset of test cases for evaluating product performance

> [card] [Session](https://docs.galtea.ai/v5.14/concepts/product/version/session.md)
>
> A full conversation between a user and an AI system.

> [card] [Trace](https://docs.galtea.ai/v5.14/concepts/product/version/session/trace.md)
>
> A single turn in a conversation between a user and the AI.

> [card] [Evaluation](https://docs.galtea.ai/v5.14/concepts/product/version/session/evaluation.md)
>
> The assessment of an evaluation using a specific metric's criteria

> [card] [Metric](https://docs.galtea.ai/v5.14/concepts/metric.md)
>
> Ways to evaluate and score product performance

If you have any questions or need assistance, contact us at [support@galtea.ai](mailto:support@galtea.ai).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.