> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GitHub Actions

> Learn how to integrate Galtea's evaluation capabilities into your GitHub Actions workflow

> [!WARNING]
>
> You are reading the docs of Galtea 5.17. [Read this page in the latest version](https://docs.galtea.ai/sdk/integrations/github-actions.md).

## Initial Setup

> [!NOTE]
>
> To initialize the `Galtea` class, you need to provide your API key obtained in the [settings page](https://platform.galtea.ai/settings) of the Galtea platform.

Before you can use GitHub Actions with Galtea, you need to perform some credentials and variables configuration:

* **Configure Repository Secrets and Variables**
  * Go to your repository's "Settings" tab
  * Navigate to "Secrets and variables" > "Actions"
  * Add the following:
    * Secret: `GALTEA_API_KEY` - Your Galtea API key
    * Variable: `GALTEA_PRODUCT_ID` - Your Galtea Product ID
  * Click "Add" after each entry

## Dependencies

Create a `requirements.txt` file in your repository and add the dependencies required for your project. At minimum, you'll need galtea library:

```txt requirements.txt theme={"system"}
galtea
```

## Create your GitHub Action

Create a `.github/workflows/evaluate.yml` file in your repository with the following content:

```yml .github/workflows/evaluate.yml theme={"system"}
name: Galtea Evaluation

on:
  push:
  pull_request:
  workflow_dispatch:

jobs:
  evaluate:
    env:
      GALTEA_API_KEY: ${{ secrets.GALTEA_API_KEY }}
      GALTEA_PRODUCT_ID: ${{ vars.GALTEA_PRODUCT_ID }}
      AGENT_MODEL: gpt-4o-mini
      AGENT_TEMPERATURE: "0.7"
    runs-on: ubuntu-latest
    steps:
     - name: Checkout
       uses: actions/checkout@v4

     - name: Setup Python
       uses: actions/setup-python@v5
       with:
         python-version: '3.10'

     - name: Install dependencies
       run: |
        pip install -r requirements.txt

     - name: Run Evaluation
       run: |
        python evaluate.py
```

## Create your Test Script

Create an `evaluate.py` file in your repository with the following content.

You give `galtea.evaluations.run(version_id=..., agent=my_agent)` one thing: a Python function that calls your agent. From there it does the whole evaluation in one call. It finds the [Specifications](https://docs.galtea.ai/v5.17/concepts/product/specification.md) of your product, resolves their linked [Datasets](https://docs.galtea.ai/v5.17/concepts/product/dataset.md) and [Metrics](https://docs.galtea.ai/v5.17/concepts/metric.md), calls `my_agent` once per [Test Case](https://docs.galtea.ai/v5.17/concepts/product/dataset/test-case.md) inside this CI job, and submits the answers for scoring. Your agent does not need to be deployed anywhere, so this also works as a pre-deploy gate.

```python theme={"system"}
import os

from galtea import Galtea

# GALTEA_API_KEY and GALTEA_PRODUCT_ID come from the workflow's `env:` block above.
# The GITHUB_* ones are provided by GitHub Actions itself, so you never declare them.
galtea = Galtea(api_key=os.environ["GALTEA_API_KEY"])

PRODUCT_ID = os.environ["GALTEA_PRODUCT_ID"]

# The settings your agent runs with. The workflow sets them in its `env:` block, and the
# defaults apply when you run this script on your own machine.
AGENT_MODEL = os.environ.get("AGENT_MODEL", "gpt-4o-mini")
AGENT_TEMPERATURE = os.environ.get("AGENT_TEMPERATURE", "0.7")

# One version per agent state. The facts come from the same settings the agent uses, so they
# always describe what was tested. `auto_detect_commit_hash` adds the commit: GITHUB_SHA in CI,
# `git rev-parse HEAD` on your machine. A re-run of the same state gets the same version back.
version = galtea.versions.get_or_create(
    product_id=PRODUCT_ID,
    facts={"model": AGENT_MODEL, "temperature": AGENT_TEMPERATURE},
    auto_detect_commit_hash=True,
)


# Your product under test. Galtea calls it once per test case.
def my_agent(user_message: str) -> str:
    # In a real scenario, this would call your model with AGENT_MODEL and AGENT_TEMPERATURE
    return "This is a placeholder model answer."


# Open a run named after the workflow run, so every result of this CI job is grouped under one
# entry you can find again from the build number. The label must be unique per product, and
# "Re-run all jobs" keeps the run id and only bumps the attempt, so both go in the label.
# Leaving the block closes the run.
with galtea.runs.start(
    product_id=PRODUCT_ID,
    version_id=version.id,
    custom_id=f"{os.environ['GITHUB_RUN_ID']}-{os.environ['GITHUB_RUN_ATTEMPT']}",
):
    # One call runs the whole evaluation: it finds the product's specifications, resolves their
    # linked datasets and metrics, runs the agent on every test case, and submits the results.
    # It joins the open run, so you pass no run id here.
    result = galtea.evaluations.run(version_id=version.id, agent=my_agent)

print(f"Evaluated {result['testCaseCount']} test cases against version {version.name}")
print(f"Results are grouped under run {result['runId']}")
```

> [!NOTE]
>
> `versions.get_or_create` gives each agent state one [Version](https://docs.galtea.ai/v5.17/concepts/product/version.md). The script reads the facts from the same environment variables your agent uses, so a change to `AGENT_MODEL` in the workflow gives a new version with no other edit. With `auto_detect_commit_hash=True`, the SDK also adds the commit from `GITHUB_SHA`. So each commit gets its own version, and a re-run of the same commit reuses it. See [Get or Create Version](https://docs.galtea.ai/v5.17/sdk/api/version/get-or-create.md).

> [!NOTE]
>
> `run()` discovers work through specifications, so each one needs both its metrics and a dataset linked. A specification missing either is skipped silently, and the run evaluates nothing. That setup is done once (see [Writing Specifications](https://docs.galtea.ai/v5.17/sdk/tutorials/writing-specifications.md)), not on every CI run. If you need per-test-case control instead, see [Run Dataset-Based Evaluations](https://docs.galtea.ai/v5.17/sdk/tutorials/run-dataset-based-evaluations.md).

> [!NOTE]
>
> The `with galtea.runs.start(...)` block groups everything the job evaluates into one [Run](https://docs.galtea.ai/v5.17/concepts/product/run.md), labelled with the GitHub run id and attempt. Find it again later with `galtea.runs.get_by_custom_id(product_id=..., custom_id=...)`. See the [Runs Service](https://docs.galtea.ai/v5.17/sdk/api/run/service.md).
>
> The label must be unique per product among runs that still exist. "Re-run all jobs" reuses `GITHUB_RUN_ID` and only bumps `GITHUB_RUN_ATTEMPT`, so the label carries both; a label already taken makes `runs.start()` fail.

> **Success!** 🎉 Your GitHub Actions workflow is now configured to run evaluations with Galtea. Each time you push changes, it will automatically evaluate your product using the latest version of your code.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.