​
What are Scenarios?

Scenarios in Galtea are designed to evaluate the conversational capabilities of your product through multi-turn dialogue interactions. These tests use the Conversation Simulator to create realistic conversations with synthetic users that have specific goals, personalities, and behaviors. Unlike Quality Tests that focus on single-turn question-answer pairs, or Red Teaming Tests that test security vulnerabilities, Scenario based tests evaluate how well your AI product can:
  • Maintain context across multiple conversation turns
  • Guide users toward successful task completion
  • Handle unexpected but realistic user inputs
  • Stay in character and follow conversation guidelines
  • Manage complex dialogue flows

​
Creating Scenarios

You can create scenarios in Galtea through two methods:
1

Define Your Product

Start by defining your product in Galtea. These detailed product properties will be used as the foundation for generating realistic scenarios.
2

Configure the test

Select User Scenarios as the test type and Generated as the test origin
The test creation process can be done via the SDK or the Galtea dashboard
3

Generate Scenarios

Galtea will automatically analyze your product definition to create tailored conversation scenarios, complete with:
  • Realistic user personas based on your target audience
  • Goals aligned with your product’s use cases
  • Natural conversation flows that test key features

​
Conversation Flow Categories

Scenarios can cover various types of conversational interactions:

​
Example Scenarios and File Format

Here are examples of scenario content and structure:
This structure is required by Galtea to automatically generate the test cases for your scenarios as tests within the platform. If this format is not provided, you can still manually create the test cases.
For automatic processing, the file format must be CSV.
The examples provided above are simplified demonstrations. In actual CSV files, scenarios can be much more detailed and the number of test cases (rows) can be significantly higher to provide comprehensive conversation testing coverage.

​
Structure of Scenarios

Scenarios have a specific structure designed to enable realistic conversation simulation:
​
goal
Text
required
The overall objective the synthetic user is trying to achieve during the conversation. Example: “Book a one-way flight from San Francisco (SFO) to New York (JFK) for next Tuesday”
​
user_persona
Text
required
The personality and communication style of the synthetic user that will be simulated throughout the conversation. Example: “A busy professional who is direct and values efficiency. They prefer to get things done quickly without much small talk.”
​
initial_prompt
Text
The first message the synthetic user sends to start the conversation. If not provided, the system will generate an appropriate opening based on the goal and persona. Example: “Hello, I need to book a flight.”
​
stopping_criterias
Text
A delimited string of conditions that, if met, will cause the conversation simulation to end. Use ; or | as delimiters to separate multiple criteria. Example: “The user has confirmed the flight booking|The chatbot indicates it cannot fulfill the request;User expresses satisfaction with the service”
​
max_iterations
Number
The maximum number of conversation turns before the simulation automatically ends. This prevents infinite loops and controls test duration. Example: 10
​
scenario
Text
A brief description of the scenario for documentation and organizational purposes. This helps categorize and manage different types of conversation tests. Example: “Flight booking scenario for business travelers”

​
Using Scenarios with the Conversation Simulator

Scenarios are specifically designed to work with Galtea’s Conversation Simulator. When you run evaluations using scenarios, the system will:
  1. Initialize the conversation using the initial_prompt and user_persona
  2. Generate realistic user responses based on the goal and conversation history
  3. Continue the dialogue until one of the stopping_criterias is met or max_iterations is reached
  4. Evaluate the conversation using your selected metrics to assess performance
For detailed implementation examples, see the Conversation Simulator Tutorial.

​
Best Practices for Manually Creating Scenarios