Skip to main content
You are reading the docs of Galtea 5.16. Read this page in the latest version.

What is a User Group?

A User Group organizes users, metrics, and products within an organization to streamline human evaluation workflows. User Groups serve three key purposes:
  1. Assigning evaluators: Group users who should review and annotate AI outputs for specific metrics.
  2. Scoping metrics: Link metrics to groups so that only users in those groups see pending evaluations for those metrics.
  3. Linking products: Link products to a group, or mark a group as linked to all products of the organization, current and future. A MEMBER, READER or HUMAN_EVALUATOR reaches the union of the products that all its groups link. When one of its groups links all products, it reaches every product, including a product that no other group links. A MEMBER, READER or HUMAN_EVALUATOR in no group sees no product. A MEMBER creates and edits only inside those products, and creates a product only when one of its groups links all products and its list allows product.create (a group that follows Editor does). Unless one of its groups links all products and allows product.read, a READER or HUMAN_EVALUATOR sees only the metrics that its products evaluate, link to a specification, or monitor. A MEMBER sees every metric of the organization when its lists allow metric.create or metric.update, or when one of its groups links all products and allows product.read. It edits them when its lists allow metric.update (see Permission lists). OWNER and ADMIN users see every product and every metric of the organization, whatever their groups.
A Human Evaluation metric is routed strictly: only users in a group linked to that metric see its pending evaluations in the Human Evaluations page of the dashboard. A MEMBER, READER or HUMAN_EVALUATOR also sees only the evaluations of the products their groups grant. A metric with no linked group reaches nobody, and its metric page shows a warning. A new organization gets an empty “all products” group. Organizations created before this feature also got this group, filled with their READER and HUMAN_EVALUATOR users. Every existing MEMBER was added to an all-products group, so members kept the products they had. A MEMBER in no group reaches no product until an OWNER or ADMIN adds a group. A member who signs up through SSO, or is invited with no group, starts that way. More than one group can be linked to all products.

Permission lists

An OWNER or ADMIN can give a group a permission list: the actions its users may do on the products the group links, and on metrics, models and comments across the organization. A permission is one action on one entity, written entity.action, for example product.read or analytics.readCoverage. A group with a list allows only the actions on it. Permission list ids lists every id a list may hold.
  • The role caps the list. A MEMBER, READER or HUMAN_EVALUATOR may do an action on a product when their role allows it and at least one of their groups that reaches the product lists it. A list entry above the role does nothing. A list never limits an OWNER or ADMIN. For example, a READER never writes, whatever a list says.
  • Groups add up. A user in two groups on one product may do an action when either group lists it. A list never removes an action that another group allows.
  • Editor means every action. A group that follows the Editor preset allows every action its members’ role allows, including actions added later. An empty list grants nothing. A user in no group reaches no product, and product lists answer 200 with an empty page.
  • Metrics, models and comments are shared across products. A metric, model or comment entry is not limited to the products the group links: metric.update edits any metric of the organization. What See shows can still follow the products, as What See shows says. A group with no products can hold these entries, for example a Metric authors group with metric.read, metric.create and metric.update. A user in no group gets no metric, model or comment action. comment.delete deletes only the user’s own comments. Creating a revision of a metric needs both metric.create and metric.update.
  • A list needs reads. A list that holds an entry on a product entity holds product.read, and an action on an entity needs the read of that entity. Some actions also need a read of another entity: an action that uses metrics (evaluations, runs, specifications, monitors, analytics) needs metric.read, a version action needs model.read, comment.create and evaluation.readInference need inferenceResult.read, and reading traces, sessions or inference results needs comment.read. The API answers 400 otherwise, and names the missing read. It also answers 400 for an entry that no list can grant, such as userGroup.update.
  • An all-products group that follows Editor still applies. A member who is also in one keeps every action their role allows. A narrower group binds that user only after an OWNER removes them from the all-products group, or gives it another preset or a list of its own.
  • A list without a read refuses some lists. When every group of a user leaves out test.read, GET /tests answers 403. The same holds for monitor.read and GET /monitors, metric.read and GET /metrics, and model.read and GET /models, so a user in no group gets 403 from GET /metrics and GET /models, and an empty page from GET /tests and GET /monitors. Other lists, such as GET /sessions and GET /runs, answer 200 with an empty page.
  • Some actions stay with the role. Only actions on entities that belong to a product, and on metrics, models and comments, come from lists. Every other action stays with the role. Datasets with no product (Galtea-wide datasets) stay readable by id whatever the list says. GET /tests lists one only when ids or testCaseIds names it. product.softDelete in a list means the user’s own products only. Creating a product needs an all-products group whose list allows product.create.
  • Analytics follows analytics.read. Without analytics.read on a product, a route for that one product answers 403. The dashboard hides the product’s Analytics section and opens the first section the user may see. The two routes for many products keep the product: GET /analytics/summary answers an empty row for it, and GET /analytics/products answers its row with no stats. The summary boxes of the organization products page count only the products with analytics.read, and answer 403 when the request names another. Without analytics.readCoverage, the dashboard hides the Coverage section.
  • The insights narrative summarizes evaluation reasons. It is written from the reasons the judge gave in evaluations. A group with analytics.read but without evaluation.read can still read text drawn from those reasons.

What See shows

For example, a Support group that links one product and lists metric.read sees only the metrics that product uses. The dashboard’s Reaches column shows the same answer for the boxes ticked now. A Business group is a typical example. List product.read, version.read, specification.read, analytics.read, analytics.readCoverage, metric.read and model.read, and link the products. Its users see the analytics views of those products, including the version filter and the insights, which read the versions and specifications. They see no tests, sessions or evaluations: the dataset and monitor lists answer 403, and the other lists are empty. A list change takes effect on the next API request. The dashboard shows it when the user returns to the tab or reloads. The API decides: a dashboard control can still show an action that the API then refuses with 403. In a frozen organization, a custom list never grants a write: the freeze keeps users read-only.

Presets

A preset is a permission list that Galtea maintains: Editor, Human evaluator or Reader. Reader holds every read a READER has, and Human evaluator adds to Reader what a HUMAN_EVALUATOR may do on evaluations. Editor is the default, and allows every action the role allows. A group either follows a preset or holds its own list:
  • Follows a preset. The group stores only the preset. When Galtea changes what the preset grants, for example to add the permissions of a new feature, the group changes with it. Its owner never updates it by hand. The dashboard names it, for example “Reader permissions”.
  • Holds its own list. The group stores its permissions, and a change to a preset changes nothing. The dashboard shows its counts, for example “Reads 5, Writes 2, Deletes 1”.
In the dashboard, an OWNER or ADMIN sets the permissions in the group’s create and edit forms, or in its Permissions tab. They pick one of four cards: Editor, Human evaluator and Reader make the group follow that preset, marked Updates automatically. Custom gives the group its own list, marked You update it: new permissions reach it only when someone ticks them. Customize under a preset, or the Custom card, starts the list from the permissions ticked now. The Permissions tab shows a grid, and in the forms Show permissions opens it; picking Custom opens it too. On a preset the grid is read only, and on Custom an OWNER or ADMIN edits it. The grid has one row per entity, with See, Create, Edit and Delete columns and the other actions under Other. Metrics, models and comments sit under Shared across products, and its Reaches column says what each See shows with the boxes ticked now. The i button next to each action under Other says what it does. Ticking an action also ticks the reads it needs, and a read stays ticked while an action needs it. In the tab, the permissions are read only until an OWNER or ADMIN picks Edit permissions. Then Save stores the change and Cancel drops it, and both end the edit. Other users see no Edit permissions button. Through the API, an OWNER or ADMIN sets the permissions when POST /userGroups creates the group, or later with PATCH /userGroups/{id} or user_groups.update:
  • permissionPreset set to EDITOR, HUMAN_EVALUATOR or READER makes the group follow that preset.
  • permissionPreset: null with a permissions list makes the group hold that list.
  • A permissions list alone also makes the group hold it. permissions: null, or no field on POST, makes the group follow Editor. Leaving both out of PATCH keeps the permissions.
  • A preset together with a list, or permissionPreset: null with no list, answers 400.
On POST, productIds links products to the new group. A list that breaks a rule, or a product that does not exist or is in another organization, answers 400 and creates no group. Both endpoints answer 400 for a body key they do not know. GET /userGroups/grantablePermissions lists every permission a list may hold, with a short description of each and the reads it requires, and the presets, each with its description and its permissions today. The default preset has isDefault: true.

Managing User Groups

Only an OWNER or ADMIN may create, edit, or delete a user group, or change its users or products. A MEMBER can read every group of the organization. A MEMBER can also link or unlink a group of the organization to a metric that the MEMBER may edit. User groups can be managed in two places:
  • Dashboard: Navigate to an organization’s Groups tab to create, edit, and delete groups. The create and edit forms set the group’s products, or All products. In a group’s Users, Metrics and Products tabs, an OWNER or ADMIN links entries and unlinks the selected ones. In the Products tab, the row menu also unlinks one product. Unlinking never deletes a user, a metric or a product. Creating a user, a metric or a product from one of these tabs links it to the group: the new user joins the group, the new product is linked to it, and the metric form selects the group. A blank metric form starts as Human Evaluation. Only a Human Evaluation metric keeps the group. This needs permission to edit the group, and a new metric or product is linked only to a group of your own organization. The Metrics tab links only Human Evaluation metrics. The platform refuses to unlink a metric that has evaluations in the group’s organization, or a Human Evaluation metric from its last group. In the Products tab, while All products is on, the tab shows only a note. The group keeps its stored links, and they apply again when All products is turned off. The edit form shows them. In the create form, turning on All products clears the products that you picked. From the user detail page, an OWNER or ADMIN can assign and unassign groups. From the metric detail page, any user who may edit the metric can assign and unassign groups.
    • Invite form: When an OWNER or ADMIN invites a MEMBER, the form asks for their groups. Each group shows its products, users and permissions, and a click on one opens a dialog with the details. A summary beside the form shows the permissions the user gets on each product. When the organization has no group yet, a dialog after the save of a MEMBER offers to create the first one. The new group holds the user, unless the invited address already had an account.
    • Invite an OWNER: The form hides the groups for an OWNER, who reaches every product. An OWNER still needs a group to evaluate a Human Evaluation metric. So an OWNER invited from a group’s Users tab joins that group.
    • Roles on the form: The role list offers OWNER and MEMBER, plus the staff roles to an ADMIN. It does not offer READER or HUMAN_EVALUATOR, and Galtea refuses both when it invites a user or changes a role. Give MEMBER and a group with the Reader or Human evaluator preset instead.
    • Edit user page: The edit user page has the same Groups section. When a role change makes a user a MEMBER while the organization has no group, the same dialog opens after the save. A user who already holds READER or HUMAN_EVALUATOR keeps that role on the page.
    • Users tab: The organization’s Users tab shows each user’s role and their groups with their permissions. A click on a group opens its products, users and permissions. It flags a MEMBER, READER or HUMAN_EVALUATOR with no group.
  • SDK: Use the galtea.user_groups service for programmatic management. See the Human Evaluation tutorial for a step-by-step guide.

SDK Integration

User Group SDK API

Create, list, update, and delete user groups programmatically. Link and unlink users, metrics, and products.

User Group Properties

string
Unique identifier for the user group.
string
The organization this group belongs to.
string
required
Name of the user group.
string
required
Type of user group. Currently only EVALUATOR is supported.
string
Description of the group’s purpose.
list[string]
IDs of users in this group.
list[string]
IDs of metrics linked to this group. A MEMBER, READER or HUMAN_EVALUATOR gets only the IDs of the metrics it can see. A MEMBER sees all of them when its lists allow metric.create or metric.update, or when one of its groups links all products and allows product.read.
list[string]
IDs of products linked to this group. The products a group links are the only products its MEMBER, READER and HUMAN_EVALUATOR users can reach. A MEMBER, READER or HUMAN_EVALUATOR gets only the IDs of the products it can see, in every group.
boolean
When true, the group is linked to every product of the organization, current and future, regardless of product_ids. A MEMBER, READER or HUMAN_EVALUATOR in the group then reaches every product.
string | None
The preset the group follows: EDITOR, HUMAN_EVALUATOR or READER. None means the group holds its own list.
list[string] | None
What the group grants, sorted: the list of the preset it follows, or its own list. None means the group follows Editor and allows every action the role of its members allows. An empty list grants nothing.

Concepts overview

How Galtea’s concepts connect — diagram + per-entity quick reference.

Human Evaluation Tutorial

Step-by-step guide to the full annotation workflow with screenshots.

User Group SDK API

Programmatic management of user groups.