Roles
- OWNER: Manages users and resources in their own organization. Can permanently delete only Sessions and User Groups; every other entity is soft delete at most. Only OWNER may create, edit, or delete a User Group, or change its users or products.
- MEMBER: Day-to-day access in their organization: view and contribute, with limited edits. Can permanently delete only Sessions; every other entity is soft delete at most. Reads every User Group of the organization. Cannot create, edit, or delete a User Group, or change its users or products. May link or unlink a group of the organization to a metric it may edit.
- HUMAN_EVALUATOR: Read-only on the products their user groups grant, plus may annotate AI metrics and submit human evaluations on them.
- READER: Read-only on the products their user groups grant. Cannot create, update, or delete anything.
Each cell shows what a role can do for that action. ✅ means allowed and ❌ means not allowed. A short rule (for example, “If product in own org”) means the action is allowed only when that rule is true at the time, checked against your organization and the record’s ownership.
Permissions by area
Identity & Organization
User
Organization
An OWNER may update only the credit warning threshold, MFA enforcement, and the SAML SSO settings. An OWNER cannot change the organization name.
Subscription
User Group
Test Assets
Product
The Tutorial and Configuration entry rows only control what the dashboard shows for this product (the sidebar entry, its page and its list URL): the API still returns this data to every role that can read the product.
Version
The Versions entry row only controls what the dashboard shows for this product (the sidebar entry, its page and its list URL): the API still returns versions to every role that can read them.
Dataset
The Datasets entry row only controls what the dashboard shows for this product (the sidebar entry, its page and its list URL): the API still returns datasets to every role that can read them.
Test Case
Specification
Connections
Endpoint Connection
Phone Connection
WebRTC Connection
Evaluation & Results
Session
Evaluation
Human evaluation also requires membership in a User Group linked to the metric. A HUMAN_EVALUATION metric with no linked group reaches nobody. Annotating an AI metric has no group check. One gate covers the whole annotation, so anyone who may annotate may also edit or remove an annotation another person wrote. The table shows the role-level gate only. The evaluation metadata and inference metrics rows only control what the dashboard shows on the evaluation page: the API still returns this data to every role that can read the evaluation, which for a READER or HUMAN_EVALUATOR means only on the products their user groups grant. The inference metrics row also covers the Turn metrics tab of the Trace card. The Trace card’s Evaluations tab still shows judge tokens, and its Spans tab shows span latency.
Trace
Span
Comment
Free-form reviewer text on a trace. Access also requires the parent trace to be in your organization. Deleting is a soft delete; you may delete your own comments, and organization owners may delete any.
Monitor
Run
Reference Data
Metric
Model
Analytics & Insights
Analytics
The Coverage, Scores comparison, and Inference rows only control what the dashboard shows on the Analytics page: the API still returns this data to every role that can read Analytics, and to a READER or HUMAN_EVALUATOR only for the products their user groups grant.
Dashboard
Dashboard views
These rows decide what the dashboard shows or hides for a role. They do not grant access: the platform still checks the permission behind each action.
Notes
- Read-only roles: READER and HUMAN_EVALUATOR read what a MEMBER reads, except that products and everything inside a product are limited to the products granted by their user groups; the dashboard also hides three Analytics sections, and the Datasets, Versions, Tutorial and Configuration entries of the product sidebar, from them. They cannot create, update, or delete. HUMAN_EVALUATOR additionally may annotate AI metrics and submit human evaluations, on those products only; READER cannot. On an evaluation page, the dashboard also hides the evaluation metadata, the inference metrics and the Turn metrics tab of the Trace card from READER.
- Product access: A READER or HUMAN_EVALUATOR sees a product only when one of their user groups links it. A user group with the all-products setting grants every product of the organization, current and future. A user in no group sees no product. Records with no product, such as a global dataset, stay readable. Products outside the granted set answer like a record of another organization. In a user group, their
product_idslist only the granted products. OWNER and MEMBER are not limited this way. - Organization scope: “In own org” means the record (or its parent, such as a Product) has an organizationId equal to the signed-in user’s organization. This keeps each client organization’s data separate.
- Parent scope: For nested records (Version, Dataset, Evaluation, and so on) access is derived from the parent’s organization. Example: a Version belongs to a Product, and the Product belongs to an Organization. An Evaluation is reached through session to version to product.
- Global records: Some records (Metric, Model, and a Dataset with no product) can be global, meaning not tied to one organization. Global records are readable by every role in addition to their own organization’s records.
- Hard delete: Each entity table has two delete rows. Soft delete hides a record without removing it. Delete is a permanent removal, shown as allowed for Session (OWNER and MEMBER, within their own organization) and for User Group (OWNER only, within their own organization); for every other entity it is denied to all roles listed here.