Roles
- OWNER: Manages users and resources in their own organization. Can permanently delete only Sessions and User Groups; every other entity is soft delete at most. Only OWNER may create, edit, or delete a User Group, or change its users or products.
- MEMBER: Day-to-day access in the products their user groups grant: view and contribute, with limited edits. Creates a product only with a user group that grants all products and allows
product.create. Can permanently delete only Sessions; every other entity is soft delete at most. Reads and edits metrics, models and comments across the organization as their user groups’ lists allow, and reads every User Group of the organization. Cannot create, edit, or delete a User Group, or change its users or products. May link or unlink a group of the organization to a metric it may edit.
Each cell shows what a role can do for that action. ✅ means allowed and ❌ means not allowed. A short rule (for example, “If product in own org”) means the action is allowed only when that rule is true at the time, checked against your organization and the record’s ownership.
Permissions by area
Identity & Organization
User
An Assign action applies only together with Create or Update: a role gives a role only to a user it may invite or edit. A MEMBER invites no one and edits only itself, so its Assign Member role has no effect.
Organization
An OWNER may update only the credit warning threshold, MFA enforcement, the SAML SSO settings, and the OpenTelemetry settings. An OWNER cannot change the organization name.
Subscription
User Group
Test Assets
Product
The Tutorial and Configuration entry rows only control what the dashboard shows for this product (the sidebar entry, its page and its list URL): the API still returns this data to every role that can read the product.
Version
The Versions entry row only controls what the dashboard shows for this product (the sidebar entry, its page and its list URL): the API still returns versions to every role that can read them.
Dataset
The Datasets entry row only controls what the dashboard shows for this product (the sidebar entry, its page and its list URL): the API still returns datasets to every role that can read them.
Test Case
Specification
Connections
Endpoint Connection
Phone Connection
WebRTC Connection
Evaluation & Results
Session
Evaluation
Human evaluation also requires membership in a User Group linked to the metric. A HUMAN_EVALUATION metric with no linked group reaches nobody. Annotating an AI metric has no group check. One gate covers the whole annotation, so anyone who may annotate may also edit or remove an annotation another person wrote. The table shows the role-level gate only. The evaluation metadata row only controls what the dashboard shows: the API still returns the metadata to every role that can read the evaluation, which for a MEMBER means only on the products their user groups grant. The inference metrics row is enforced by the API: a role without it gets null judge tokens and credits on evaluations, null tokens, latency, cost and credits on traces, a null aggregated latency and total cost, and null spent and pending monitor credits (the monthly cap stays visible), and null cost per session and total cost in the organization products table, from the dashboard, the SDK, the CLI and the assistant alike. The API also refuses the cost summary of the organization products page and of a product’s Analytics Summary, the cost per session trend and the cost breakdown of a product, and the latency, cost and tokens per version that a product’s Analytics Details reads (
GET /analytics/inference), to such a role with 403. The Analytics export leaves those values out for such a role and keeps the scores. Span latency (Spans tab, trace page) stays visible. GET /analytics still returns the latency, token and average cost summary and GET /analytics/insights/comparison states the average cost or latency of an outlier version in its flags, to every role that reads Analytics; the dashboard shows none of it.
Trace
Span
Comment
Free-form reviewer text on a trace. Access also requires being able to read the parent trace: it must be in your organization, and for a MEMBER in a product your user groups grant. Deleting is a soft delete; you may delete your own comments, and organization owners may delete any.
Monitor
Run
Reference Data
Metric
To create a revision of a metric, a user also needs Update on the parent metric, so a MEMBER needs both
metric.create and metric.update.
Model
Analytics & Insights
Analytics
The Read row is
analytics.read. Without it, the dashboard hides the product’s Analytics section, a route for one product answers 403, GET /analytics/summary answers an empty row for that product, and GET /analytics/products returns the product’s row with empty stats. The summary boxes of the organization products page count only the products with analytics.read, and answer 403 when the request names another. The Coverage row (analytics.readCoverage) only controls what the dashboard shows on the Analytics page: the API still returns this data to every role that can read Analytics. The Inference row follows the inference metrics permission of Evaluation: without it, GET /analytics/inference refuses the latency, cost and tokens per version with 403, the API returns the aggregated latency and total cost as null, and leaves those values out of the Analytics export. The dashboard then hides the Inference section and the Performance and Cost blocks of the version card, but GET /analytics still returns the latency, token and average cost summary and GET /analytics/insights/comparison states the average cost or latency of an outlier version in its flags, to every role that reads Analytics.
Dashboard
Dashboard views
These rows decide what the dashboard shows or hides for a role. They do not grant access: the platform still checks the permission behind each action.
Permission list ids
A user group’s permission list may hold only these ids. The role still caps each one.comment, metric and model ids apply across the organization; the others apply to the products the group links. inferenceResult is a Trace in the dashboard, and trace is a Span.
Notes
- Read-only users: A read-only user is a MEMBER in a user group that follows the Reader or Human evaluator preset. The Reader preset lists only read actions. It holds
evaluation.readInference, so its members see cost and latency, and it leaves outevaluation.readMetadata. The Human evaluator preset addsevaluation.humanAnnotation,evaluation.humanEvaluationandevaluation.readMetadata. To hide cost and latency from readers, give their group its own list withoutevaluation.readInference. The role still caps every list. - Permission lists: An OWNER or ADMIN can give a user group a list of permissions, each written
entity.action, for exampleanalytics.readCoverage. A product entry applies to the products the group links. Ametric,modelorcommententry applies across the organization, whatever products the group links, so a group with no products can hold it. A MEMBER may do an action on a product when the role allows it and at least one of the user’s groups that reaches the product lists it, and a metric, model or comment action when the role allows it and any of the user’s groups lists it, so the role caps the list: a list entry above the role does nothing. A list never limits an OWNER or ADMIN. A group with no list grants every permission the role allows, and an empty list grants nothing. A user in no group reaches no product and gets no metric, model or comment action, and product lists answer an empty page. When every group of a user leaves outtest.read,GET /testsanswers 403; the same holds formonitor.readandGET /monitors,metric.readandGET /metrics, andmodel.readandGET /models. Other lists answer an empty page. A list that holds an entry on a product entity needsproduct.read, and an action on an entity needs that entity’sread. An action that uses metrics (evaluations, runs, specifications, monitors, analytics) needsmetric.read, a version action needsmodel.read,comment.createandevaluation.readInferenceneedinferenceResult.read, and reading traces, sessions or inference results needscomment.read. Only actions on entities that belong to a product, and on metrics, models and comments, come from a list; the Permission list ids table shows which. Every other action stays with the role. - Product access: A MEMBER reaches a product, and everything inside it, only when one of their user groups links it. A MEMBER may also create and edit inside those products. A user group with the all-products setting grants every product of the organization, current and future. A user in no group reaches no product. Only a user in a group with the all-products setting whose list allows
product.create(a group with no list does) creates a product. When a user who cannot create a product reaches exactly one product, sign-in opens that product instead of the product list. It lands on Analytics, or on the first section the user may open when Analytics is not allowed. Records with no product, such as a global dataset, stay readable. When read by id, a product outside the granted set answers like a record of another organization. On the dataset list, naming an ungranted product answers 403, and naming another organization’s product returns no dataset. In a user group, theirproduct_idslist only the granted products. For a MEMBER whose lists hold neithermetric.createnormetric.update, metrics follow the products: unless one of their groups links all products and allowsproduct.read, a metric of the organization is visible only when a granted product evaluates it, links it to a specification, or monitors it, and every revision of that metric is visible then. Any other metric of the organization answers like a metric of another organization. Global metrics stay visible whatever products the groups grant, so a user can pick one that no product uses yet. A MEMBER whose lists allowmetric.createormetric.updatesees every metric of the organization, and edits them withmetric.update. A MEMBER cannot delete a metric that has results or specification links in a product where their user groups together do not list bothevaluation.softDeleteandspecification.update: the API answers 403. A metric’sspecification_idslist only the specifications of granted products, and a user group’smetric_idsonly the visible metrics. The analytics impact of excluding a metric revision, and the data metric optimization learns from, count only the granted products. An OWNER is not limited this way. - Organization scope: “In own org” means the record (or its parent, such as a Product) has an organizationId equal to the signed-in user’s organization. This keeps each client organization’s data separate.
- Parent scope: For nested records (Version, Dataset, Evaluation, and so on) access is derived from the parent’s organization. Example: a Version belongs to a Product, and the Product belongs to an Organization. An Evaluation is reached through session to version to product.
- Global records: Some records (Metric, Model, and a Dataset with no product) can be global, meaning not tied to one organization. Global records are readable by every role in addition to their own organization’s records, whatever products their user groups grant. A MEMBER also needs a user group that lists
metric.readto read a global metric, ormodel.readto read a global model (a group with no list does). - Hard delete: Each entity table has two delete rows. Soft delete hides a record without removing it. Delete is a permanent removal, shown as allowed for Session (OWNER within their own organization, MEMBER within the products its user groups grant) and for User Group (OWNER only, within their own organization); for every other entity it is denied to all roles listed here.