Monitor Real User Traffic via OpenTelemetry

​

Prerequisites

• A Product in Galtea. A Version too, if you route by version ID.
• A Galtea API key.
• Your system emits OpenTelemetry traces.

# Add this to the Collector config from the "Send OpenTelemetry Traces to Galtea" setup page.
# The otlp receiver, the batch processor, and the otlphttp/galtea exporter
# are defined there; here you only add the resource/galtea processor and
# reference it in the traces pipeline.
processors:
  resource/galtea:
    attributes:
      - key: galtea.version.id
        value: "<your-version-id>"
        action: upsert

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource/galtea, batch]
      exporters: [otlphttp/galtea]

processors:
  resource/galtea:
    attributes:
      - key: galtea.product.id
        value: "<your product id>"
        action: upsert
      - key: galtea.version.fact.release
        from_attribute: service.version
        action: upsert
      - key: galtea.version.fact.model
        value: "gpt-4o"
        action: upsert
      - key: galtea.version.fact.temperature
        value: "0.7"
        action: upsert

export OTEL_RESOURCE_ATTRIBUTES="galtea.product.id=<your product id>,galtea.version.fact.model=gpt-4o,galtea.version.fact.release=1.4.2"

curl -X PATCH "https://api.galtea.ai/organizations/<organization-id>" \
  -H "Authorization: Bearer ${GALTEA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"otelVersionAttributes": ["service.version", "app.agent.model"]}'

{"service.version": "1.4.0", "app.agent.model": "gpt-4.1"}

# The resource form. Same Collector processor as the previous step, with one more attribute.
processors:
  resource/galtea:
    attributes:
      - key: galtea.version.id
        value: "<your-version-id>"
        action: upsert
      - key: galtea.test_case.id
        value: "<your-test-case-id>"
        action: upsert

​

What You Get

• Production sessions. Sessions created this way are production sessions, so Monitors evaluate them automatically. The one exception is a conversation that names a test case: it creates a non-production session, which no Monitor scores.
• Automatic input and output. Each turn’s input and output are read from the turn’s own spans. A managed platform whose LLM text lives only in gen_ai.* attributes or vendor span events still produces turns with non-empty input and output. When one span is the root of the turn (for example, an agent span that wraps child LLM and tool spans, the usual shape from MLflow autolog or an agent framework), the turn takes the root span’s input and output: the user message and the agent’s final answer. This applies per field. The input always comes from the root when the root carries it. The output comes from the root only when the root’s output is a plain answer (a text reply, or a chat-completion or assistant_message envelope that reduces to text); when the root’s output is a wrapper object that is not the answer (for example a status or tool-decision blob), the turn keeps the child span’s real reply instead. A field the root does not carry, or a turn with no single root (for example, several sibling LLM calls), falls back to the first prompt as the turn input and the last completion as the turn output. A configured input or output source rule replaces that fallback and the root span for its field, and leaves the field empty when it matches no span rather than falling back: see Recover the turn input from a non-LLM span. If the chosen output is a raw OpenAI chat-completion object, the turn shows its message text; the span record keeps the full object. See How span content maps to Span records for the per-field precedence. For turns answered without any LLM call, see Recover replies that do not come from the LLM.
• Idempotent ingestion. Re-sent or late span batches do not create duplicate sessions, turns, or spans. Later turns append to the same session.
• Works together with the SDK. If a session with the same custom ID already exists (for example, created by the SDK), ingestion attaches to it instead of creating a new one.

​

How Galtea Routes a Trace

1. galtea.version.id: Galtea uses this Version and reads nothing else.
2. galtea.version.fact.*: the facts are these attributes, without the prefix.
3. The version attributes: the facts are the listed attributes the resource carries. A product’s own list replaces the organization list.
4. None of these: Galtea rejects the spans as no_version_id.

1. galtea.product.id.
2. The key values: the product whose key values the resource carries. When several products match, the key with the most attributes wins. A tie routes nowhere.

• The version attributes:
  • Dashboard: the OpenTelemetry card of the Update Organization page. A product replaces the list in Advanced Options of its edit form.
  • API: PATCH /organizations/{id} and PATCH /products/{id}, field otelVersionAttributes.
  • CLI: galtea organizations update and galtea products update.
• The key and the key values:
  • Dashboard: the OpenTelemetry card of the Update Organization page, for the key. Advanced Options of each product’s edit form, for its values or its own key.
  • API: the same two endpoints, fields otelProductKey and otelProductKeyValues.
  • CLI: the same two commands.
• The input and output source rules (output, input):
  • Dashboard: the OpenTelemetry card of the Update Organization page, and Advanced Options of each product’s edit form.
  • API: the same two endpoints, fields otelInputSourceRule and otelOutputSourceRule.
  • CLI: the same two commands.

​

Route Traces to a Product by Resource Attributes

• The product key belongs to the organization. It lists 1 to 10 resource attribute names that identify a product, for example service.namespace and service.name. A name cannot start with galtea. and cannot be customer.id. A new organization uses service.name.
• The key values belong to the product. They give one value for each attribute of the key, for example service.namespace is checkout and service.name is checkout-agent.

• Every attribute is required. A resource must carry every attribute of the key. A resource that misses one matches no product. Matching is exact and case-sensitive.
• A product can use its own key. Set otelProductKey on the product to use other attributes than the organization. It is null while the product follows the organization key, and then it follows every later change.
• The key with the most attributes wins. When a resource matches several products, the product whose key has the most attributes wins, whether that key is its own or the organization key. A tie routes the resource nowhere.
• Key values are unique. Two live products of one organization cannot hold the same key values. The second request answers 409.
• A value cannot be unknown_service. The OpenTelemetry SDK sets service.name to unknown_service or unknown_service:<process> when you set no name. Galtea refuses these as key values, because they would name no real product.
• The key is locked while products use it. The organization cannot change the set of attributes of its key while a live product that follows the key holds key values. The request answers 409. You can reorder the same attributes. To change the set, clear those key values first or give the products their own key.
• Stored traces keep their product. Galtea routes once, when it ingests a trace. A later change of a key or of key values moves no stored trace.

curl -X PATCH "https://api.galtea.ai/organizations/<organization-id>" \
  -H "Authorization: Bearer ${GALTEA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"otelProductKey": ["service.namespace", "service.name"]}'

curl -X PATCH "https://api.galtea.ai/products/<product-id>" \
  -H "Authorization: Bearer ${GALTEA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"otelProductKeyValues": {"service.namespace": "checkout", "service.name": "checkout-agent"}}'

export OTEL_RESOURCE_ATTRIBUTES="service.namespace=checkout,service.name=checkout-agent,galtea.version.fact.model=gpt-4o,galtea.version.fact.release=1.4.2"

​

Batched Exporters and the Conversation Key

• The wait is bounded. Spans wait 15 minutes by default. If no request ever carries a key for that trace, they are dropped and the reason is logged. So an instrumentation that never sets a conversation key stores nothing, exactly as before. Self-hosted deployments can change the window with the OTEL_KEYLESS_HOLD_WINDOW_SECONDS setting, and the per-trace cap on held spans with OTEL_KEYLESS_HOLD_MAX_SPANS.
• You may still set the key on every span. It is not required, and it costs nothing to keep if your instrumentation already does it.

​

Recover Replies That Do Not Come from the LLM

• spanName — the exact OTLP span name to match. For a Temporal (managed-platform) activity, this is the activity span name.
• attribute — the span attribute that holds the reply, stored as a JSON string.
• path — a JSONPath into that parsed JSON that points at the output text. JSONPath is a small query syntax for JSON, for example $[0].message.content. It must start with $, the root of the parsed value; a path without it matches nothing, so the API and the dashboard both reject one.
• predicate (optional) — an object { path, equals }. The rule applies only when the node at predicate.path equals equals. Use it when the user message and the assistant message share the same span and attribute, so you do not pick the user’s own message as the reply.

{
  "spanName": "ExecuteActivity:save_reply",
  "attribute": "app.arguments",
  "path": "$[0].message.content",
  "predicate": { "path": "$[0].message.role", "equals": "assistant" }
}

​

Recover the Turn Input from a Non-LLM Span

• spanName — the exact OTLP span name to match. For a Temporal (managed-platform) activity, this is the activity span name.
• attribute — the span attribute that holds the user message, stored as a JSON string.
• path — a JSONPath into that parsed JSON that points at the input text. JSONPath is a small query syntax for JSON, for example $[0].message.content. It must start with $, the root of the parsed value; a path without it matches nothing, so the API and the dashboard both reject one.
• predicate (optional) — an object { path, equals }. The rule applies only when the node at predicate.path equals equals. Use it when the user message and the assistant message share the same span and attribute, so you do not pick the assistant’s reply as the input.

{
  "spanName": "ExecuteActivity:save_message",
  "attribute": "app.arguments",
  "path": "$[0].message.content",
  "predicate": { "path": "$[0].message.role", "equals": "user" }
}

​

How a Rule Is Resolved and Changed

• A version inherits the product rule, and a product inherits the organization rule. Galtea uses the version rule when the version has one, then the product rule, then the organization rule. So a version whose rule is null uses the product rule, and a product whose rule is null uses the organization rule. A version cannot opt out of an inherited rule today. Pointing the version rule at a span name that version never emits does not restore the inherited rule: it empties the field for every turn of that version. To stop recovering a field, clear the product and organization rules, or point the version rule at a span that version does emit.
• A rule change reaches a stored turn when it extracts a value. Galtea keeps every span it stored, so a turn that still receives new spans is recomputed with the rule configured at that moment, over all of its spans. Correcting a rule therefore repairs those turns, and a turn recorded before you set the rule is repaired the same way. A turn that receives no further spans keeps the value it has.
• A change never empties a field that holds a value. If your new rule extracts nothing from a turn, Galtea leaves that turn’s stored value alone rather than clearing it. Clearing stored values is a separate operation, so an edit to a rule cannot blank recorded traffic as a side effect. A rule change is never what clears a stored field. One platform action can: when a turn’s boundary span arrives after its content, Galtea moves that content to the turn it belongs to, and the turn it came from is left empty rather than keeping the next turn’s words. That clears only a field those same spans filled.
• A rule that matches nothing empties its field. A configured rule decides its direction outright. When it matches no span of the turn, that field is empty, and the root-span and earliest/latest fallbacks do not apply. Each direction is independent, so an input rule that matches nothing does not stop the output rule. If a rule of yours needs the fallback instead, ask us: it becomes an explicit setting on the rule, never the default.
• A recovered field is marked on the span. A span whose field a rule filled carries galtea.span.input_from_source_rule or galtea.span.output_from_source_rule in its metadata. Galtea writes and reads these keys itself, and strips them from any value you send. Use them to confirm which span a rule fired on.

​

Learn More