Monitor Real User Traffic via OpenTelemetry

​

Prerequisites

• A Product in Galtea. A Version too, if you route by version ID.
• A Galtea API key.
• Your system emits OpenTelemetry traces.

# Add this to the Collector config from the "Send OpenTelemetry Traces to Galtea" setup page.
# The otlp receiver, the batch processor, and the otlphttp/galtea exporter
# are defined there; here you only add the resource/galtea processor and
# reference it in the traces pipeline.
processors:
  resource/galtea:
    attributes:
      - key: galtea.version.id
        value: "<your-version-id>"
        action: upsert

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource/galtea, batch]
      exporters: [otlphttp/galtea]

processors:
  resource/galtea:
    attributes:
      - key: galtea.product.id
        value: "<your product id>"
        action: upsert
      - key: galtea.version.fact.release
        from_attribute: service.version
        action: upsert
      - key: galtea.version.fact.model
        value: "gpt-4o"
        action: upsert
      - key: galtea.version.fact.temperature
        value: "0.7"
        action: upsert

export OTEL_RESOURCE_ATTRIBUTES="galtea.product.id=<your product id>,galtea.version.fact.model=gpt-4o,galtea.version.fact.release=1.4.2"

# The resource form. Same Collector processor as the previous step, with one more attribute.
processors:
  resource/galtea:
    attributes:
      - key: galtea.version.id
        value: "<your-version-id>"
        action: upsert
      - key: galtea.test_case.id
        value: "<your-test-case-id>"
        action: upsert

​

What You Get

• Production sessions. Sessions created this way are production sessions, so Monitors evaluate them automatically. The one exception is a conversation that names a test case: it creates a non-production session, which no Monitor scores.
• Automatic input and output. Each turn’s input and output are read from the turn’s own spans. A managed platform whose LLM text lives only in gen_ai.* attributes or vendor span events still produces turns with non-empty input and output. When one span is the root of the turn (for example, an agent span that wraps child LLM and tool spans, the usual shape from MLflow autolog or an agent framework), the turn takes the root span’s input and output: the user message and the agent’s final answer. This applies per field. The input always comes from the root when the root carries it. The output comes from the root only when the root’s output is a plain answer (a text reply, or a chat-completion or assistant_message envelope that reduces to text); when the root’s output is a wrapper object that is not the answer (for example a status or tool-decision blob), the turn keeps the child span’s real reply instead. A field the root does not carry, or a turn with no single root (for example, several sibling LLM calls), falls back to the first prompt as the turn input and the last completion as the turn output. A configured input or output source rule replaces that fallback and the root span for its field, and leaves the field empty when it matches no span rather than falling back: see Recover the turn input from a non-LLM span. If the chosen output is a raw OpenAI chat-completion object, the turn shows its message text; the span record keeps the full object. See How span content maps to Span records for the per-field precedence. For turns answered without any LLM call, see Recover replies that do not come from the LLM.
• Idempotent ingestion. Re-sent or late span batches do not create duplicate sessions, turns, or spans. Later turns append to the same session.
• Works together with the SDK. If a session with the same custom ID already exists (for example, created by the SDK), ingestion attaches to it instead of creating a new one.

​

Batched Exporters and the Conversation Key

• The wait is bounded. Spans wait 15 minutes by default. If no request ever carries a key for that trace, they are dropped and the reason is logged. So an instrumentation that never sets a conversation key stores nothing, exactly as before. Self-hosted deployments can change the window with the OTEL_KEYLESS_HOLD_WINDOW_SECONDS setting, and the per-trace cap on held spans with OTEL_KEYLESS_HOLD_MAX_SPANS.
• You may still set the key on every span. It is not required, and it costs nothing to keep if your instrumentation already does it.

​

Recover Replies That Do Not Come from the LLM

• spanName — the exact OTLP span name to match. For a Temporal (managed-platform) activity, this is the activity span name.
• attribute — the span attribute that holds the reply, stored as a JSON string.
• path — a JSONPath into that parsed JSON that points at the output text. JSONPath is a small query syntax for JSON, for example $[0].message.content. It must start with $, the root of the parsed value; a path without it matches nothing, so the API and the dashboard both reject one.
• predicate (optional) — an object { path, equals }. The rule applies only when the node at predicate.path equals equals. Use it when the user message and the assistant message share the same span and attribute, so you do not pick the user’s own message as the reply.

{
  "spanName": "ExecuteActivity:save_reply",
  "attribute": "app.arguments",
  "path": "$[0].message.content",
  "predicate": { "path": "$[0].message.role", "equals": "assistant" }
}

​

Recover the Turn Input from a Non-LLM Span

• spanName — the exact OTLP span name to match. For a Temporal (managed-platform) activity, this is the activity span name.
• attribute — the span attribute that holds the user message, stored as a JSON string.
• path — a JSONPath into that parsed JSON that points at the input text. JSONPath is a small query syntax for JSON, for example $[0].message.content. It must start with $, the root of the parsed value; a path without it matches nothing, so the API and the dashboard both reject one.
• predicate (optional) — an object { path, equals }. The rule applies only when the node at predicate.path equals equals. Use it when the user message and the assistant message share the same span and attribute, so you do not pick the assistant’s reply as the input.

{
  "spanName": "ExecuteActivity:save_message",
  "attribute": "app.arguments",
  "path": "$[0].message.content",
  "predicate": { "path": "$[0].message.role", "equals": "user" }
}

​

How a Rule Is Resolved and Changed

• A version inherits the product rule. Galtea uses the version rule when the version has one, and the product rule when it does not. So a version whose rule is null uses the product rule. A version cannot opt out of a product rule today. Pointing the version rule at a span name that version never emits does not restore the product rule: it empties the field for every turn of that version. To stop recovering a field, clear the product rule, or point the version rule at a span that version does emit.
• A rule change reaches a stored turn when it extracts a value. Galtea keeps every span it stored, so a turn that still receives new spans is recomputed with the rule configured at that moment, over all of its spans. Correcting a rule therefore repairs those turns, and a turn recorded before you set the rule is repaired the same way. A turn that receives no further spans keeps the value it has.
• A change never empties a field that holds a value. If your new rule extracts nothing from a turn, Galtea leaves that turn’s stored value alone rather than clearing it. Clearing stored values is a separate operation, so an edit to a rule cannot blank recorded traffic as a side effect. A rule change is never what clears a stored field. One platform action can: when a turn’s boundary span arrives after its content, Galtea moves that content to the turn it belongs to, and the turn it came from is left empty rather than keeping the next turn’s words. That clears only a field those same spans filled.
• A rule that matches nothing empties its field. A configured rule decides its direction outright. When it matches no span of the turn, that field is empty, and the root-span and earliest/latest fallbacks do not apply. Each direction is independent, so an input rule that matches nothing does not stop the output rule. If a rule of yours needs the fallback instead, ask us: it becomes an explicit setting on the rule, never the default.
• A recovered field is marked on the span. A span whose field a rule filled carries galtea.span.input_from_source_rule or galtea.span.output_from_source_rule in its metadata. Galtea writes and reads these keys itself, and strips them from any value you send. Use them to confirm which span a rule fired on.

​

Learn More