feat(go/eino): CloudWeGo Eino instrumentor (TH-8275) - #265

Summary

New Go module github.com/future-agi/traceAI/go/eino (package traceaieino) for CloudWeGo Eino (TH-8275, parent TH-8103).

It is an Eino callbacks.Handler, Eino's own extension point. Register it once with callbacks.AppendGlobalHandlers, or pass it to one run with compose.WithCallbacks / adk.WithCallbacks. Every component run Eino reports then becomes one OpenTelemetry span, with the attribute names Python traceAI uses:

• Parenting comes from the callback context, so one graph or agent run is one trace. Fixture result (D-LR11): Eino passes the parent context, and a tool span is a child of its ToolsNode. A span from another handler on another provider (Langfuse, APMPlus, ...) is never made the parent. A span the caller starts is the parent.
• Streams are one span, which ends after the handler has read its own copy of the stream. Usage comes from the chunk that reports it, and when several chunks report it the largest total wins. Chunks are never summed.
• OnError then OnEnd ends the span once, including when the two run concurrently. An OnEnd without a matching OnStart never ends another run's span (the RunInfo pointer is checked). An OnError without an OnStart gets its own ERROR span.
• Shutdown: Handler.Shutdown(ctx) makes later callbacks no-ops (no panic, no export), waits for spans that are still being finished in the background, and calls ForceFlush on the provider.
• Privacy follows the traceAI convention (coordinator decision 2026-10-16). Content is on by default and the standard FI_HIDE_* switches or TraceConfig turn it off. error.message is a scrubbed first line with no URL, path, IP, host:port, email, user name or home dir; it is dropped when content is hidden. No stack traces, no exception events, no resource attributes, no credentials.
• Dependencies: the OpenTelemetry API and Eino only. The handler has no go/fiotel import and no replace lines. The example wires the stock otlptracehttp exporter.

Files: handler.go (callbacks, parenting, end-once, Shutdown), mapping.go (component to attributes), stream.go (streams, agent events), errors.go (error type and scrubber), traceconfig.go / context.go (same API as the other Go modules), example/ (ADK ChatModelAgent with the eino-ext OpenAI model and a tool), README.md.

How to run

cd go/eino
go test -race ./...                              # unit + example tests, no network, no key
export FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=my-service OPENAI_API_KEY=...  # OPENAI_BASE_URL optional
go run ./example -stream -fail

The docs companion is future-agi/docs PR #979. Its examples/traceai/eino-go/run_e2e.sh builds the example the way the page says (go mod init, go get) from this branch's commit and sends three runs: default capture, plain, and hidden.

Versions

Re-pin: the PRD pinned v0.9.19. v0.9.20 and v0.9.21 have shipped since. The callback interface (callbacks/interface.go, internal/callbacks) has no diff between v0.9.19 and v0.9.21. The only change in a callback payload is the new PromptTokenDetails.CacheWriteTokens, which this module does not read (so it still builds on v0.9.19).

E2E evidence (real, local Future AGI)

Real Eino v0.9.21 ADK ChatModelAgent (eino-ext OpenAI chat model plus a get_weather tool via utils.InferTool) ran through adk.Runner. The handler was registered once with AppendGlobalHandlers, as the page tells customers. It exported through the stock otlptracehttp exporter to the local Future AGI collector (http://localhost:14338/tracer/v1/traces, both headers, project_name, no gzip). The module was fetched with go get github.com/future-agi/traceAI/go/eino@f5c35b610572 (GOPRIVATE, straight from this branch). The model calls went through an OpenAI-compatible local proxy (CLIProxy) to gpt-6-luna; the endpoint was the only thing swapped. Project th8103-rest-eino, commit f5c35b6:

Matrix leg Eino v0.9.19, same commit, project th8103-rest-eino-v0919: f2e27f9a54d60ccc96fc68a2b84c3495, a562af6d74ca87c9ef8ab87b590de994, dd294638c7b4d256687533dfdb25d9f7, 3f30b6b1562c98ca5f153920b66ee171, 1b3e12eddae92cfb77c444905ff62209. fi_verify PASS, same shape. An earlier run of both versions on the first commit 73c26fb also passed; its traces are still in both projects.

fi_verify --expect-type llm --expect-type tool --expect-type agent --expect-type chain excerpt:

project 'th8103-rest-eino': type=['observe']   traces=10 spans=106
trace d541f30b-31ff-6e42-77dc-dab1bcd1b40f (13 spans)
- [chain] trip_request in=109B out=126B (11478ms)
  - [agent] trip_assistant in=139B out=126B (11477ms)
    - [chain] trip_assistant in=0B out=126B (11475ms)
      - [chain] Lambda in=0B out=255B (0ms)
      - [chain] ReAct in=255B out=126B (11475ms)
        - [chain] Init in=255B out=242B (0ms)
        - [llm] ChatModel model=gpt-6-luna tok=394/42 in=242B out=132B (7545ms)
        - [chain] CancelCheck in=132B out=132B (0ms)
        - [chain] ToolNode in=132B out=193B (0ms)
          - [tool] get_weather in=16B out=100B (0ms)
        - [chain] AfterToolCalls in=193B out=193B (0ms)
        - [chain] AfterToolCallsCancelCheck in=193B out=193B (0ms)
        - [llm] ChatModel model=gpt-6-luna tok=454/62 in=565B out=126B (3927ms)
trace b18c11e1-c10f-ff78-93d3-e197643c39f5 (7 spans)
- [chain] trip_request status=ERROR:agent failed
  - [agent] trip_assistant status=ERROR:*openai.APIError
        - [llm] ChatModel model=no-such-model status=ERROR:*openai.APIError (9ms)
PASS

Failing LLM span as stored: error.type=*openai.APIError, error.message="error, status code: 400, status: 400 Bad Request, message: unknown provider for model no-such-model". In the hidden run there is no error.message. Screenshots of the Observe trace view (chat view with system, user, tool and assistant messages, model and tokens), the failed span, the hidden span (REDACTED) and the trace list are kept locally in the workspace evidence/.

Privacy proof

run_e2e.sh puts a unique marker in the run-A question (MARK_DEFAULT) and in the run-C question (MARK_HIDDEN, plus [email protected] and /Users/someone/MARK_HIDDEN.txt), and asks the model to repeat it in the answer. A scan then searched every stored row of the project: all 11 ClickHouse tables with a project_id (spans with all 46 columns, including resource_attrs, span_events, attributes_extra, input, output, status_message) plus Postgres tracer_*.

needle MARK_DEFAULT:  rows_hit=16 tables=['clickhouse.spans','clickhouse.traces'] span_columns=['attributes_extra','attrs_string','input','output']
needle MARK_HIDDEN:   rows_hit=0     needle MARK_HIDDEN_EMAIL: rows_hit=0     needle RUN1_MARK_HIDDEN: rows_hit=0
needle LOCAL_USER / HOME_PATH / HOSTNAME / LOCAL_HOSTNAME / USERS_PATH ("/Users/"): rows_hit=0
needle GOROUTINE / GO_SOURCE (".go:") / PROXY_HOSTPORT ("127.0.0.1:8317"): rows_hit=0
needle FI_API_KEY / FI_SECRET_KEY / CLIPROXY_KEY: rows_hit=0

The v0.9.19 project gives the same result. In other words, the default mode shows content (the opt-out works as documented), and the hidden mode exports nothing of prompts, replies, tool arguments or results, emails, paths or error text. In neither mode do local identity, stack traces or keys reach the collector.

Tests

go test -race ./... is green on Go 1.27.2 and 1.26.9, and on Eino v0.9.19 and v0.9.21. Coverage is 87.5%, and staticcheck 2026.2.2 is clean. The tests use real Eino graphs, chains, lambdas (invoke/stream/collect), ToolsNode and ADK ChatModelAgent runs (streamed and not), with a scripted chat model (callback-enabled and graph-wrapped), against the in-memory recorder. They cover the PRD acceptance rows:

• AC-03: one trace, nodes are children.
• AC-04: a streamed model node is one span; usage comes from the last chunk.
• AC-05: OnError then OnEnd gives one ERROR span, also when concurrent.
• AC-06: an unknown component is kept.
• AC-07: no usage attribute without usage, streamed or not; never summed.
• AC-08: content on by default and off with the hide switches (all attributes, events and status scanned).
• AC-10: Shutdown, then callbacks do not panic or export, and after the provider shuts down they do nothing either.
• AC-12: no parent context starts a new trace, and a stranger's OnEnd ends nothing.
• V-D: double registration, per run and global, gives two spans per component.

They also cover a foreign handler's span (never made the parent), the caller's span (is the parent), a nil RunInfo, the error scrubber, multimodal messages, and the retriever, embedding and prompt mappings. The example test runs example/main.go against a fake OpenAI server and a fake collector. It checks AC-09 (path, both headers, project_name, no gzip) and AC-11 (no key in the body), one trace per question, and the failing run as ERROR.

Limitations / unsupported

• Components that fire no callbacks and are called outside a graph produce no span. Inside a graph, Eino reports every node itself (tested).
• ADK interrupt/resume, checkpoints, DeepAgent, transfer and workflow agents are not tested. The handler marks eino.agent.interrupted / eino.agent.resumed but no test covers them. AgenticModel / AgenticAgent content is written only as generic JSON (not tested).
• An ADK ChatModelAgent shows Eino's internal ReAct nodes (Init, CancelCheck, AfterToolCalls, ...) as CHAIN spans because Eino reports them. There is no filter option yet; that could be a follow-up.
• With streaming, a streamed child span can end a moment (well under 50 ms in tests) after the agent span, because both copies of the stream are read concurrently.
• gen_ai.provider.name is not written: RunInfo.Type names the component (OpenAI), not the endpoint. gen_ai.usage.cache_write_tokens is not read (the field only exists in v0.9.21+). Request temperature and top_p are written only when non-zero, because Eino's Config cannot tell 0 from unset.
• Upstream finding: in streaming ADK runs, Eino writes a message id into the first chunk's Extra map after the chat model's stream has been copied to the callback handlers. A handler that calls schema.ConcatMessages on its copy (which reads Extra) races with that write (seen with -race). This module joins chunks without reading Extra, so it is race-free. Not reported upstream (no upstream contact per the TH-8103 rules).

Deviations from the approved spec (and why)

• No go/fiotel import. The architecture said to import the shared helper landed by TH-8271. The coordinator decision of 2026-10-16 replaced that: each Go module depends only on the OTel API plus its upstream SDK, carries its own TraceConfig, and wires the stock exporter in its example. TH-8271 (feat(go/openai): OpenAI Go instrumentor (TH-8271) #260) did not add go/fiotel. Consolidating the five Go modules into one shared module is a follow-up after the first Go PR merges.
• Content on by default. PRD J6 said off by default. The coordinator's privacy clarification (2026-10-16) says traceAI packages follow TraceConfig (DEFAULT_HIDE_* = False) with the standard FI_HIDE_* switches. Error text is always scrubbed, and local identity is never exported in either mode.
• gen_ai.span.kind is written. D-LR07 said no span kind. The coordinator decision says Go modules use the same attribute names as the Python instrumentors, and Python's span-kind key is gen_ai.span.kind (as in go/openai, go/anthropic, go/genai, go/a2a).
• Re-pinned primary to Eino v0.9.21, with the go.mod minimum kept at the PRD's v0.9.19. Both were tested.

BLOCKED-ON

None. Eino is free OSS, and the run needed no account or spend. Kept as a draft per the TH-8103 process.

🤖 Generated with Claude Code