Skip to content

fix: Foundry Local agent reliability + telemetry pipeline - #28

Merged
nicholasdbrady merged 9 commits into
mainfrom
fix/stats-for-nerds-components
Apr 26, 2026
Merged

nicholasdbrady merged 9 commits into
mainfrom
fix/stats-for-nerds-components

Conversation

@nicholasdbrady

Copy link
Copy Markdown
Contributor

This branch fixes the long-standing RUN_ERROR / ChatClientException that broke every chat through /api/command, plus a related telemetry pipeline bug discovered during verification, plus two minor production papercuts found along the way.

Verified end-to-end through the UI: chat works, tasks created, ai.response + flow + http.request spans appear in the Stats panel with token counts and durations. Status: ready to merge.

What broke and why

Three stacked Foundry Local + agent-framework bugs:

  1. AG-UI auto-injects a Current state of the application: system message on every turn that's >75k chars when there are 200+ todos in the dict. The 0.5B CPU model has a 32k context window. Foundry returned HTTP 500 → RemoteProtocolError → ChatClientException → RUN_ERROR. The wrapper now strips this message before sending to Foundry (model already has a compact view via _build_instructions()). Refs Python: [Bug]: FoundryLocalClient agent.run() fails with JSON serialization error on Foundry Local service microsoft/agent-framework#4084.

  2. Foundry Local streaming is unreliable inside long-running uvicorn processes — stream=true requests drop the connection mid-response. Wrapper translates AG-UI's forced stream=True into a stream=False Foundry call, then synthesizes a single ChatCompletionChunk so the rest of the framework sees what it expects. Refs Streaming endpoint hangs on stream: true requests (Critical) microsoft/foundry-local#422.

  3. init_otel() was not idempotent — uvicorn imports main.py twice, OpenTelemetry refuses to replace the active TracerProvider, and the second call left _ui_span_exporter pointing at an orphaned exporter. /api/telemetry/snapshot returned 0 events even when chats completed successfully. Added an _otel_initialized guard.

What's in this PR

Component Files Tests
Streaming wrapper + AG-UI state strip app/agent/agent.py 5 unit tests
Smoke test + rollback + persist preferred app/agent/main.py 6 unit tests
EP-aware startup app/agent/agent.py 3 unit tests
Idempotent OTel init app/agent/telemetry.py empirical (single enable_instrumentation log line on startup)
timezone import shadow fix app/agent/app_config.py covered by persist test
Get-FileHash PS5.1 fallback app/launch.ps1 local CI sim (BOM, -UseBasicParsing, WindowsApps filter, parses under PS5.1+PS7)
Playwright MCP recovery doc .github/copilot-instructions.md n/a

Smoke test design

The post-switch smoke test (main.py:_smoke_test_inference) sends the exact production payload — live _build_instructions() + serialized ALL_TOOLS — before declaring a model switch successful. NPU runtimes that crash on agent payloads (Foundry-Local#506, see below) get rolled back at switch time with a clear error rather than silently dying on the first chat.

Known limitation: NPU + agent payloads

NPU runtimes (VitisAI, QNN) reliably crash Foundry on requests containing the full system prompt + 10-tool definition list. This is not a regression from this PR — it's microsoft/foundry-local#506, an upstream Foundry runtime bug, and is out of taskbean's control. Without this PR, NPU model switches succeeded and the first chat crashed Foundry → ConnectionResetError → uvicorn died. With this PR, the smoke test catches the crash at switch time and surfaces:

Smoke test failed: Foundry closed the connection on <model>. This usually means the model can't handle agent-shaped requests (system prompt + tools). NPU models are particularly prone to this — see Foundry-Local#506.

The user can fall back to a working CPU/GPU model. CPU + GPU models are unaffected.

Tests

  • 17/17 new unit tests pass (pytest test_phase0_fixes.py)
  • 113/113 broader Python suite (excluding test_telemetry.py which has pre-existing OGA SDK leak issues unrelated to this PR)
  • 149/149 CLI suite (npm test in cli/)
  • All 4 launcher.yml CI checks verified locally (BOM, -UseBasicParsing, WindowsApps filter, parses under both PS 5.1 and PS 7)

Pre-merge code review

Reviewed by code-review agent. Two blocking findings addressed in commit fb8688a:

  1. Pinned agent-framework==1.0.1, agent-framework-openai==1.0.1, agent-framework-ag-ui==1.0.0b260409 because the wrapper overrides private framework internals. Added an upgrade-checklist comment.
  2. Smoke test fallback (when ALL_TOOLS import breaks) now logs a WARNING and explicitly fails over when no tools serialize, instead of silently masking a critical bug.

Also addressed one design concern: don't clear preferredModel on smoke failure (only on load failure). Foundry NPU crashes can be transient — wiping a user's explicit choice on a hiccup is bad UX.

References

nicholasdbrady and others added 9 commits April 24, 2026 22:44
…lth code

Fix critical bugs in the telemetry nerd panel:

- Events were appended to #nerdEvents (overflow:hidden) instead of
  #nerdEventsList (the actual scroll container), making events invisible
  once they overflowed and breaking auto-scroll entirely
- Scroll listener was attached to #nerdEvents which never fires scroll
  events due to overflow:hidden — moved to #nerdEventsList
- Auto-scroll now targets the correct scrollable element
- clearTelemetry now targets #nerdEventsList (also avoids accidentally
  deleting the filter bar via row-cap trimming)
- Add rehydrateEventsList() so events received while on another tab
  appear when switching back to Events
- Validate nerdTab localStorage against known tabs to prevent blank
  panel if stale 'health' value is stored
- Remove dead renderHealth/fmtUptime/EP-fetch code and #nerdHealth CSS
  (health tab was removed but JS/CSS remained, risking NPE)

Co-authored-by: Copilot <[email protected]>
Add 5 micro-interaction enhancements to the stats-for-nerds telemetry panel:

1. Token animation — metric values pulse (1.08x scale) on change with
   fade-in delta indicators, making counter updates feel alive

2. Sparkline tooltips — hover over sparkline charts to see exact values
   with a floating tooltip and vertical crosshair line on the canvas

3. Thinking indicator — animated 'thinking...' row with pulsing dots
   appears in Events tab during model inference, removed when response
   arrives. Pure UI element (not stored in event buffer)

4. Event grouping — events sharing a traceId are visually nested under
   the parent with left-border accent, indent, and a collapsible toggle
   showing span count (e.g., '▸ 3 spans')

5. Trace tooltips — hover waterfall span bars to see operation name,
   duration, and key GenAI tags (model, tokens, finish_reason) in a
   floating tooltip without needing to expand

All animations respect prefers-reduced-motion: reduce. Tooltips include
role=tooltip for screen readers. Theme-coherent via CSS custom properties.

Co-authored-by: Copilot <[email protected]>
- Replace transition:all with explicit properties on .telem-group-toggle
  and .trace-toolbar button (anti-pattern per WIG)
- Add font-variant-numeric: tabular-nums to .metric-value and
  .trace-span-duration for aligned numeric columns

Co-authored-by: Copilot <[email protected]>
Replace bespoke telemetry plumbing with agent-framework's built-in
OpenTelemetry support:

- Add missing telem.emit() function (5 call sites were broken)
- Call enable_instrumentation() after init_otel() to activate native
  AgentTelemetryLayer, ChatTelemetryLayer, and FunctionTool span
  creation — these are already in the Agent/Client MRO, just gated
  behind a flag
- Expand UISpanExporter whitelist to pass native spans through:
  'execute_tool *' → tool.executed, 'chat *' → ai.response,
  'invoke_agent *' → flow
- Enrich span export with durationMs (from span timestamps), spanName,
  and gen_ai.tool.name normalization
- Remove _traced_tool() wrapper entirely (−76 lines) — native
  FunctionTool.invoke() creates execute_tool spans with proper
  gen_ai.tool.name, gen_ai.tool.call.id, args, result. Delivered
  synchronously via SimpleSpanProcessor → UISpanExporter → SSE.
- Keep manual gen_ai.* attributes in _foundry_complete() — verified
  it bypasses agent framework (direct httpx to Foundry Local endpoint)
- Add invoke_agent summary rendering in nerd panel Events tab

Co-authored-by: Copilot <[email protected]>
…roup leak

- Remove dead _tagSummary() call and unused tooltipContent variable in
  renderTraceWaterfall() — _tagSummary was never defined, would throw
  ReferenceError when expanding any trace
- Add ai.error to thinking indicator removal conditions — indicator
  would remain stuck on screen indefinitely if AI request failed
- Clean up _traceGroups Map when parent rows are trimmed by the
  NERD_EVENT_MAX cap — prevents stale DOM references and unbounded
  Map growth

Co-authored-by: Copilot <[email protected]>
- Explicitly note no build step and no linter for either half
- Add Node.js >=22.5.0 engine requirement
- Add CLI test section with single-test commands
- Note that CLI tests must run from cli/ directory

Co-authored-by: Copilot <[email protected]>
- Snapshot-first workflow (snapshot → act → snapshot)
- Never use screenshots for element targeting
- App-specific notes: Enter key for chat, service worker, large SPA snapshots

Co-authored-by: Copilot <[email protected]>
The /api/command endpoint was reliably returning RUN_ERROR for all chats
because of three stacked Foundry Local + agent-framework bugs. This
commit fixes all three plus a telemetry pipeline bug discovered during
verification, plus two minor production papercuts found along the way.

## Streaming wrapper (agent.py)

Added _NormalizingChatClient._inner_get_response that translates AG-UI's
forced stream=True into a non-streaming Foundry call, then synthesizes
a single ChatCompletionChunk so the rest of agent-framework sees what
it expects. Foundry-Local#422 makes streaming unreliable inside the
long-running uvicorn process; non-streaming works.

Also strips AG-UI's auto-injected "Current state of the application:"
system message before sending to Foundry. With 200+ todos this message
is 75k+ chars and blows the 32k context window of qwen2.5-coder-0.5b.

## Smoke test (main.py)

Added _smoke_test_inference that probes the FULL agent payload (live
_build_instructions() + serialized ALL_TOOLS) before declaring a model
switch successful. Catches Foundry NPU runtime crashes (#506) at switch
time so users see a clear rollback instead of the server dying on the
first chat. _commit_switch now snapshots prior state and restores on
failure. Persists target_model.id (full ID, not alias) so switches
survive restart and don't bounce back to broken NPU defaults.

## EP-aware startup (agent.py)

Captures EpDownloadResult and filters cached models by registered
execution providers. CPU + WebGPU are always considered registered.
Without this, cached[0] auto-selection lands on a NPU model when
VitisAI EP failed to register, leaving the agent unusable.

## Idempotent telemetry init (telemetry.py)

Uvicorn imports main.py twice (worker + main process). OpenTelemetry
refuses to replace the active TracerProvider, but the old code still
mutated the module-level _ui_span_exporter on every call — leaving it
pointing at an orphaned exporter that never received spans. Added an
_otel_initialized guard so init_otel() is no-op on subsequent calls.
This restored ai.response, flow, and http.request events in the Stats
panel along with token counts.

## App config timezone shadow (app_config.py)

The local timezone() function shadowed the datetime.timezone import,
breaking app_config.set() with "'function' object has no attribute
'utc'". Imported datetime.timezone as _tz instead.

## Launch script Get-FileHash fallback (launch.ps1)

On rare PS 5.1 configurations Microsoft.PowerShell.Utility isn't
auto-imported when launch.ps1 runs via the taskbean:// protocol
handler, surfacing a confusing red error. Added a manual SHA256
fallback so dep-stamp gating still works silently.

## Testing

- 17 new unit tests in app/agent/test_phase0_fixes.py covering all
  five fixes including edge cases (size guard on state-context filter,
  smoke test rollback, persist-on-success, clear-on-failure).
- Full Python suite: 113/113 (test_phase0_fixes + test_agui_refactor
  + test_version_endpoint + test_fs_endpoints + test_projects_api
  + test_usage_scanners).
- CLI suite: 149/149.
- Manual E2E verified end-to-end through the UI: chat works, tasks
  created, ai.response/flow/http.request spans appear in the Stats
  panel with token counts and durations.

## Known limitation: NPU + agent payloads

NPU runtimes (VitisAI, QNN) reliably crash Foundry on requests
containing the full system prompt + 10-tool definition list. This is
Foundry-Local#506 and is out of taskbean's control. The smoke test
catches this at switch time and rolls back with an actionable error
message ("Foundry closed the connection ... NPU models are
particularly prone to this — see Foundry-Local#506") rather than
letting the real chat take down the server. CPU and GPU paths are
unaffected.

## Documentation

Updated .github/copilot-instructions.md with the Playwright MCP
recovery procedure (use configure-copilot agent to restart the MCP
server, then click the bridge extension to reconnect).

References:
- microsoft/foundry-local#422 (streaming bug)
- microsoft/foundry-local#506 (NPU + tools)
- microsoft/foundry-local#558 (VitisAI EP)
- microsoft/agent-framework#4084 (multi-modal content)

Co-authored-by: Copilot <[email protected]>
Three follow-up changes from the pre-merge review of 14332f2:

## 1. Pin agent-framework versions (requirements.txt)

The streaming wrapper in agent.py overrides private framework internals
(_inner_get_response, _parse_response_update_from_openai,
_build_response_stream, _prepare_options) to work around
Foundry-Local#422. A version bump can silently break the wrapper. Pinned
to the versions the wrapper was written and tested against:

- agent-framework==1.0.1
- agent-framework-openai==1.0.1
- agent-framework-ag-ui==1.0.0b260409

Added a comment with the upgrade checklist (re-read internals, run
test_phase0_fixes, manual E2E + telemetry check).

## 2. Smoke test fallback now logs a warning (main.py)

Previously a broken tool import would silently fall back to a 1-tool
probe, defeating the smoke test's purpose (which is to catch
agent-payload incompatibilities). Now logs an actionable WARNING
explaining the smoke test is no longer reliable until the import is
fixed. Also explicitly fails over when _serialize_tools_for_smoke
returns an empty list (every tool failed to serialize).

## 3. Don't clear preferredModel on smoke test failure (main.py)

Previously _clear_preferred_if_matches was called on BOTH load failure
and smoke test failure. Smoke test failures can be transient — Foundry
NPU runtimes crash intermittently on agent payloads (Foundry-Local#506),
but a fresh process restart often recovers. Wiping the user's explicit
model choice on a transient hiccup is bad UX. Load failures (model
fundamentally cannot load) still clear the preference; that's correct
because the model is unrecoverable from the user's perspective.

All 17 unit tests still pass.

Co-authored-by: Copilot <[email protected]>
@nicholasdbrady
nicholasdbrady merged commit df12d55 into main Apr 26, 2026
6 checks passed
@nicholasdbrady
nicholasdbrady deleted the fix/stats-for-nerds-components branch April 26, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant