Repository navigation
fix: Foundry Local agent reliability + telemetry pipeline - #28
Merged
Merged
Conversation
…lth code Fix critical bugs in the telemetry nerd panel: - Events were appended to #nerdEvents (overflow:hidden) instead of #nerdEventsList (the actual scroll container), making events invisible once they overflowed and breaking auto-scroll entirely - Scroll listener was attached to #nerdEvents which never fires scroll events due to overflow:hidden — moved to #nerdEventsList - Auto-scroll now targets the correct scrollable element - clearTelemetry now targets #nerdEventsList (also avoids accidentally deleting the filter bar via row-cap trimming) - Add rehydrateEventsList() so events received while on another tab appear when switching back to Events - Validate nerdTab localStorage against known tabs to prevent blank panel if stale 'health' value is stored - Remove dead renderHealth/fmtUptime/EP-fetch code and #nerdHealth CSS (health tab was removed but JS/CSS remained, risking NPE) Co-authored-by: Copilot <[email protected]>
Add 5 micro-interaction enhancements to the stats-for-nerds telemetry panel: 1. Token animation — metric values pulse (1.08x scale) on change with fade-in delta indicators, making counter updates feel alive 2. Sparkline tooltips — hover over sparkline charts to see exact values with a floating tooltip and vertical crosshair line on the canvas 3. Thinking indicator — animated 'thinking...' row with pulsing dots appears in Events tab during model inference, removed when response arrives. Pure UI element (not stored in event buffer) 4. Event grouping — events sharing a traceId are visually nested under the parent with left-border accent, indent, and a collapsible toggle showing span count (e.g., '▸ 3 spans') 5. Trace tooltips — hover waterfall span bars to see operation name, duration, and key GenAI tags (model, tokens, finish_reason) in a floating tooltip without needing to expand All animations respect prefers-reduced-motion: reduce. Tooltips include role=tooltip for screen readers. Theme-coherent via CSS custom properties. Co-authored-by: Copilot <[email protected]>
- Replace transition:all with explicit properties on .telem-group-toggle and .trace-toolbar button (anti-pattern per WIG) - Add font-variant-numeric: tabular-nums to .metric-value and .trace-span-duration for aligned numeric columns Co-authored-by: Copilot <[email protected]>
Replace bespoke telemetry plumbing with agent-framework's built-in OpenTelemetry support: - Add missing telem.emit() function (5 call sites were broken) - Call enable_instrumentation() after init_otel() to activate native AgentTelemetryLayer, ChatTelemetryLayer, and FunctionTool span creation — these are already in the Agent/Client MRO, just gated behind a flag - Expand UISpanExporter whitelist to pass native spans through: 'execute_tool *' → tool.executed, 'chat *' → ai.response, 'invoke_agent *' → flow - Enrich span export with durationMs (from span timestamps), spanName, and gen_ai.tool.name normalization - Remove _traced_tool() wrapper entirely (−76 lines) — native FunctionTool.invoke() creates execute_tool spans with proper gen_ai.tool.name, gen_ai.tool.call.id, args, result. Delivered synchronously via SimpleSpanProcessor → UISpanExporter → SSE. - Keep manual gen_ai.* attributes in _foundry_complete() — verified it bypasses agent framework (direct httpx to Foundry Local endpoint) - Add invoke_agent summary rendering in nerd panel Events tab Co-authored-by: Copilot <[email protected]>
…roup leak - Remove dead _tagSummary() call and unused tooltipContent variable in renderTraceWaterfall() — _tagSummary was never defined, would throw ReferenceError when expanding any trace - Add ai.error to thinking indicator removal conditions — indicator would remain stuck on screen indefinitely if AI request failed - Clean up _traceGroups Map when parent rows are trimmed by the NERD_EVENT_MAX cap — prevents stale DOM references and unbounded Map growth Co-authored-by: Copilot <[email protected]>
- Explicitly note no build step and no linter for either half - Add Node.js >=22.5.0 engine requirement - Add CLI test section with single-test commands - Note that CLI tests must run from cli/ directory Co-authored-by: Copilot <[email protected]>
- Snapshot-first workflow (snapshot → act → snapshot) - Never use screenshots for element targeting - App-specific notes: Enter key for chat, service worker, large SPA snapshots Co-authored-by: Copilot <[email protected]>
The /api/command endpoint was reliably returning RUN_ERROR for all chats
because of three stacked Foundry Local + agent-framework bugs. This
commit fixes all three plus a telemetry pipeline bug discovered during
verification, plus two minor production papercuts found along the way.
## Streaming wrapper (agent.py)
Added _NormalizingChatClient._inner_get_response that translates AG-UI's
forced stream=True into a non-streaming Foundry call, then synthesizes
a single ChatCompletionChunk so the rest of agent-framework sees what
it expects. Foundry-Local#422 makes streaming unreliable inside the
long-running uvicorn process; non-streaming works.
Also strips AG-UI's auto-injected "Current state of the application:"
system message before sending to Foundry. With 200+ todos this message
is 75k+ chars and blows the 32k context window of qwen2.5-coder-0.5b.
## Smoke test (main.py)
Added _smoke_test_inference that probes the FULL agent payload (live
_build_instructions() + serialized ALL_TOOLS) before declaring a model
switch successful. Catches Foundry NPU runtime crashes (#506) at switch
time so users see a clear rollback instead of the server dying on the
first chat. _commit_switch now snapshots prior state and restores on
failure. Persists target_model.id (full ID, not alias) so switches
survive restart and don't bounce back to broken NPU defaults.
## EP-aware startup (agent.py)
Captures EpDownloadResult and filters cached models by registered
execution providers. CPU + WebGPU are always considered registered.
Without this, cached[0] auto-selection lands on a NPU model when
VitisAI EP failed to register, leaving the agent unusable.
## Idempotent telemetry init (telemetry.py)
Uvicorn imports main.py twice (worker + main process). OpenTelemetry
refuses to replace the active TracerProvider, but the old code still
mutated the module-level _ui_span_exporter on every call — leaving it
pointing at an orphaned exporter that never received spans. Added an
_otel_initialized guard so init_otel() is no-op on subsequent calls.
This restored ai.response, flow, and http.request events in the Stats
panel along with token counts.
## App config timezone shadow (app_config.py)
The local timezone() function shadowed the datetime.timezone import,
breaking app_config.set() with "'function' object has no attribute
'utc'". Imported datetime.timezone as _tz instead.
## Launch script Get-FileHash fallback (launch.ps1)
On rare PS 5.1 configurations Microsoft.PowerShell.Utility isn't
auto-imported when launch.ps1 runs via the taskbean:// protocol
handler, surfacing a confusing red error. Added a manual SHA256
fallback so dep-stamp gating still works silently.
## Testing
- 17 new unit tests in app/agent/test_phase0_fixes.py covering all
five fixes including edge cases (size guard on state-context filter,
smoke test rollback, persist-on-success, clear-on-failure).
- Full Python suite: 113/113 (test_phase0_fixes + test_agui_refactor
+ test_version_endpoint + test_fs_endpoints + test_projects_api
+ test_usage_scanners).
- CLI suite: 149/149.
- Manual E2E verified end-to-end through the UI: chat works, tasks
created, ai.response/flow/http.request spans appear in the Stats
panel with token counts and durations.
## Known limitation: NPU + agent payloads
NPU runtimes (VitisAI, QNN) reliably crash Foundry on requests
containing the full system prompt + 10-tool definition list. This is
Foundry-Local#506 and is out of taskbean's control. The smoke test
catches this at switch time and rolls back with an actionable error
message ("Foundry closed the connection ... NPU models are
particularly prone to this — see Foundry-Local#506") rather than
letting the real chat take down the server. CPU and GPU paths are
unaffected.
## Documentation
Updated .github/copilot-instructions.md with the Playwright MCP
recovery procedure (use configure-copilot agent to restart the MCP
server, then click the bridge extension to reconnect).
References:
- microsoft/foundry-local#422 (streaming bug)
- microsoft/foundry-local#506 (NPU + tools)
- microsoft/foundry-local#558 (VitisAI EP)
- microsoft/agent-framework#4084 (multi-modal content)
Co-authored-by: Copilot <[email protected]>
Three follow-up changes from the pre-merge review of 14332f2: ## 1. Pin agent-framework versions (requirements.txt) The streaming wrapper in agent.py overrides private framework internals (_inner_get_response, _parse_response_update_from_openai, _build_response_stream, _prepare_options) to work around Foundry-Local#422. A version bump can silently break the wrapper. Pinned to the versions the wrapper was written and tested against: - agent-framework==1.0.1 - agent-framework-openai==1.0.1 - agent-framework-ag-ui==1.0.0b260409 Added a comment with the upgrade checklist (re-read internals, run test_phase0_fixes, manual E2E + telemetry check). ## 2. Smoke test fallback now logs a warning (main.py) Previously a broken tool import would silently fall back to a 1-tool probe, defeating the smoke test's purpose (which is to catch agent-payload incompatibilities). Now logs an actionable WARNING explaining the smoke test is no longer reliable until the import is fixed. Also explicitly fails over when _serialize_tools_for_smoke returns an empty list (every tool failed to serialize). ## 3. Don't clear preferredModel on smoke test failure (main.py) Previously _clear_preferred_if_matches was called on BOTH load failure and smoke test failure. Smoke test failures can be transient — Foundry NPU runtimes crash intermittently on agent payloads (Foundry-Local#506), but a fresh process restart often recovers. Wiping the user's explicit model choice on a transient hiccup is bad UX. Load failures (model fundamentally cannot load) still clear the preference; that's correct because the model is unrecoverable from the user's perspective. All 17 unit tests still pass. Co-authored-by: Copilot <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch fixes the long-standing
RUN_ERROR / ChatClientExceptionthat broke every chat through/api/command, plus a related telemetry pipeline bug discovered during verification, plus two minor production papercuts found along the way.Verified end-to-end through the UI: chat works, tasks created,
ai.response+flow+http.requestspans appear in the Stats panel with token counts and durations. Status: ready to merge.What broke and why
Three stacked Foundry Local + agent-framework bugs:
AG-UI auto-injects a
Current state of the application:system message on every turn that's >75k chars when there are 200+ todos in the dict. The 0.5B CPU model has a 32k context window. Foundry returned HTTP 500 →RemoteProtocolError→ChatClientException→RUN_ERROR. The wrapper now strips this message before sending to Foundry (model already has a compact view via_build_instructions()). Refs Python: [Bug]: FoundryLocalClient agent.run() fails with JSON serialization error on Foundry Local service microsoft/agent-framework#4084.Foundry Local streaming is unreliable inside long-running uvicorn processes —
stream=truerequests drop the connection mid-response. Wrapper translates AG-UI's forcedstream=Trueinto astream=FalseFoundry call, then synthesizes a singleChatCompletionChunkso the rest of the framework sees what it expects. Refs Streaming endpoint hangs onstream: truerequests (Critical) microsoft/foundry-local#422.init_otel()was not idempotent — uvicorn importsmain.pytwice, OpenTelemetry refuses to replace the activeTracerProvider, and the second call left_ui_span_exporterpointing at an orphaned exporter./api/telemetry/snapshotreturned0events even when chats completed successfully. Added an_otel_initializedguard.What's in this PR
app/agent/agent.pyapp/agent/main.pyapp/agent/agent.pyapp/agent/telemetry.pyenable_instrumentationlog line on startup)timezoneimport shadow fixapp/agent/app_config.pyGet-FileHashPS5.1 fallbackapp/launch.ps1-UseBasicParsing, WindowsApps filter, parses under PS5.1+PS7).github/copilot-instructions.mdSmoke test design
The post-switch smoke test (
main.py:_smoke_test_inference) sends the exact production payload — live_build_instructions()+ serializedALL_TOOLS— before declaring a model switch successful. NPU runtimes that crash on agent payloads (Foundry-Local#506, see below) get rolled back at switch time with a clear error rather than silently dying on the first chat.Known limitation: NPU + agent payloads
NPU runtimes (VitisAI, QNN) reliably crash Foundry on requests containing the full system prompt + 10-tool definition list. This is not a regression from this PR — it's microsoft/foundry-local#506, an upstream Foundry runtime bug, and is out of taskbean's control. Without this PR, NPU model switches succeeded and the first chat crashed Foundry →
ConnectionResetError→ uvicorn died. With this PR, the smoke test catches the crash at switch time and surfaces:The user can fall back to a working CPU/GPU model. CPU + GPU models are unaffected.
Tests
pytest test_phase0_fixes.py)test_telemetry.pywhich has pre-existing OGA SDK leak issues unrelated to this PR)npm testincli/)-UseBasicParsing, WindowsApps filter, parses under both PS 5.1 and PS 7)Pre-merge code review
Reviewed by code-review agent. Two blocking findings addressed in commit
fb8688a:agent-framework==1.0.1,agent-framework-openai==1.0.1,agent-framework-ag-ui==1.0.0b260409because the wrapper overrides private framework internals. Added an upgrade-checklist comment.ALL_TOOLSimport breaks) now logs aWARNINGand explicitly fails over when no tools serialize, instead of silently masking a critical bug.Also addressed one design concern: don't clear
preferredModelon smoke failure (only on load failure). Foundry NPU crashes can be transient — wiping a user's explicit choice on a hiccup is bad UX.References
stream: truerequests (Critical) microsoft/foundry-local#422 (streaming bug)