Skip to content

NPU models fail during Agent Framework fan‑out (parallel inference) while CPU models succeed #506

New issue
New issue
Closed
Closed
NPU models fail during Agent Framework fan‑out (parallel inference) while CPU models succeed#506
Labels
StaleNo activity for an extended period

Description

@leestott
Lee Stott (leestott)
opened on Mar 10, 2026

Issue reported via Foundry Discord https://aka.ms/foundry/discord
Foundry Local

Description
When using Foundry Local with the Microsoft Agent Framework, CPU models run successfully, but NPU‑backed models fail at the fan‑out stage when multiple parallel LLM calls are made.
This occurs even on systems with sufficient shared memory (32 GB). Single, serialized inference works on NPU; failures only appear when fan‑out / parallel orchestration is used.
This makes NPU models unusable today for multi‑agent or fan‑out workflows.

Repro steps
Install Foundry Local on a Copilot+ PC with NPU support (Windows 11 24H2)
Clone and run:
https://github.com/leestott/agentframework--foundrylocal
Configure the agent to use:

CPU model → ✅ succeeds
NPU model → ❌ fails during fan‑out

Observe failure when multiple concurrent LLM calls are triggered

Expected behavior
NPU models should either:
Support fan‑out / parallel inference, or
Fail gracefully with a clear error indicating that parallel execution is not supported

Actual behavior
CPU models complete successfully
NPU models fail at fan‑out / parallel execution stage
Failure appears related to NPU execution provider session / memory constraints rather than total system memory

Environment
Foundry Local: latest 0.8.119 (as of March 2026)
OS: Windows 11 24H2
Hardware: Copilot+ PC with NPU QNN ARM
Memory: 32 GB shared system memory
Agent Framework: used via agentframework--foundrylocal sample

Notes
NPU works correctly for single-agent, serialized inference
Issue only occurs with parallel / fan‑out agent execution
This appears to be a limitation of current NPU execution providers (session concurrency, buffer/KV cache duplication)

Clarifying whether this is a known limitation (and documenting it) would help developers avoid confusion when targeting NPUs with Agent Framework.

User asked to validate issue confirm using NPU model Qwen2.5-7b
Upload logs and Screenshot of demo and issue

Activity

  1. leestott commented on Mar 10, 2026

    @leestott
    Lee Stott (leestott)
    on Mar 10, 2026
    Author

    I ran the demo on a ARM Snapdragon machine

    NO ISSUE Found and demo runs successfully Validated on ARM64 Windows 11 AI PC

    Spec
    System Model: 21N10007UK
    System Type: ARM64-based PC
    Processor(s): 1 Processor(s) Installed.
    BIOS Version: LENOVO N42ET95W (2.25 ), 23/10/2025
    Total Physical Memory: 32,317 MB
    Windows Version 10.0.26200 Build 26200
    Version 25H2

    Foundry Local Version
    Foundry Local version 0.8.117

    Log file from /users/username/.foundry/logs

    .log file No error shown

    2026-03-10 19:16:23.235 +00:00 [INF] Loaded cached model info for 30 models. SavedAt:10/03/2026 19:05:07
    2026-03-10 19:16:23.255 +00:00 [INF] HandleChatCompletionRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:(null) maxTokens:(null) temperature:(null) topP:(null)
    2026-03-10 19:16:23.302 +00:00 [INF] AppendTokenSequences -> numTokensAppended:684
    2026-03-10 19:16:28.846 +00:00 [INF] Using new generator
    2026-03-10 19:16:48.961 +00:00 [INF] HandleChatCompletionRequest elapsed time:00:00:25.7042709
    2026-03-10 19:16:48.963 +00:00 [INF] UserAgent:agent-framework-python/0.0.0 Command:OpenAIChatCompletions Status:Success Direct:True Time:25741ms
    2026-03-10 19:16:50.556 +00:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:(null) maxTokens:(null) temperature:(null) topP:(null)
    2026-03-10 19:16:50.568 +00:00 [INF] Loaded cached model info for 30 models. SavedAt:10/03/2026 19:05:07
    2026-03-10 19:16:50.584 +00:00 [INF] HandleChatCompletionRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:(null) maxTokens:(null) temperature:(null) topP:(null)
    2026-03-10 19:16:50.605 +00:00 [INF] AppendTokenSequences -> numTokensAppended:853
    2026-03-10 19:16:58.036 +00:00 [INF] Using new generator
    2026-03-10 19:17:17.964 +00:00 [INF] HandleChatCompletionRequest elapsed time:00:00:27.3783167
    2026-03-10 19:17:17.964 +00:00 [INF] UserAgent:agent-framework-python/0.0.0 Command:OpenAIChatCompletions Status:Success Direct:True Time:27407ms
    2026-03-10 19:18:54.096 +00:00 [INF] Command:SessionInteractive Status:Success Direct:False Time:697270ms
    2026-03-10 19:18:54.096 +00:00 [INF] Command:ModelRun Status:Success Direct:True Time:719474ms

    Application and multi agent successfully executed NO ERRORS

    Image

  2. natke commented on Mar 10, 2026

    @natke
    Nat Kershaw (MSFT) (natke)
    on Mar 10, 2026
    Collaborator

    Can you please check your NPU driver version?

    • Open the Device Manager on your Windows system.
    • Look for "Neural Processors" in the list of devices.
    • Right-click on the NPU device and select "Properties".
    • In the Properties window, you will find the driver version listed under the "Driver Version" tab.

  3. leestott commented on Mar 10, 2026

    @leestott
    Lee Stott (leestott)
    on Mar 10, 2026
    Author

    Confirmed working with following driver

    To check your driver run the Powershell Command

    Get-WmiObject Win32_PnPSignedDriver | Where-Object { $_.DeviceName -match "Hexagon.*NPU" } | Select-Object DeviceName, DriverVersion, DriverDate, Manufacturer, InfName | Format-List

    My machine is running

    DeviceName : Snapdragon(R) X Elite - X1E78100 - Qualcomm(R) Hexagon(TM) NPU
    DriverVersion : 30.0.146.0
    DriverDate : 20250208000000.***+
    Manufacturer : Qualcomm Technologies, Inc.
    InfName : oem211.inf

  4. JasonB-IBCP commented on Mar 10, 2026

    @JasonB-IBCP
    JasonB-IBCP
    on Mar 10, 2026

    I'm running into a similar issue. I'll post the log tomorrow.
    Here's my driver version

    Get-WmiObject Win32_PnPSignedDriver | Where-Object { $_.DeviceName -match "Hexagon.*NPU" } | Select-Object DeviceName, DriverVersion, DriverDate, Manufacturer, InfName | Format-List

    DeviceName : Snapdragon(R) X Elite - X1E80100 - Qualcomm(R) Hexagon(TM) NPU
    DriverVersion : 30.0.31.180
    DriverDate : 20240902000000.***+
    Manufacturer : Qualcomm Technologies, Inc.
    InfName : oem349.inf

  5. natke commented on Mar 11, 2026

    @natke
    Nat Kershaw (MSFT) (natke)
    on Mar 11, 2026
    Collaborator

    Please update to the latest driver here: https://softwarecenter.qualcomm.com/catalog/item/Qualcomm_HND

  6. leestott commented on Mar 11, 2026

    @leestott
    Lee Stott (leestott)
    on Mar 11, 2026
    Author

    JasonB-IBCP please update your driver I have validated that this works successfully with
    DriverVersion : 30.0.146.0

  7. JasonB-IBCP commented on Mar 11, 2026

    @JasonB-IBCP
    JasonB-IBCP
    on Mar 11, 2026

    I found a driver pack for the surface last night on Microsoft's site. it brought me to

    DeviceName : Snapdragon(R) X Elite - X1E80100 - Qualcomm(R) Hexagon(TM) NPU
    DriverVersion : 30.0.219.1000
    DriverDate : 20251109000000.***+
    Manufacturer : Qualcomm Technologies, Inc.
    InfName : oem424.inf

    I'll test with it and a NPU model this morning

  8. JasonB-IBCP commented on Mar 11, 2026

    @JasonB-IBCP
    JasonB-IBCP
    on Mar 11, 2026

    11:10:50 app.foundry_boot INFO Bootstrapping Foundry Local with alias 'qwen2.5-7b' …
    11:10:52 app.foundry_boot INFO ✅ Foundry Local ready → endpoint=http://127.0.0.1:59543/v1 model=qwen2.5-7b-instruct-qnn-npu:2
    Model : qwen2.5-7b-instruct-qnn-npu:2 (alias: qwen2.5-7b)
    Endpoint: http://127.0.0.1:59543/v1

    Log
    2026-03-11 11:07:47.760 -04:00 [INF] 🟢 Model qwen2.5-7b-instruct-qnn-npu:2 loaded successfully
    2026-03-11 11:07:47.760 -04:00 [INF] Command:ModelLoad Status:Success Direct:False Time:9497ms
    2026-03-11 11:07:47.761 -04:00 [INF] Accepting pipe incoming pipeName:inference_agent numOfSession:13
    2026-03-11 11:08:06.261 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
    2026-03-11 11:08:06.265 -04:00 [INF] Loaded cached model info for 30 models. SavedAt:3/10/2026 4:40:18 PM
    2026-03-11 11:08:06.273 -04:00 [INF] UserAgent:FoundryLocal OpenAI/2.2.0-beta.4 (.NET 9.0.8; Microsoft Windows 10.0.26100) Command:OpenAIChatCompletions Status:Success Direct:True Time:10ms
    2026-03-11 11:08:06.273 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
    2026-03-11 11:08:06.305 -04:00 [INF] AppendTokenSequences -> numTokensAppended:39
    2026-03-11 11:08:06.549 -04:00 [INF] Using new generator
    2026-03-11 11:08:15.716 -04:00 [INF] TTFT: 0 ms, TTST: 83
    2026-03-11 11:08:15.716 -04:00 [INF] Completion stats: Total Tokens/Second: 12.3, Time to First Token: 276 ms, Average Token Generation Time: 80.1 ms, Total Time: 9444 ms, Total Tokens: 116
    2026-03-11 11:08:15.717 -04:00 [INF] Ran qwen2.5-7b-instruct-qnn-npu:2 on QNNExecutionProvider, TimeToFirstTokenMs:276, TotalTimeMs:9444, TotalTokens:116, Turns:1
    2026-03-11 11:08:37.188 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
    2026-03-11 11:08:37.193 -04:00 [INF] Loaded cached model info for 30 models. SavedAt:3/10/2026 4:40:18 PM
    2026-03-11 11:08:37.199 -04:00 [INF] UserAgent:FoundryLocal OpenAI/2.2.0-beta.4 (.NET 9.0.8; Microsoft Windows 10.0.26100) Command:OpenAIChatCompletions Status:Success Direct:True Time:11ms
    2026-03-11 11:08:37.199 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
    2026-03-11 11:08:37.240 -04:00 [INF] AppendTokenSequences -> numTokensAppended:175
    2026-03-11 11:08:37.942 -04:00 [INF] Using new generator
    2026-03-11 11:09:19.326 -04:00 [INF] TTFT: 1 ms, TTST: 83
    2026-03-11 11:09:19.326 -04:00 [INF] Completion stats: Total Tokens/Second: 11.6, Time to First Token: 740 ms, Average Token Generation Time: 85.2 ms, Total Time: 42125 ms, Total Tokens: 490
    2026-03-11 11:09:19.328 -04:00 [INF] Ran qwen2.5-7b-instruct-qnn-npu:2 on QNNExecutionProvider, TimeToFirstTokenMs:740, TotalTimeMs:42125, TotalTokens:490, Turns:2
    2026-03-11 11:10:33.142 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
    2026-03-11 11:10:33.147 -04:00 [INF] Loaded cached model info for 30 models. SavedAt:3/10/2026 4:40:18 PM
    2026-03-11 11:10:33.153 -04:00 [INF] UserAgent:FoundryLocal OpenAI/2.2.0-beta.4 (.NET 9.0.8; Microsoft Windows 10.0.26100) Command:OpenAIChatCompletions Status:Success Direct:True Time:11ms
    2026-03-11 11:10:33.153 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
    2026-03-11 11:10:33.207 -04:00 [INF] AppendTokenSequences -> numTokensAppended:676
    2026-03-11 11:10:35.891 -04:00 [INF] Using new generator
    2026-03-11 11:10:38.147 -04:00 [INF] TTFT: 1 ms, TTST: 90
    2026-03-11 11:10:38.147 -04:00 [INF] Completion stats: Total Tokens/Second: 5.2, Time to First Token: 2738 ms, Average Token Generation Time: 188 ms, Total Time: 4992 ms, Total Tokens: 26
    2026-03-11 11:10:38.148 -04:00 [INF] Ran qwen2.5-7b-instruct-qnn-npu:2 on QNNExecutionProvider, TimeToFirstTokenMs:2738, TotalTimeMs:4992, TotalTokens:26, Turns:3
    2026-03-11 11:10:39.859 -04:00 [INF] Command:SessionInteractive Status:Success Direct:False Time:172098ms
    2026-03-11 11:10:39.859 -04:00 [INF] Command:ModelRun Status:Success Direct:True Time:182152ms
    2026-03-11 11:10:39.859 -04:00 [INF] Stream disconnected
    2026-03-11 11:10:50.307 -04:00 [INF] Starting Foundry Local CLI with 'service status'
    2026-03-11 11:10:50.318 -04:00 [INF] Found service endpoints: http://127.0.0.1:59543
    2026-03-11 11:10:50.338 -04:00 [INF] UserAgent:foundry-local-CLI/0.8.119 Command:ServiceStatus Status:Success Direct:True Time:0ms
    2026-03-11 11:10:50.339 -04:00 [INF] Command:ServiceStatus Status:Success Direct:True Time:27ms
    2026-03-11 11:10:50.339 -04:00 [INF] Stream disconnected
    2026-03-11 11:10:50.935 -04:00 [INF] Starting Foundry Local CLI with 'service status'
    2026-03-11 11:10:50.950 -04:00 [INF] Found service endpoints: http://127.0.0.1:59543
    2026-03-11 11:10:50.969 -04:00 [INF] Command:ServiceStatus Status:Success Direct:True Time:28ms
    (.venv) PS C:\Users\JBenway.foundry\logs>

  9. JasonB-IBCP commented on Mar 11, 2026

    @JasonB-IBCP
    JasonB-IBCP
    on Mar 11, 2026

    Foundry version: 0.8.119

    System Model: Microsoft Surface Laptop, 7th Edition
    System Type: ARM64-based PC
    Processor(s): 1 Processor(s) Installed. [01]: ARMv8 (64-bit) Family 8 Model 1 Revision 201 Qualcomm Technologies Inc ~1670 Mhz
    BIOS Version: Microsoft Corporation 175.138.235, 8/19/2025
    Total Physical Memory: 65,017 MB
    Windows Version 10.0.26100 N/A Build 26100
    Version 24H2

  10. natke commented on Mar 11, 2026

    @natke
    Nat Kershaw (MSFT) (natke)
    on Mar 11, 2026
    Collaborator

    Is that working for you now? JasonB-IBCP

  11. JasonB-IBCP commented on Mar 11, 2026

    @JasonB-IBCP
    JasonB-IBCP
    on Mar 11, 2026

    The fan-out is still erroring.

    13:36:00 app.foundry_boot INFO Bootstrapping Foundry Local with alias 'qwen2.5-7b' …
    13:36:10 app.foundry_boot INFO ✅ Foundry Local ready → endpoint=http://127.0.0.1:59543/v1 model=qwen2.5-7b-instruct-qnn-npu:2
    Model : qwen2.5-7b-instruct-qnn-npu:2 (alias: qwen2.5-7b)

    Nat Kershaw (MSFT) (@natke)

  12. natke commented on Mar 12, 2026

    @natke
    Nat Kershaw (MSFT) (natke)
    on Mar 12, 2026
    Collaborator

    What is the error? Can you set foundry service set --default-log-level Debug and run again please?

  13. leestott commented on Mar 12, 2026

    @leestott
    Lee Stott (leestott)
    on Mar 12, 2026
    Author

    JasonB-IBCP

    So your driver is still an old version, so I think the Issues is with your Qualcomm Snapdragon
    Fix to try: Install this latets driver https://softwarecenter.qualcomm.com/catalog/item/QHND (you will need a Qualcomm Developer Account to access the download)

  14. natke commented on Mar 12, 2026

    @natke
    Nat Kershaw (MSFT) (natke)
    on Mar 12, 2026
    Collaborator

    Lee Stott (@leestott) what is failing here and how do we observe the failure?

  15. Nick Brady (nicholasdbrady)
    added a commit that references this issue on Apr 26, 2026

    fix: Foundry Local agent reliability + telemetry pipeline (#28)

    df12d55

  16. samuel100 commented on May 15, 2026

    @samuel100
    Samuel Kemp (samuel100)
    on May 15, 2026
    Contributor

    This issue has been automatically closed due to extended inactivity. If this is still a problem, please re-open the issue or file a new one with updated details. Thank you for your contributions!

  17. Samuel Kemp (samuel100)
    closed this as completedon May 15, 2026
  18. Samuel Kemp (samuel100)
    added
    StaleNo activity for an extended period
    on May 15, 2026

  19. samuel100 commented on May 15, 2026

    @samuel100
    Samuel Kemp (samuel100)
    on May 15, 2026
    Contributor

    Re-opening — this issue was closed prematurely as part of a stale-issue sweep. Apologies for the noise.

  20. Samuel Kemp (samuel100)
    reopened this on May 15, 2026
  21. Samuel Kemp (samuel100)
    removed
    StaleNo activity for an extended period
    on May 15, 2026
  22. Samuel Kemp (samuel100)
    added
    StaleNo activity for an extended period
    on Jun 6, 2026

  23. samuel100 commented on Jun 6, 2026

    @samuel100
    Samuel Kemp (samuel100)
    on Jun 6, 2026
    Contributor

    Closing as stale - from the thread it looks like an outdated driver.

  24. Samuel Kemp (samuel100)
    closed this as completedon Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    StaleNo activity for an extended period

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions