Skip to content

Splitting the mixers' projections with the Neural Engine #690

Open

Description

@fishy-ops

On an M4 Pro, the FFN split from #260 leaves the ANE waiting while the GPU runs the mixers' input and output projections, which are most of the rest of each layer. I extended the split to those projections, calibrated per Mac the same way as the FFN. At 14K and 32K tokens it is 1.62× and 1.53× faster than main on Qwen3.8-27B (MLX 4-bit). Decode is unchanged, and upstream's regression check passes. I'm opening a PR with the details and would appreciate any feedback on the approach.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions