Skip to content

Cache Exp in SoftMax calculation #111615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tannergooding merged 3 commits into from
Jan 21, 2025
Merged

Cache Exp in SoftMax calculation #111615

tannergooding merged 3 commits into from
Jan 21, 2025

Conversation

@BarionLP
Copy link
Contributor

@BarionLP BarionLP commented Jan 20, 2025

solves #109327 by computing Exp(x) once and caching it in destination instead of computing it twice

| Method  | Count   | Mean         | Error      | StdDev     | Ratio | Allocated | Alloc Ratio |
|-------- |-------- |-------------:|-----------:|-----------:|------:|----------:|------------:|
| BuiltIn | 1000    |     4.211 us |  0.0220 us |  0.0206 us |  1.00 |         - |          NA |
| Mine    | 1000    |     2.308 us |  0.0296 us |  0.0277 us |  0.55 |         - |          NA |
|         |         |              |            |            |       |           |             |
| BuiltIn | 1000000 | 4,200.558 us | 30.2029 us | 28.2518 us |  1.00 |         - |          NA |
| Mine    | 1000000 | 2,435.979 us | 21.0321 us | 19.6735 us |  0.58 |         - |          NA |

benchmark: https://gist.github.com/BarionLP/dd17244a59cd4470756e9219ceaeec49

@dotnet-policy-service dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jan 20, 2025
@dotnet-policy-service
Copy link
Contributor

dotnet-policy-service bot commented Jan 20, 2025

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

@BarionLP
Copy link
Contributor Author

BarionLP commented Jan 20, 2025

@dotnet-policy-service agree

@BarionLP BarionLP changed the title Cache Exp in Softmax calculation Cache Exp in SoftMax calculation Jan 20, 2025
@BarionLP BarionLP marked this pull request as draft January 20, 2025 13:14
This was referenced Jan 20, 2025
Open
Open
@BarionLP BarionLP marked this pull request as ready for review January 21, 2025 07:52
@BarionLP
Copy link
Contributor Author

BarionLP commented Jan 21, 2025

I think the fails are unrelated?

@tannergooding tannergooding merged commit b8c55f8 into dotnet:main Jan 21, 2025
82 checks passed
@BarionLP BarionLP deleted the barion-softmax-speed branch January 21, 2025 17:00
@github-actions github-actions bot locked and limited conversation to collaborators Feb 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@tannergooding tannergooding tannergooding approved these changes

+1 more reviewer

@lilinus lilinus lilinus left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area-System.Numerics.Tensors community-contribution Indicates that the PR has been added by a community member

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@BarionLP @tannergooding @lilinus