Skip to content

PyTorch Data Sampler benchmark #156974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
divyanshk wants to merge 6 commits into from
Closed

PyTorch Data Sampler benchmark #156974

divyanshk wants to merge 6 commits into from

Conversation

@divyanshk
Copy link
Contributor

@divyanshk divyanshk commented Jun 26, 2025

Motivation

Many PRs optimizing samplers (for eg #147706, #137423) are leveraging an adhoc script for benchmarking samplers. The script and outputs are often copied over in PRs. We want to begin centralizing benchmarks for torch.utils.data components.

What ?

  • This PR adds a new sub-folder in benchmarks for data. This is aimed to cover benchmarking scripts for torch.utils.data components like dataloader and sampler.
  • Specifically, this PR includes a simple script to time samplers. This is often "copy-pasted" in PRs optimizing samplers. Having it in a centralized location should prevent that, and allow a common standard.

Output

Benchmark Results:
+--------------+-------------+----------------+-----------+-----------+
|   Batch Size | Drop Last   |   Original (s) |   New (s) | Speedup   |
+==============+=============+================+===========+===========+
|            4 | True        |         0.004  |    0.0088 | -119.62%  |
+--------------+-------------+----------------+-----------+-----------+
|            4 | False       |         0.0083 |    0.009  | -9.23%    |
+--------------+-------------+----------------+-----------+-----------+
|            8 | True        |         0.003  |    0.0074 | -147.64%  |
+--------------+-------------+----------------+-----------+-----------+
|            8 | False       |         0.0054 |    0.0075 | -38.72%   |
+--------------+-------------+----------------+-----------+-----------+
|           64 | True        |         0.0021 |    0.0056 | -161.92%  |
+--------------+-------------+----------------+-----------+-----------+
|           64 | False       |         0.0029 |    0.0055 | -92.50%   |
+--------------+-------------+----------------+-----------+-----------+
|          640 | True        |         0.002  |    0.0055 | -168.75%  |
+--------------+-------------+----------------+-----------+-----------+
|          640 | False       |         0.0024 |    0.0062 | -161.35%  |
+--------------+-------------+----------------+-----------+-----------+
|         6400 | True        |         0.0021 |    0.0055 | -160.13%  |
+--------------+-------------+----------------+-----------+-----------+
|         6400 | False       |         0.0021 |    0.0068 | -215.46%  |
+--------------+-------------+----------------+-----------+-----------+
|        64000 | True        |         0.0042 |    0.0065 | -55.29%   |
+--------------+-------------+----------------+-----------+-----------+
|        64000 | False       |         0.0029 |    0.0077 | -169.56%  |
+--------------+-------------+----------------+-----------+-----------+

@pytorch-bot
Copy link

pytorch-bot bot commented Jun 26, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156974

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

⏳ 1 Pending, 2 Unrelated Failures

As of commit a36401b with merge base d061a02 (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jun 26, 2025 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

@pytorch-bot pytorch-bot bot added the release notes: dataloader release notes category label Jun 26, 2025
@divyanshk
Copy link
Contributor Author

divyanshk commented Jun 26, 2025

/easycla

@divyanshk
Copy link
Contributor Author

divyanshk commented Jun 26, 2025

/easycla :-)

@divyanshk divyanshk marked this pull request as ready for review June 26, 2025 23:03
@divyanshk divyanshk requested a review from ramanishsingh as a code owner June 26, 2025 23:03
@divyanshk divyanshk requested a review from scotts June 26, 2025 23:03
@divyanshk
Copy link
Contributor Author

divyanshk commented Jun 26, 2025

The CI error seem to be fixed in a recent PR#157010.

@ramanishsingh
Copy link

ramanishsingh commented Jun 26, 2025

Thanks for adding this benchmarking script. :)

@divyanshk
Copy link
Contributor Author

divyanshk commented Jun 26, 2025

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 26, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jun 26, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ramanishsingh ramanishsingh ramanishsingh approved these changes

@scotts scotts Awaiting requested review from scotts

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: dataloader release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@divyanshk @ramanishsingh @pytorchmergebot