Record PR time benchmark results in JSON format #140493
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140493
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit d361a76 with merge base 4acd56e ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@pytorchbot drci |
|
@pytorchbot drci |
To ease the process of gathering the benchmark metadata before uploading the the database, I'm adding a script `.github/scripts/benchmarks/gather_metadata.py` to gather this information and pass it to the upload script. From #5839, the benchmark metadata includes the following required fields: ``` -- Metadata `timestamp` UInt64, `schema_version` String DEFAULT 'v3', `name` String, -- About the change `repo` String DEFAULT 'pytorch/pytorch', `head_branch` String, `head_sha` String, `workflow_id` UInt64, `run_attempt` UInt32, `job_id` UInt64, -- The raw records on S3 `s3_path` String, ``` I'm going to test this out with PT2 compiler instruction count benchmark at pytorch/pytorch#140493 ### Testing https://github.com/pytorch/test-infra/actions/runs/11831746632/job/32967412160?pr=5918#step:5:105 gathers the metadata and upload the benchmark results correctly Also, an actual upload at https://github.com/pytorch/pytorch/actions/runs/11831781500/job/33006545698#step:24:138
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
pytorchmergebot
force-pushed
the
upload-pr_time_benchmarks
branch
from
5216bd0 to
91eed3d
Compare
| opt_m(self.input) | ||
|
|
||
| def _write_to_json(self, output_dir: str): | ||
| records = [] |
There was a problem hiding this comment.
def properties() {
device = None
is_dynmao = None
type = None
backend = None
}
| device: str, | ||
| backend: str = "", | ||
| mode: str = "", | ||
| dynamic=False, |
There was a problem hiding this comment.
can we default it to None? since if user does not pass we do not know its not dynamic.
There was a problem hiding this comment.
Sure, let's default this and fullgraph to None
| backend: str = "", | ||
| mode: str = "", | ||
| dynamic=False, | ||
| fullgraph=False, |
| dynamic=False, | ||
| fullgraph=False, | ||
| ): | ||
| self._model_type = model_type |
There was a problem hiding this comment.
_model_type, mmhmm shall we rename it to catagory_name maybe
or group_name?
we do not always benchmark models it can be confusing.
There was a problem hiding this comment.
Yeah, i'm looking for a better name for this variable. category_name sounds like a good choice
| self._backend = backend | ||
| self._mode = mode # Training or inference | ||
| self._dynamic = dynamic | ||
| self._fullgraph = fullgraph |
There was a problem hiding this comment.
I do not think we need the _fullgraph? we do not have benchmarks that run with and without fullgraph modes.
i also do not think it would be interesting to filter on this
| self._force_shape_pad = force_shape_pad | ||
|
|
||
| super().__init__( | ||
| model_type=ModuleClass.__name__, |
There was a problem hiding this comment.
maybe just used "basic_modules" for the group_name?
| N = 100 | ||
|
|
||
| def __init__(self): | ||
| super().__init__(model_type="sum_floordiv", device="cpu") |
There was a problem hiding this comment.
i think backend=export for this?
|
@pytorchbot revert -m 'I think I missed something in the workflow setup as the test is failing in non-test CI jobs' -c weird |
|
@pytorchbot successfully started a revert job. Check the current status here. |
This reverts commit 783cd9c. Reverted #140493 on behalf of https://github.com/huydhn due to I think I missed something in the workflow setup as the test is failing in non-test CI jobs ([comment](#140493 (comment)))
|
@huydhn your PR has been successfully reverted. |
|
I need to get pytorch/test-infra#5946 review and land before I can reland this one. This is my miss in setting up the upload GHA |
It's ok if the benchmark results directory doesn't exists, for example, on non-benchmark job like `doc_test` https://github.com/pytorch/pytorch/actions/runs/11925173700/job/33237580806. I missed this after landing pytorch/pytorch#140493
|
@pytorchbot merge -f 'No need to run trunk jobs, pr_time_benchmark job has passed and pytorch/test-infra#5946 has landed' |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
I'm trying to make this benchmark results available on OSS benchmark database, so that people can query it from outside. The first step is to also record the results in the JSON format compatible with the database schema defined in pytorch/test-infra#5839. Existing CSV files remain unchanged. ### Testing The JSON results are uploaded as artifacts to S3 https://github.com/pytorch/pytorch/actions/runs/11809725848/job/32901411180#step:26:13, for example https://gha-artifacts.s3.amazonaws.com/pytorch/pytorch/11809725848/1/artifact/test-jsons-test-pr_time_benchmarks-1-1-linux.g4dn.metal.nvidia.gpu_32901411180.zip Pull Request resolved: pytorch#140493 Approved by: https://github.com/laithsakka
…)" This reverts commit 783cd9c. Reverted pytorch#140493 on behalf of https://github.com/huydhn due to I think I missed something in the workflow setup as the test is failing in non-test CI jobs ([comment](pytorch#140493 (comment)))
I'm trying to make this benchmark results available on OSS benchmark database, so that people can query it from outside. The first step is to also record the results in the JSON format compatible with the database schema defined in pytorch/test-infra#5839. Existing CSV files remain unchanged. ### Testing The JSON results are uploaded as artifacts to S3 https://github.com/pytorch/pytorch/actions/runs/11809725848/job/32901411180#step:26:13, for example https://gha-artifacts.s3.amazonaws.com/pytorch/pytorch/11809725848/1/artifact/test-jsons-test-pr_time_benchmarks-1-1-linux.g4dn.metal.nvidia.gpu_32901411180.zip Pull Request resolved: pytorch#140493 Approved by: https://github.com/laithsakka
3 participants
I'm trying to make this benchmark results available on OSS benchmark database, so that people can query it from outside. The first step is to also record the results in the JSON format compatible with the database schema defined in pytorch/test-infra#5839.
Existing CSV files remain unchanged.
Testing
The JSON results are uploaded as artifacts to S3 https://github.com/pytorch/pytorch/actions/runs/11809725848/job/32901411180#step:26:13, for example https://gha-artifacts.s3.amazonaws.com/pytorch/pytorch/11809725848/1/artifact/test-jsons-test-pr_time_benchmarks-1-1-linux.g4dn.metal.nvidia.gpu_32901411180.zip
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames