Skip to content

Update base image to 2410#8164

Merged
KumoLiu merged 5 commits intoProject-MONAI:devfrom
KumoLiu:update-base-image
Nov 11, 2024
Merged

Update base image to 2410#8164
KumoLiu merged 5 commits intoProject-MONAI:devfrom
KumoLiu:update-base-image

Conversation

@KumoLiu
Copy link
Copy Markdown
Contributor

@KumoLiu KumoLiu commented Oct 22, 2024

Fixes # .

Description

A few sentences describing the changes proposed in this pull request.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@KumoLiu
Copy link
Copy Markdown
Contributor Author

KumoLiu commented Oct 22, 2024

Hi @borisfom, could you please take a look at this KeyError that occurred when I upgraded the base image to version 24.09?


[2024-10-22T07:05:01.446Z] ======================================================================

[2024-10-22T07:05:01.446Z] ERROR: test_handler (tests.test_trt_compile.TestTRTCompile)

[2024-10-22T07:05:01.446Z] ----------------------------------------------------------------------

[2024-10-22T07:05:01.446Z] Traceback (most recent call last):

[2024-10-22T07:05:01.446Z]   File "/tmp/tmp.xs4H7I9CC3/tests/test_trt_compile.py", line 66, in test_handler

[2024-10-22T07:05:01.446Z]     net1.forward(torch.tensor([[0.0, 1.0], [1.0, 2.0]], device="cuda"))

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 496, in trt_forward

[2024-10-22T07:05:01.446Z]     return self._trt_compiler.forward(self, argv, kwargs)

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 351, in forward

[2024-10-22T07:05:01.446Z]     raise e

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 342, in forward

[2024-10-22T07:05:01.446Z]     self._build_and_save(model, build_args)

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 447, in _build_and_save

[2024-10-22T07:05:01.446Z]     engine_bytes = torch_tensorrt.convert_method_to_trt_engine(

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 355, in convert_method_to_trt_engine

[2024-10-22T07:05:01.446Z]     serialized_engine: bytes = ts_convert_method_to_trt_engine(

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compiler.py", line 270, in convert_method_to_trt_engine

[2024-10-22T07:05:01.446Z]     module._c, method_name, _parse_compile_spec(compile_spec)

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compile_spec.py", line 225, in _parse_compile_spec

[2024-10-22T07:05:01.446Z]     elif compile_spec["input_signature"] is not None:

[2024-10-22T07:05:01.446Z] KeyError: 'input_signature'

@KumoLiu KumoLiu mentioned this pull request Oct 22, 2024
Merged
1 task
@borisfom
Copy link
Copy Markdown
Contributor

borisfom commented Oct 22, 2024

@KumoLiu : Unfortunately, this looks like internal Torch-TensorRT error I don't see an easy workaround for. Please just disable the failing test (test_handler) for now.

@KumoLiu
Copy link
Copy Markdown
Contributor Author

KumoLiu commented Oct 22, 2024

@KumoLiu : Unfortunately, this looks like internal Torch-TensorRT error I don't see an easy workaround for. Please just disable the failing test (test_handler) for now.

Yes, the error originates from the internal code in Torch. However, I'm wondering if we need to update the base image. As you mentioned here, if not, we can simply wait for the 24.10 release. Thanks.

@KumoLiu
Copy link
Copy Markdown
Contributor Author

KumoLiu commented Oct 23, 2024

[2024-10-22T07:05:01.446Z] KeyError: 'input_signature'

report a bug here: pytorch/pytorch#138674

@borisfom
Copy link
Copy Markdown
Contributor

borisfom commented Oct 23, 2024

@KumoLiu : yes we should just wait until 24.10 release

KumoLiu added a commit to KumoLiu/MONAI that referenced this pull request Oct 23, 2024
@KumoLiu
workaround for Project-MONAI#8164
fdf16eb 
Signed-off-by: YunLiu <[email protected]>
@KumoLiu KumoLiu marked this pull request as draft October 23, 2024 08:35
borisfom
borisfom approved these changes Nov 7, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@borisfom borisfom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The title should say '2410' though.

@KumoLiu KumoLiu changed the title Update base image to 2409 Update base image to 2410 Nov 8, 2024
@KumoLiu
Copy link
Copy Markdown
Contributor Author

KumoLiu commented Nov 8, 2024

/build

@KumoLiu
skip
a47e988 
Signed-off-by: YunLiu <[email protected]>
@KumoLiu
Copy link
Copy Markdown
Contributor Author

KumoLiu commented Nov 8, 2024

/build

@KumoLiu KumoLiu marked this pull request as ready for review November 8, 2024 11:23
@KumoLiu KumoLiu merged commit 0bb20a8 into Project-MONAI:dev Nov 11, 2024
@KumoLiu KumoLiu deleted the update-base-image branch November 11, 2024 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@binliunls binliunls binliunls approved these changes

@ericspod ericspod Awaiting requested review from ericspod ericspod is a code owner

@Nic-Ma Nic-Ma Awaiting requested review from Nic-Ma Nic-Ma is a code owner

+1 more reviewer

@borisfom borisfom borisfom approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@KumoLiu @borisfom @binliunls