Implementation of TRT wrapping via inference.json by borisfom · Pull Request #620 · Project-MONAI/model-zoo

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default
python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

binliunls · 2024-09-06T14:57:27Z

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

Hi @borisfom ,
Thank you for your update. As shown in the inference config file, you should compile the diffusion and autoencoder with TRT instead of network, since the network parameter is not defined in either bundle. Please check if you can compile both diffusion and autoencoder.

Thanks,
Bin

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

borisfom · 2024-09-06T18:03:06Z

@binliunls : Thanks, you are correct! Works as expected now.

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv · 2024-09-09T05:43:56Z

/build

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv · 2024-09-09T06:52:06Z

/build

Signed-off-by: binliu <[email protected]>

yiheng-wang-nv · 2024-09-10T03:25:32Z

/build

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv · 2024-09-10T06:14:55Z

/build

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv · 2024-09-10T06:26:13Z

/build

yiheng-wang-nv · 2024-09-10T06:27:52Z

/build

POC implementation of TRT wrapping via inference.json

85efcc3

Signed-off-by: Boris Fomitchev <[email protected]>

borisfom mentioned this pull request Aug 16, 2024

Proposal: TRT Acceleration API improvement Project-MONAI/MONAI#8018

Open

borisfom and others added 3 commits August 16, 2024 11:54

Bumped bundle version

98a615c

Signed-off-by: Boris Fomitchev <[email protected]>

Adding pathology_nuclei_classification and swin_unetr_btcv_segmentation

d3275e9

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

18c580f

for more information, see https://pre-commit.ci

Nic-Ma reviewed Aug 21, 2024

View reviewed changes

models/pathology_nuclei_classification/configs/inference_trt.json Show resolved Hide resolved

Nic-Ma reviewed Aug 21, 2024

View reviewed changes

models/brats_mri_generative_diffusion/configs/inference.json Outdated Show resolved Hide resolved

Nic-Ma reviewed Aug 21, 2024

View reviewed changes

models/brats_mri_generative_diffusion/configs/inference.json Outdated Show resolved Hide resolved

borisfom and others added 5 commits August 21, 2024 19:11

Adjusting arguments

936103d

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…

58d76de

…wrapper Signed-off-by: Boris Fomitchev <[email protected]>

Merge remote-tracking branch 'origin/dev' into trt_wrapper

9e4bac4

Signed-off-by: Boris Fomitchev <[email protected]>

Added vista3 inference_trt.json

8461033

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8588674

for more information, see https://pre-commit.ci

yiheng-wang-nv reviewed Aug 23, 2024

View reviewed changes

models/vista3d/configs/inference_trt.json Outdated Show resolved Hide resolved

borisfom and others added 12 commits August 22, 2024 22:34

Renamed trt_wrap -> trt_compile

f7da0ec

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…

c8d64bf

…wrapper Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

aab37c5

for more information, see https://pre-commit.ci

Added TRT for pathology_nuclei_segmentation_classification

7b64083

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…

43da1f5

…wrapper

Merge remote-tracking branch 'origin/dev' into trt_wrapper

3d8a738

[pre-commit.ci] auto fixes from pre-commit.com hooks

4a9f2ba

for more information, see https://pre-commit.ci

Merge remote-tracking branch 'origin/dev' into trt_wrapper

7496dfb

Added Vista2D TRT config

337fcf3

Signed-off-by: Boris Fomitchev <[email protected]>

Cleaned up using new config syntax, batch size added

16f720f

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…

6cc423d

…wrapper Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

a205bf4

for more information, see https://pre-commit.ci

borisfom requested review from Nic-Ma and yiheng-wang-nv August 27, 2024 00:29

borisfom changed the title ~~POC implementation of TRT wrapping via inference.json~~ Implementation of TRT wrapping via inference.json Aug 27, 2024

borisfom and others added 4 commits September 5, 2024 17:11

Merge remote-tracking branch 'origin/dev' into trt_wrapper

a099cd5

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

f6507c8

for more information, see https://pre-commit.ci

Fixed models/brats_mri_axial_slices_generative_diffusion

c6744e9

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…

1b634d2

…wrapper

borisfom mentioned this pull request Sep 6, 2024

Benchmark-trt-compiler #640

Closed

8 tasks

borisfom and others added 3 commits September 6, 2024 09:50

Merge remote-tracking branch 'origin/dev' into trt_wrapper

956d931

Fixing TRT configs for brats_mri_xx

affbf12

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

ee06c91

for more information, see https://pre-commit.ci

update metadata

f0a9329

Signed-off-by: Yiheng Wang <[email protected]>

update mgpu test script

bde5d52

Signed-off-by: Yiheng Wang <[email protected]>

binliunls added 5 commits September 9, 2024 16:08

add benchmark for non generative models

c7c4b50

Signed-off-by: binliu <[email protected]>

add benchmark results for generative bundles

626ba3c

Signed-off-by: binliu <[email protected]>

add version for trt benchmark

7f47fb4

Signed-off-by: binliu <[email protected]>

update vista2d version

5d58414

Signed-off-by: binliu <[email protected]>

update format

19ba83c

Signed-off-by: binliu <[email protected]>

use nvidia host large files

3e80c93

Signed-off-by: Yiheng Wang <[email protected]>

update vista2d

35e697c

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv mentioned this pull request Sep 10, 2024

Update large_files.yml for bundles #642

Closed

yiheng-wang-nv enabled auto-merge (squash) September 10, 2024 06:28

yiheng-wang-nv approved these changes Sep 10, 2024

View reviewed changes

yiheng-wang-nv merged commit efadc57 into Project-MONAI:dev Sep 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of TRT wrapping via inference.json#620

Implementation of TRT wrapping via inference.json#620
yiheng-wang-nv merged 38 commits intoProject-MONAI:devfrom
borisfom:trt_wrapper

borisfom commented Aug 16, 2024

Uh oh!

borisfom commented Aug 16, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borisfom commented Sep 6, 2024

Uh oh!

binliunls commented Sep 6, 2024

Uh oh!

borisfom commented Sep 6, 2024

Uh oh!

yiheng-wang-nv commented Sep 9, 2024

Uh oh!

yiheng-wang-nv commented Sep 9, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

borisfom commented Aug 16, 2024

Uh oh!

borisfom commented Aug 16, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borisfom commented Sep 6, 2024

Uh oh!

binliunls commented Sep 6, 2024

Uh oh!

borisfom commented Sep 6, 2024

Uh oh!

yiheng-wang-nv commented Sep 9, 2024

Uh oh!

yiheng-wang-nv commented Sep 9, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

yiheng-wang-nv commented Sep 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants