TF-TRT Backward compatibility test by pooyadavoodi · Pull Request #34660 · tensorflow/tensorflow

pooyadavoodi · 2019-11-27T22:55:15Z

Adds a new test to ensure TF-TRT can execute models saved in previous minor releases of TF2.x.
The checked in model is generated using tensorflow-gpu==2.0.0

This model can be loaded and executed in tests to ensure backwards compatibility across TF versions.

rthadur · 2019-12-04T17:34:40Z

@pooyadavoodi can you please check build failures ?

pooyadavoodi · 2019-12-04T18:18:53Z

I just fixed a few bugs.

Any idea about these errors which seem to be related to Eigen?

WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found 
WARNING: Download from https://bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was d8469c8be9ded39e914a7cd3c1246bae36ab0737d9c173328303766a3c7187e2 but wanted 9edd4860b52813eaf8c023f0de1767ec58e2d67a290b718e6702469208ac5be1
ERROR: An error occurred during the fetch of repository 'eigen_archive':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz, https://bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz] to /tmpfs/bazel_output/_bazel_kbuilder/f2d52ca1f092ccbe254cc98c3dc90790/external/eigen_archive/54bca9936424.tar.gz: Checksum was d8469c8be9ded39e914a7cd3c1246bae36ab0737d9c173328303766a3c7187e2 but wanted 9edd4860b52813eaf8c023f0de1767ec58e2d67a290b718e6702469208ac5be1
Analyzing: 1471 targets (431 packages loaded, 3071 targets configured)
Analyzing: 1471 targets (440 packages loaded, 3250 targets configured)
ERROR: /tmpfs/src/github/tensorflow/third_party/eigen3/BUILD:33:1: //third_party/eigen3:eigen3 depends on @eigen_archive//:eigen in repository @eigen_archive which failed to fetch. no such package '@eigen_archive//': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz, https://bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz] to /tmpfs/bazel_output/_bazel_kbuilder/f2d52ca1f092ccbe254cc98c3dc90790/external/eigen_archive/54bca9936424.tar.gz: Checksum was d8469c8be9ded39e914a7cd3c1246bae36ab0737d9c173328303766a3c7187e2 but wanted 9edd4860b52813eaf8c023f0de1767ec58e2d67a290b718e6702469208ac5be1

pooyadavoodi · 2019-12-04T19:56:07Z

I just fixed a few bugs.

Any idea about these errors which seem to be related to Eigen?

WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found 
WARNING: Download from https://bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was d8469c8be9ded39e914a7cd3c1246bae36ab0737d9c173328303766a3c7187e2 but wanted 9edd4860b52813eaf8c023f0de1767ec58e2d67a290b718e6702469208ac5be1
ERROR: An error occurred during the fetch of repository 'eigen_archive':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz, https://bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz] to /tmpfs/bazel_output/_bazel_kbuilder/f2d52ca1f092ccbe254cc98c3dc90790/external/eigen_archive/54bca9936424.tar.gz: Checksum was d8469c8be9ded39e914a7cd3c1246bae36ab0737d9c173328303766a3c7187e2 but wanted 9edd4860b52813eaf8c023f0de1767ec58e2d67a290b718e6702469208ac5be1
Analyzing: 1471 targets (431 packages loaded, 3071 targets configured)
Analyzing: 1471 targets (440 packages loaded, 3250 targets configured)
ERROR: /tmpfs/src/github/tensorflow/third_party/eigen3/BUILD:33:1: //third_party/eigen3:eigen3 depends on @eigen_archive//:eigen in repository @eigen_archive which failed to fetch. no such package '@eigen_archive//': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz, https://bitbucket.org/eigen/eigen/get/54bca9936424.tar.gz] to /tmpfs/bazel_output/_bazel_kbuilder/f2d52ca1f092ccbe254cc98c3dc90790/external/eigen_archive/54bca9936424.tar.gz: Checksum was d8469c8be9ded39e914a7cd3c1246bae36ab0737d9c173328303766a3c7187e2 but wanted 9edd4860b52813eaf8c023f0de1767ec58e2d67a290b718e6702469208ac5be1

That error is hopefully fixed by this: e3a7bdb

aaroey · 2019-12-05T22:02:48Z

Also @sanjoy

aaroey · 2019-12-06T20:29:48Z

+from __future__ import division
+from __future__ import print_function
+
+import tensorflow as tf


There is an error when trying to submit this PR:
'import tensorflow' statements are no allowed inside TF's code base... Please import modules directly.

Also could you help to add a build rule for this file?

This file is not supposed to be used by any test. It's supposed to be used manually by developers to generate a test model if needed. Do we still need to add it to BUILD?

I think adding a build rule will make our life easier: we just need to run bazel run to get the model. But I don't have strong opinion. But at least replace import tensorflow with direct module imports.

I removed the import.

I also noticed the test model was missing in the data dependency of trt_convert_test which is fixed now. I missed this because I run tests without bazel :(

I haven't added gen_tftrt_model.py to BUILD. I would need to learn how to do that first. I suggest to postpone this for now.

Also remove gen_tftrt_model.py from BUILD because this file is not supposed to be used by any test. It's supposed to be used manually by developers to generate a test model if needed.

aaroey

Thanks for the fix Pooya!

pooyadavoodi · 2019-12-06T23:06:11Z

Getting this now:

Missing the following dependencies from pip_packages:

Missing dependency: //tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/variables/variables.index
Affected Tests:
//tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/variables/variables.index (null)
//tensorflow/python/compiler/tensorrt:trt_convert_test (70635f4b39bbe1d08d95efe3e379ef53)

Missing dependency: //tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/variables/variables.data-00000-of-00002
Affected Tests:
//tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/variables/variables.data-00000-of-00002 (null)
//tensorflow/python/compiler/tensorrt:trt_convert_test (70635f4b39bbe1d08d95efe3e379ef53)

Missing dependency: //tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/variables/variables.data-00001-of-00002
Affected Tests:
//tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/variables/variables.data-00001-of-00002 (null)
//tensorflow/python/compiler/tensorrt:trt_convert_test (70635f4b39bbe1d08d95efe3e379ef53)

Missing dependency: //tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/saved_model.pb
Affected Tests:
//tensorflow/python/compiler/tensorrt:test/testdata/tftrt_2.0_saved_model/saved_model.pb (null)
//tensorflow/python/compiler/tensorrt:trt_convert_test (70635f4b39bbe1d08d95efe3e379ef53)
Traceback (most recent call last):
  File "pip_smoke_test.py", line 184, in <module>
    main()
  File "pip_smoke_test.py", line 177, in main
    or add no_pip tag to the test.""")
RuntimeError:
    One or more added test dependencies are not in the pip package.
If these test dependencies need to be in TensorFlow pip package, please add them to //tensorflow/tools/pip_package/BUILD.
Else either blacklist the dependencies in //tensorflow/tools/pip_package/pip_smoke_test.py
or add no_pip tag to the test.

What do you suggest?

…model.

aaroey

Unapprove and try to trigger the copybara import again.

PiperOrigin-RevId: 285002333 Change-Id: I698a30181a97fb48cf3fadffefb1a7a4e046d6d4

Pooya Davoodi added 4 commits November 27, 2019 14:52

Add script to generate TF-TRT model to be used for testing

a26b560

Add tftrt_2.0_saved_model

f1129b3

This model can be loaded and executed in tests to ensure backwards compatibility across TF versions.

Add backward compatibility test

a55e9c2

Move data of quantization_mnist_test to testdata/mnist

ebde7cc

pooyadavoodi requested a review from aaroey November 27, 2019 22:55

tensorflow-bot Bot added the size:M CL Change Size: Medium label Nov 27, 2019

googlebot added the cla: yes label Nov 27, 2019

pooyadavoodi changed the title ~~TF-TRT Backward compatibility test~~ WIP: TF-TRT Backward compatibility test Nov 27, 2019

rthadur self-assigned this Nov 28, 2019

aaroey requested changes Dec 3, 2019

View reviewed changes

Comment thread tensorflow/python/compiler/tensorrt/test/quantization_mnist_test.py Resolved

Comment thread tensorflow/python/compiler/tensorrt/trt_convert_test.py Resolved

Pooya Davoodi added 2 commits December 3, 2019 16:31

Add mnist dir, add correctness check for bwd compatibility test

3ac3685

Fix pylint for backwards compatibility test

6ab11cb

pooyadavoodi changed the title ~~WIP: TF-TRT Backward compatibility test~~ TF-TRT Backward compatibility test Dec 4, 2019

rthadur requested a review from aaroey December 4, 2019 03:07

aaroey previously approved these changes Dec 4, 2019

View reviewed changes

tensorflow-bot Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Dec 4, 2019

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 4, 2019

rthadur removed the ready to pull PR ready for merge process label Dec 4, 2019

Sort targets in BUILD and shorten statement for pylint

8105a6e

pooyadavoodi dismissed aaroey’s stale review via 8105a6e December 4, 2019 18:07

rthadur added the kokoro:force-run Tests on submitted change label Dec 5, 2019

rthadur requested a review from aaroey December 5, 2019 13:50

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 5, 2019

aaroey approved these changes Dec 5, 2019

View reviewed changes

tensorflow-bot Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Dec 5, 2019

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 5, 2019

pooyadavoodi mentioned this pull request Dec 6, 2019

Add implicit batch experimental #34293

Merged

aaroey requested changes Dec 6, 2019

View reviewed changes

Pooya Davoodi added 2 commits December 6, 2019 13:34

Remove import tensorflow from gen_tftrt_model.py

3461d94

Add tftrt_2.0_saved_model to data deps of trt_convert_test in BUILD

51f0450

Also remove gen_tftrt_model.py from BUILD because this file is not supposed to be used by any test. It's supposed to be used manually by developers to generate a test model if needed.

aaroey previously approved these changes Dec 6, 2019

View reviewed changes

tensorflow-bot Bot added the kokoro:force-run Tests on submitted change label Dec 6, 2019

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 6, 2019

Add no_pip tag to avoid pip_smoke_test depending on newly added saved…

168b722

…model.

aaroey dismissed their stale review via 168b722 December 9, 2019 21:15

aaroey previously approved these changes Dec 9, 2019

View reviewed changes

tensorflow-bot Bot added the kokoro:force-run Tests on submitted change label Dec 9, 2019

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 9, 2019

aaroey added ready to pull PR ready for merge process and removed ready to pull PR ready for merge process labels Dec 10, 2019

Merge branch 'master' into tftrt_bwd_compatibility_test

d066b24

aaroey dismissed their stale review via d066b24 December 10, 2019 17:29

aaroey approved these changes Dec 10, 2019

View reviewed changes

tensorflow-bot Bot added the kokoro:force-run Tests on submitted change label Dec 10, 2019

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 10, 2019

aaroey requested changes Dec 10, 2019

View reviewed changes

aaroey removed the ready to pull PR ready for merge process label Dec 10, 2019

aaroey approved these changes Dec 10, 2019

View reviewed changes

tensorflow-bot Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Dec 10, 2019

kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 10, 2019

tensorflow-copybara pushed a commit that referenced this pull request Dec 11, 2019

Merge pull request #34660 from pooyadavoodi:tftrt_bwd_compatibility_test

e5c8713

PiperOrigin-RevId: 285002333 Change-Id: I698a30181a97fb48cf3fadffefb1a7a4e046d6d4

tensorflow-copybara merged commit d066b24 into tensorflow:master Dec 11, 2019

Navigation Menu

Repository navigation

TF-TRT Backward compatibility test - #34660

TF-TRT Backward compatibility test#34660
tensorflow-copybara merged 11 commits into
tensorflow:masterfrom
pooyadavoodi:tftrt_bwd_compatibility_test

pooyadavoodi commented Nov 27, 2019

Uh oh!

Uh oh!

Uh oh!

rthadur commented Dec 4, 2019

Uh oh!

pooyadavoodi commented Dec 4, 2019

Uh oh!

pooyadavoodi commented Dec 4, 2019

Uh oh!

aaroey commented Dec 5, 2019

Uh oh!

aaroey Dec 6, 2019

Uh oh!

pooyadavoodi Dec 6, 2019

Uh oh!

aaroey Dec 6, 2019

Uh oh!

pooyadavoodi Dec 6, 2019

Uh oh!

aaroey left a comment

Uh oh!

pooyadavoodi commented Dec 6, 2019

Uh oh!

aaroey left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

pooyadavoodi commented Nov 27, 2019

Uh oh!

Uh oh!

Uh oh!

rthadur commented Dec 4, 2019

Uh oh!

pooyadavoodi commented Dec 4, 2019

Uh oh!

pooyadavoodi commented Dec 4, 2019

Uh oh!

aaroey commented Dec 5, 2019

Uh oh!

aaroey Dec 6, 2019

Choose a reason for hiding this comment

Uh oh!

pooyadavoodi Dec 6, 2019

Choose a reason for hiding this comment

Uh oh!

aaroey Dec 6, 2019

Choose a reason for hiding this comment

Uh oh!

pooyadavoodi Dec 6, 2019

Choose a reason for hiding this comment

Uh oh!

aaroey left a comment

Choose a reason for hiding this comment

Uh oh!

pooyadavoodi commented Dec 6, 2019

Uh oh!

aaroey left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants