Golang Tensorflow v2.2.0 fails install

System information

OS Platform and Distribution (Macos catalina 10.15.4):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version:
Python version:
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem

Provide the exact sequence of commands / steps that you executed before running into the problem

Following https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.mdhttps://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.md

go generate github.com/tensorflow/tensorflow/tensorflow/go/op
fails with failed to process "api_def_VarHandleOp.pbtxt": Attribute allowed_devices not defined in base api for VarHandleOp exit status 1 go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status 1

go test github.com/tensorflow/tensorflow/tensorflow/go passes

building or running go code using tensorflow then proceeds to fail with the following output
go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest build command-line-arguments: cannot load github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find module providing package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto

the code run is the example that prints tf library version from https://www.tensorflow.org/install/lang_go

Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

`FROM golang:1.13 AS builder

WORKDIR /app

COPY libtf22.tar.gz /app/libtf22.tar.gz

RUN tar -C /usr/local -xzf libtf22.tar.gz

RUN ldconfig`

Running in Docker with the above Dockerfile, where libtf22.tar.gz is a libtensorflow build from from the v2.2.0 tag. (Which seems to work, if i use that tar.gz file and install tf 2.1 via go get, and run the tf_hello.go example, i get hello from version 2.2).

doing go get -d github.com/tensorflow/tensorflow/tensorflow/go
Produces
package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find package "github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto" in any of: /usr/local/go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto (from $GOROOT) /go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto (from $GOPATH)

the files are downloaded to /${GOPATH}/src/github.com/tensorflow though

Skipping Step 2 build the TensorFlow C Library
cd ${GOPATH}/src/github.com/tensorflow/tensorflow ./configure bazel build -c opt //tensorflow:libtensorflow.so

As it has already been build from, following: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/lib_package/README.md

go generate github.com/tensorflow/tensorflow/tensorflow/go/op in linux container also fails with:

failed to process "api_def_VarHandleOp.pbtxt": Attribute allowed_devices not defined in base api for VarHandleOp exit status 1 /go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status 1

@ymodak

Okay managed to get go generate to work, by doing
cd ${GOPATH}/github.com/src/tensorflow/tensorflow
git checkout tags/v2.2.0
go generate github.com/tensorflow/tensorflow/tensorflow/go/op
This has now placed a lot of generated protobufs in tensorflow/go/vendor/github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto

Now doing
go mod tidy in my src directory seems to understand that there the core_protos for go exists.

Outputting the following:

go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/framework/types_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/framework latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest

But doing
go run main.go
or
go build main.go
Seems to try and download the module and fails with the following output
go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest build command-line-arguments: cannot load github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find module providing package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto

Perhaps @ymodak has an idea on what could be wrong in this case(Perhaps i am just doing something wrong with go mod)

@JacobSMoller

@JacobSMoller Have you found a solution for this? I have the exact same issue.

@ConverJens

@ConverJens Sorry didn't get a notification for you writing me.

Yes not a nice solution. But i did make it work...

Here is a public bucket with working libtensorflow 2.2 for unix and osx (If you haven't got them already)

https://console.cloud.google.com/storage/browser/vml-tf-lib?forceOnBucketsSortingFiltering=false&project=dev-vml-cm

The next step is the not so nice step...

`FROM golang:1.13 AS builder

ARG GO_MOD_TF_PATH=/go/pkg/mod/github.com/tensorflow/[email protected]+incompatible/tensorflow/go
ARG GO_SRC_TF_PATH=/go/src/github.com/tensorflow/tensorflow/tensorflow/go/vendor/github.com/tensorflow/tensorflow/tensorflow/go/core/

WORKDIR /build-lib
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates autoconf automake libtool curl make g++ unzip curl

RUN wget https://storage.googleapis.com/pub/gsutil.tar.gz && tar xfz gsutil.tar.gz -C $HOME && rm gsutil.tar.gz
ENV PATH ${PATH}:/root/gsutil

RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip
RUN unzip -o protoc-3.11.4-linux-x86_64.zip -d /usr/local bin/protoc
RUN unzip -o protoc-3.11.4-linux-x86_64.zip -d /usr/local 'include/*'
RUN rm protoc-3.11.4-linux-x86_64.zip

RUN gsutil cp gs://vml-tf-lib/libtensorflow-unix.tar.gz .
RUN tar -C /usr/local -xzf libtensorflow-unix.tar.gz
RUN ldconfig
RUN rm libtensorflow-unix.tar.gz

WORKDIR /build-lib
RUN go get -d github.com/tensorflow/tensorflow/tensorflow/go/op; exit 0
WORKDIR /go/src/github.com/tensorflow/tensorflow/tensorflow
RUN git checkout tags/v2.2.0
WORKDIR /build-lib
RUN go generate github.com/tensorflow/tensorflow/tensorflow/go/op
WORKDIR /app
COPY go.sum /app/go.sum
COPY go.mod /app/go.mod
RUN go mod download
RUN mkdir -p ${GO_MOD_TF_PATH}/core
RUN cp -r ${GO_SRC_TF_PATH}/core_protos_go_proto ${GO_MOD_TF_PATH}/core/core_protos_go_proto
RUN cp -r ${GO_SRC_TF_PATH}/framework ${GO_MOD_TF_PATH}/core/framework

RUN rm ${GO_MOD_TF_PATH}/core/core_protos_go_proto/autotuning.pb.go
RUN rm ${GO_MOD_TF_PATH}/core/core_protos_go_proto/conv_autotuning.pb.go

COPY tftimer/tf.go /app/tf.go
RUN go build tf.go

FROM debian:bullseye AS runner

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates

COPY --from=builder /app/tf /app/tf
COPY --from=builder /usr/local /usr/local
RUN ldconfig`

So to make it work i do the

go get -d as suggested in go documentation
Then do go generate
then I use go mod download to get the tensorflow package.
And finally copy the generated protos over from the downloaded source code to the GOPATH/pkg/mod/github.com/tensorflow/[email protected]+incompatible/tensorflow/go/core

Not nice in anyway. But haven't been able to find a better way.

@JacobSMoller

@JacobSMoller Thanks for the detailed answer! I will try it the week after next and let you know how it goes!

@JacobSMoller

@JacobSMoller Thanks for the hint. I tried it, but failed with
Step 23/38 : COPY go.sum /app/go.sum
COPY failed: stat /var/lib/docker/tmp/docker-builder720470318/go.sum: no such file or directory

where do you get the go.sum and go.mod files from?

They are generated when using go modules.

https://blog.golang.org/using-go-modules

Fixed as part of:
e5e495d

Note that it requires from master:
bazel build -c opt //tensorflow/tools/lib_package:libtensorflow
tar -xzvf the package in /usr/local
go generate github.com/tensorflow/tensorflow/tensorflow/go/op

Are you satisfied with the resolution of your issue?
Yes
No

@jhseu

@jhseu This as far as I can tell, still doesn't enable users to do go get github.com/tensorflow/tensorflow/tensorflow/go
and simply run things.

When using go modules I still need to generate the protobufs in $GOPATH/src/github.com/tensorflow/tensorflow And then move the generated protobufs in to $GOPATH/pkg/mod/github.com/tensorflow/...

Also doing the go generate command from master now returns a new error:

2020/07/27 13:07:50 failed to process "api_def_BatchFunction.pbtxt": Attribute enable_large_batch_splitting not defined in base api for BatchFunction exit status 1 go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status

@JacobSMoller

@JacobSMoller Yeah, go get doesn't work for now. I made a pull request to fix it, but there were some objections to supporting both the bazel and generated protobufs to make it work.

For that error: it occurs when there's a version mismatch between libtensorflow.so and the TF ops in the source code. See my command above for building libtensorflow.so.

@jhseu

@jhseu alright thanks :)

So no road map for having a go gettable Tensorflow isntallation :)?

I've spend a good day and a half truign to get this installed on my Ubuntu machine. Tried most things mentioned above, and have built everything from scratch. After installing Protobuf, it's not able to generate the protobufs

$ go generate github.com/tensorflow/tensorflow/tensorflow/go/op
google/protobuf/any.proto: File not found.
google/protobuf/duration.proto: File not found.
tensorflow/core/protobuf/autotuning.proto:10:1: Import "google/protobuf/any.proto" was not found or had errors.
tensorflow/core/protobuf/autotuning.proto:11:1: Import "google/protobuf/duration.proto" was not found or had errors.
tensorflow/core/protobuf/autotuning.proto:61:3: "google.protobuf.Duration" is not defined.
tensorflow/core/protobuf/autotuning.proto:74:3: "google.protobuf.Any" is not defined.
../genop/main.go:17: running "bash": exit status 1
go/op/generate.go:17: running "go": exit status 1

@silverark

@silverark How are you installing protoc? It looks like you're missing the default protos from your installation. Typically they're in /usr/local/include after installation, like so:

jhseu0:~$ locate any.proto
/usr/include/google/protobuf/any.proto

@jhseu

Thanks @jhseu. Turns out I had a different version of proto hidden in my ~/local

Finally managed to get it working but only without module. If I use go mod in my project it downloads the version into pkg which is still broken. I'm currently at the stage @JacobSMoller is describing.

Navigation Menu

Repository navigation

Golang Tensorflow v2.2.0 fails install #39307

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Golang Tensorflow v2.2.0 fails install #39307

Description

Activity

JacobSMoller commented on May 10, 2020

JacobSMoller commented on May 13, 2020

ConverJens commented on May 20, 2020

JacobSMoller commented on May 27, 2020

ConverJens commented on May 29, 2020

NiBurhe commented on Jun 8, 2020

JacobSMoller commented on Jun 9, 2020

jhseu commented on Jul 23, 2020

google-ml-butler commented on Jul 23, 2020

JacobSMoller commented on Jul 27, 2020

jhseu commented on Jul 27, 2020

JacobSMoller commented on Aug 4, 2020

mattnelsonuk commented on Sep 23, 2020

jhseu commented on Sep 23, 2020

mattnelsonuk commented on Sep 24, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions