-
Notifications
You must be signed in to change notification settings - Fork 11.1k
Segmentation fault in python google cloud libraries #13327
Description
Please answer these questions before submitting your issue.
What version of gRPC and what language are you using?
I'm using Python. We are using several python gcloud libs:
google-api-core==0.1.1
google-auth==1.1.1
google-cloud==0.29.0
google-cloud-bigquery==0.27.0
google-cloud-bigtable==0.28.1
google-cloud-core==0.27.1
google-cloud-datastore==1.4.0
google-cloud-dns==0.28.0
google-cloud-error-reporting==0.28.0
google-cloud-firestore==0.28.0
google-cloud-language==0.31.0
google-cloud-logging==1.4.0
google-cloud-monitoring==0.28.0
google-cloud-pubsub==0.29.0
google-cloud-resource-manager==0.28.0
google-cloud-runtimeconfig==0.28.0
google-cloud-spanner==0.29.0
google-cloud-speech==0.30.0
google-cloud-storage==1.6.0
google-cloud-trace==0.16.0
google-cloud-translate==1.3.0
google-cloud-videointelligence==0.28.0
google-cloud-vision==0.28.0
google-gax==0.15.15
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4
grpcio==1.7.0
What operating system (Linux, Windows, …) and version?
(venv) tanakaed@triage-bot:~/server$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
What runtime / compiler are you using (e.g. python version or version of gcc)
Python info:
`{noformat}
(venv) tanakaed@triage-bot:/server$ python --version/server$ conda info
Python 3.6.2 :: Anaconda, Inc.
(venv) tanakaed@triage-bot:
Current conda install:
platform : linux-64
conda version : 4.3.27
conda is private : False
conda-env version : 4.3.27
conda-build version : 3.0.23
python version : 3.6.2.final.0
requests version : 2.18.4
root environment : /home/tanakaed/anaconda3 (writable)
default environment : /home/tanakaed/anaconda3/envs/venv
envs directories : /home/tanakaed/anaconda3/envs
/home/tanakaed/.conda/envs
package cache : /home/tanakaed/anaconda3/pkgs
/home/tanakaed/.conda/pkgs
channel URLs : https://repo.continuum.io/pkgs/main/linux-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/linux-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/linux-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/linux-64
https://repo.continuum.io/pkgs/pro/noarch
config file : None
netrc file : None
offline mode : False
user-agent : conda/4.3.27 requests/2.18.4 CPython/3.6.2 Linux/4.10.0-38-generic debian/stretch/sid glibc/2.23
UID:GID : 1001:1002
tanakaed@triage-bot:$ gcc --version16.04.5) 5.4.0 20160609
gcc (Ubuntu 5.4.0-6ubuntu1
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.`
What did you do?
We have python code to both publish and pull messages from pubsub. We also have python that interfaces with google datastore and google logging. I don't know which one of these codes is triggering this segmentation fault. My code runs fine for a while but after ~60 mins running and processing some cases, a segfault is raised.
I ran my python script inside gdb and this is what I got:
Thread 9 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff054a700 (LWP 29398)]
gpr_ref_non_zero (r=0x0) at src/core/lib/support/sync.c:93
93 src/core/lib/support/sync.c: No such file or directory.
(gdb) backtrace
#0 gpr_ref_non_zero (r=0x0) at src/core/lib/support/sync.c:93
#1 0x00007ffff12c8365 in grpc_stream_ref (refcount=) at src/core/lib/transport/transport.c:50
#2 0x00007ffff12f3490 in send_security_metadata (batch=0x7fff8c0820f0, elem=0x7fff8c0821a0, exec_ctx=0x7ffff0549ec0)
at src/core/lib/security/transport/client_auth_filter.c:216
#3 on_host_checked (exec_ctx=exec_ctx@entry=0x7ffff0549ec0, arg=arg@entry=0x7fff8c0820f0, error=)
at src/core/lib/security/transport/client_auth_filter.c:231
#4 0x00007ffff12f396f in auth_start_transport_stream_op_batch (exec_ctx=0x7ffff0549ec0, elem=0x7fff8c0821a0, batch=0x7fff8c0820f0)
at src/core/lib/security/transport/client_auth_filter.c:316
#5 0x00007ffff1300f68 in waiting_for_pick_batches_resume (elem=, elem=, exec_ctx=0x7ffff0549ec0)
at src/core/ext/filters/client_channel/client_channel.c:953
#6 create_subchannel_call_locked (error=0x0, elem=, exec_ctx=0x7ffff0549ec0)
at src/core/ext/filters/client_channel/client_channel.c:1016
#7 pick_done_locked (exec_ctx=0x7ffff0549ec0, elem=, error=0x0) at src/core/ext/filters/client_channel/client_channel.c:1042
#8 0x00007ffff12932f3 in grpc_combiner_continue_exec_ctx (exec_ctx=exec_ctx@entry=0x7ffff0549ec0) at src/core/lib/iomgr/combiner.c:259
#9 0x00007ffff129bdf8 in grpc_exec_ctx_flush (exec_ctx=exec_ctx@entry=0x7ffff0549ec0) at src/core/lib/iomgr/exec_ctx.c:93
#10 0x00007ffff129c3c1 in run_closures (exec_ctx=0x7ffff0549ec0, list=...) at src/core/lib/iomgr/executor.c:81
#11 executor_thread (arg=arg@entry=0x5555565d3e00) at src/core/lib/iomgr/executor.c:181
#12 0x00007ffff1285c37 in thread_body (v=) at src/core/lib/support/thd_posix.c:53
#13 0x00007ffff7bc16ba in start_thread (arg=0x7ffff054a700) at pthread_create.c:333
#14 0x00007ffff78f73dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)
What did you expect to see?
No segmentation fault