Skip to content

FLAKE: c_macos_dbg_native.bins/dbg/concurrent_connectivity_test fails with "Too many open files" #21534

@jtattermusch

Description

@jtattermusch

https://source.cloud.google.com/results/invocations/cfe84625-e024-46f3-b197-686dd5ae6285/targets/github%2Fgrpc%2Frun_tests%2Fc_macos_dbg_native/tests;group=c_macos_dbg_native;test=bins%2Fdbg%2Fconcurrent_connectivity_test%20GRPC_POLL_STRATEGY%3Dpoll;row=2

D1219 11:21:31.679461000 140735622603648 test_config.cc:384]           test slowdown factor: sanitizer=1, fixture=1, poller=1, total=1
D1219 11:21:31.680686000 140735622603648 ev_posix.cc:174]              Using polling engine: poll
D1219 11:21:31.680908000 140735622603648 dns_resolver_ares.cc:503]     Using ares dns resolver
D1219 11:21:31.681041000 140735622603648 concurrent_connectivity_test.cc:176] Wave 1
I1219 11:21:31.686913000 123145387610112 subchannel.cc:1003]           Connect failed: {"created":"@1576783291.686839000","description":"Failed to connect to remote host: Connection refused","errno":61,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":200,"os_error":"Connection refused","syscall":"connect","target_address":"ipv6:[::1]:54321"}
I1219 11:21:31.687162000 123145387610112 subchannel.cc:1003]           Connect failed: {"created":"@1576783291.687124000","description":"Failed to connect to remote host: Connection refused","errno":61,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":200,"os_error":"Connection refused","syscall":"connect","target_address":"ipv4:127.0.0.1:54321"}
I1219 11:21:31.687225000 123145386000384 subchannel.cc:942]            Subchannel 0x7f820b004cf0: Retry in 1001 milliseconds
E1219 11:21:31.826813000 123145409073152 wakeup_fd_pipe.cc:40]         pipe creation failed (24): Too many open files
E1219 11:21:31.826867000 123145409073152 ev_poll_posix.cc:942]         pollset_work: {"created":"@1576783291.826846000","description":"Too many open files","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.cc","file_line":41,"os_error":"Too many open files","syscall":"pipe"}
E1219 11:21:31.826887000 123145409073152 completion_queue.cc:1044]     Completion queue next failed: {"created":"@1576783291.826846000","description":"Too many open files","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.cc","file_line":41,"os_error":"Too many open files","syscall":"pipe"}
E1219 11:21:31.826906000 123145433755648 wakeup_fd_pipe.cc:40]         pipe creation failed (24): Too many open files
E1219 11:21:31.826907000 123145409073152 concurrent_connectivity_test.cc:77] assertion failed: grpc_completion_queue_next(cq, poll_time, nullptr).type == GRPC_OP_COMPLETE



*******************************
Caught signal SIGABRT
E1219 11:21:31.826958000 123145433755648 ev_poll_posix.cc:942]         pollset_work: {"created":"@1576783291.826939000","description":"Too many open files","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.cc","file_line":41,"os_error":"Too many open files","syscall":"pipe"}
E1219 11:21:31.827094000 123145433755648 completion_queue.cc:1044]     Completion queue next failed: {"created":"@1576783291.826939000","description":"Too many open files","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.cc","file_line":41,"os_error":"Too many open files","syscall":"pipe"}
E1219 11:21:31.827120000 123145433755648 concurrent_connectivity_test.cc:77] assertion failed: grpc_completion_queue_next(cq, poll_time, nullptr).type == GRPC_OP_COMPLETE

We already do increase ulimit at the start of the build:

+ source tools/internal_ci/helper_scripts/prepare_build_macos_rc
++ launchctl limit maxfiles
	maxfiles    256            unlimited      
++ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1418
virtual memory          (kbytes, -v) unlimited

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions