Can you rebase and get the CI build passing? I will start reviewing, mostly on IP documentation concerns over the code itself, so we can get this merged ASAP this coming so Ray can transition, and we can have this released in Arrow 0.5.0

Thanks, sounds good, I'll do these! I hope we can get this merged and do the switch soon to avoid having to backport changes from Ray.

Let me know about any IP issues you see.

Also let me know if you have ideas about improving the code!

Looks like maybe just some linting now

Yeah, there are only a few small things left, will try to finish it today! Concerning Cython I agree we should do it, but suggest to first get this merged and rebase Ray on top of it to make the latter as easy as possible.

See google's guidance on naming: https://google.github.io/styleguide/cppguide.html#Function_Names

It seems that there used to be an allowance to name function with snake case when they are "cheap". This was recently changed; I think it would be a good idea to make all functions PascalCase instead of questioning "how cheap is this function to call" google/styleguide@db0a263#diff-26120df7bca3279afbf749017c778545R4277

I suspect that this issue came up too much in TensorFlow and they decided to drop it

That's weird that the tests are hanging. Maybe killall -9 would help (perhaps not without sudo)?

I can now reproduce it locally; if I just run "./client_tests", it doesn't hang, but

/home/pcmoritz/tmp/arrow/cpp/build-support/run-test.sh "/home/pcmoritz/tmp/arrow/cpp-build" "test" "/home/pcmoritz/tmp/arrow/cpp-build/debug//client_tests"

hangs and if I do control-c, I get

^CTraceback (most recent call last):
  File "/home/pcmoritz/tmp/arrow/cpp/build-support/asan_symbolize.py", line 368, in <module>
    loop.process_stdin()
  File "/home/pcmoritz/tmp/arrow/cpp/build-support/asan_symbolize.py", line 340, in process_stdin
    line = sys.stdin.readline()
KeyboardInterrupt

Let me know if you have an idea, I'm not familiar with the address sanitizer scripts but will keep digging.

What happens when you run only /home/pcmoritz/tmp/arrow/cpp-build/debug//client_tests?

Got it now, the ASAN script seems to get confused by the output of the store. Piping that to /dev/null seems to fix it.

@pcmoritz I still have to go over this code: But could there be problems with dlmalloc and the PR #761 that makes jemalloc the default allocator in Arrow?

Plasma runs as a separate process, is that right?

Yeah it is a separate process. Also, even in plasma we are not replacing the malloc function with the dlmalloc one (the #define USE_DL_PREFIX in malloc.cc makes sure that the dlmalloc malloc has "dl" as a prefix); and then we call dlmalloc with the prefix manually when we need it to get a chunk of share memory. So for the heap of plasma it will use whichever allocator is the default, right now that's the system malloc but it could be jemalloc in the future if we prefer that.

I'm done for now, the next step is to get this merged, rebase Ray and address some of points I haven't addressed yet, and write some more documentation. Shall I rebase this PR now or wait for other changes in Arrow like the jemalloc PR? I hope the remaining Java related test failures go away once it is rebased.