Skip to content

Support GC_init (and GC_get_stack_base) from non-main thread on FreeBSD #180

Closed

Description

@dimpase

There are warnings such as "FreeBSD does not yet fully support threads with Boehm GC" issued while building on FreeBSD, but what exactly is not supported cannot be found, at least I couldn't found anything concrete online.

I am attempting to port to FreeBSD a Python (CPython, to be precise) extension that uses Boehm GC, and face rather mysterious crashes of the GC in a threaded setting.

Having a bit more documentation/examples on this would be great.

Edit: the warnings were removed in a recent (April 2017) commit 1e40b9b

Activity

  1. dimpase commented on Sep 25, 2017

    @dimpase
    ContributorAuthor

    On the topic, the most recent I have found is this thread - dated April 2012, that stops inconclusively, it seems.

  2. ivmai commented on Sep 25, 2017

    @ivmai
    Member

    To @magv , @paurkedal : could you reply, please?
    To: @dimpase : do you observe the crashes with the latest master?

  3. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor

    It's gc-7.6.0 with the libatomic_ops-7.4.4; although more or less identical crashes are observed with gc-7.2. It also does not seem to matter whether it's built with clang-3.8, clang 5.0, or with gcc6.

  4. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor

    the platform is FreeBSD 11.0-RELEASE-p1 #0 r306420, x86_64, running in a VMWare on a Linux box.

  5. paurkedal commented on Sep 26, 2017

    @paurkedal
    Contributor

    @dimpase Can you check if the thread_stress_test from https://github.com/paurkedal/bdwgc/tree/thread_stress_test also crashes? This is a version of disclam_test where I removed the disclaim notifiers, to check whether those are involved.

  6. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor

    All the tests from that branch pass. (Built with gcc6, without libatomic_ops).

    However, if used in the application in question, this branch gives essentially the same segfault as I saw earlier. In a nutshell, depending on the thread (good in the "main" CPython thread, crashes if it's coming from a thread responsible for tab completion) it is launched from, GC either initialises and works (at least in part, there are more crashes coming later, again depending upon the thread), or crashed at initialisation (launched from the embedded Lisp interpreter ECL), like this:

    Program received signal SIGSEGV, Segmentation fault.
    GC_push_all_eager (bottom=<optimized out>, top=<optimized out>) at extra/../mark.c:1608
    1608    extra/../mark.c: No such file or directory.
    (gdb) bt
    #0  GC_push_all_eager (bottom=<optimized out>, top=<optimized out>) at extra/../mark.c:1608
    #1  0x000000087eb2b8ad in GC_push_all_stacks () at extra/../pthread_stop_world.c:573
    #2  0x000000087eb27cb5 in GC_mark_some (cold_gc_frame=0x7fffdeff7e80 " \265\327~\b") at extra/../mark.c:427
    #3  0x000000087eb2801d in GC_stopped_mark (stop_func=stop_func@entry=0x87eb195c0 <GC_never_stop_func>) at extra/../alloc.c:707
    #4  0x000000087eb291ae in GC_try_to_collect_inner (stop_func=0x87eb195c0 <GC_never_stop_func>) at extra/../alloc.c:493
    #5  0x000000087eb2a84d in GC_init () at extra/../misc.c:1290
    #6  0x000000087e7cde80 in init_alloc () at /usr/home/dima/Sage/sage/local/var/tmp/sage/build/ecl-16.1.2.p4/src/src/c/alloc_2.d:786
    #7  0x000000087e69622d in cl_boot (argc=argc@entry=1, argv=argv@entry=0x7fffdeff7fc8)
        at /usr/home/dima/Sage/sage/local/var/tmp/sage/build/ecl-16.1.2.p4/src/src/c/main.d:521
    #8  0x000000087e20a728 in __pyx_pf_4sage_4libs_3ecl_4init_ecl (__pyx_self=<optimized out>) at build/cythonized/sage/libs/ecl.c:5417
    ....
    #69 0x0000000800a809f3 in PyObject_Call (func=func@entry=, arg=arg@entry=, kw=<optimized out>) at Objects/abstract.c:2547
    #70 0x0000000800b34937 in PyEval_CallObjectWithKeywords (func=, arg=, kw=<optimized out>) at Python/ceval.c:4221
    #71 0x0000000800b80272 in t_bootstrap (boot_raw=0x8799a8440) at ./Modules/threadmodule.c:620
    #72 0x0000000801066b55 in ?? () from /lib/libthr.so.3
    ...
    

    (sorry --- a non-debugging build ATM). And this all works on Linux, Windows/cygwin64, and OSX.

  7. ivmai commented on Sep 26, 2017

    @ivmai
    Member

    thread responsible for tab completion

    Is this thread created by GC_pthread_create, or by pthread_create (not intercepted by GC) with a manual thread registration by GC_register_my_thread? In the latter case, is it manually unregistered?

    Incremental mode is off (i.e. GC_enable_incremental is not called), right?

    #0 GC_push_all_eager (bottom=, top=) at extra/../mark.c:1608

    Does the region bottom..top belong to the thread which is alive and stopped by GC?

    Please compile GC with --enable-gc-assertions (if not yet).

  8. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor
  9. ivmai commented on Sep 26, 2017

    @ivmai
    Member

    none of this. In a nutshell: we run CPython and load a Python extension, which has GC embedded.
    hmm, is there any thread it must stop?

    Is GC used from a single thread? Does the region bottom..top belong to this thread?

  10. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor
  11. ivmai commented on Sep 26, 2017

    @ivmai
    Member

    For some reason on Linux/OSX we don't need to register/de-register these threads, it "just works"

    GC tracks threads on OS X automatically. On Linux, pthread_create might be intercepted by GC. Please compile GC with -D DEBUG_THREADS (it will print "About to start new thread from thread" on every intercepted pthread_create).

    Is there a function call one can use to check this?

    I think no but -D DEBUG_THREADS should log the relevant information - grep for "Pushing stacks from thread ...", "About to start new thread from thread ..."

  12. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor

    Does this mean that in scenario 2) GC might be crashing due to an attempt to do a GC_push... on the main thread area, without registering it first?

  13. ivmai commented on Sep 26, 2017

    @ivmai
    Member

    Does this mean that in scenario 2) GC might be crashing due to an attempt to do a GC_push... on the main thread area, without registering it first?

    Maybe. Please What are the top/bottom values in GC_push_all_eager (bottom=, top=), what is the value of self and p->id in GC_push_all_stacks at crash? Was p->id registered (i.e. does the log contain "About to start new thread from thread id>")?

  14. dimpase commented on Sep 26, 2017

    @dimpase
    ContributorAuthor

    Running scenario 2), I set breakpoint 1 at GC_push_all_eager and see the following;
    it crashes after the 2nd time the breakpoint is reached.
    (I've also set breakpoints at GC_push_all_stacks to see self, p, p->id, nthreads.
    no surprises here - nthreads=1)
    And no "About to start new thread from thread id>" message printed.

    Creating thread 0x801e18300
    Starting mark helper for mark number 0
    Starting mark helper for mark number 0
    Starting mark helper for mark number 0
    Starting mark helper for mark number 0
    Starting mark helper for mark number 0
    Starting mark helper for mark number 0
    Starting mark helper for mark number 0
    Stopping the world from 0x801e18300
    World stopped from 0x801e18300
    
    Breakpoint 1, GC_push_all_eager (bottom=0x7fffdf7f6c70 "\220l\177\337\377\177", top=0x7fffdf7f70d0 "\020q\177\337\377\177") at extra/../mark.c:1593
    1593    extra/../mark.c: No such file or directory.
    (gdb) cont
    Continuing.
    ...
    Breakpoint 3, GC_push_all_stacks () at extra/../pthread_stop_world.c:522
    522     in extra/../pthread_stop_world.c
    (gdb) p self
    $1 = (pthread_t) 0x801e18300
    ...
    Pushing stacks from thread 0x801e18300
    ...
    Breakpoint 5 at 0x882bd2e96: file extra/../pthread_stop_world.c, line 566.
    (gdb) cont
    Continuing.
    Stack for thread 0x801e18300 = [0x7fffdf7f7010,0x7ffffffff000)
    
    Breakpoint 5, GC_push_all_stacks () at extra/../pthread_stop_world.c:566
    566     in extra/../pthread_stop_world.c
    (gdb) p p
    $8 = (GC_thread) 0x882e21ee0 <first_thread>
    (gdb) p nthreads
    $9 = 1
    (gdb) p p->id
    $10 = (pthread_t) 0x801e18300
    ...
    (gdb) p self
    $12 = (pthread_t) 0x801e18300
    (gdb) cont
    Continuing.
    
    Breakpoint 1, GC_push_all_eager (bottom=0x7fffdf7f7010 "pp\177\337\377\177", top=0x7ffffffff000 "\377T$\020\215D$ P\270\241\001") at extra/../mark.c:1593
    1593    extra/../mark.c: No such file or directory.
    (gdb) cont
    Continuing.
    
    Program received signal SIGSEGV, Segmentation fault.
    GC_push_all_eager (bottom=0x7fffdf7f7010 "pp\177\337\377\177", top=0x7ffffffff000 "\377T$\020\215D$ P\270\241\001") at extra/../mark.c:1608
    1608    in extra/../mark.c
    (gdb) p p
    $13 = (word *) 0x7fffdfffe000
    
  15. ivmai commented on Sep 26, 2017

    @ivmai
    Member

    Stack for thread 0x801e18300 = [0x7fffdf7f7010,0x7ffffffff000)

    The 2nd value is not correct, it's the upper boundary of the primordial thread (not the current one).
    It is obtained by GC_freebsd_main_stack_base which assumes the GC to be initialized from the main thread.
    On Linux, pthread_attr_getstack() is used instead.

  16. 13 remaining items

  17. ivmai commented on Oct 2, 2017

    @ivmai
    Member

    It looks like this patch breaks kFreeBSD build. So, I will make a fix for the patch.

  18. reopened this on Oct 2, 2017
  19. ivmai commented on Oct 2, 2017

    @ivmai
    Member

    I would say that a straight C test for this functionality would be very good to have in https://github.com/ivmai/bdwgc/tree/master/tests
    ---I probably ought to contribute one...

    Right. You are welcome.

  20. ivmai commented on Oct 5, 2017

    @ivmai
    Member

    I would say that a straight C test for this functionality would be very good to have in tests

    Such test already exists initsecondarythread.c but it has an exclusion for FreeBSD - to be removed.

  21. dimpase commented on Oct 6, 2017

    @dimpase
    ContributorAuthor

    OK, good, on FreeBSD with gc-7.6.0 patched by 49d7fe7 I tried commenting out GC_INIT() in initsecondarythread.c on https://github.com/ivmai/bdwgc/blob/master/tests/initsecondarythread.c#L79
    and this test passes (and as a sanity check, without the patch from 49d7fe7 this test gives a segfault).

    (I am not sure how to write a proper condition for skipping this line on FreeBSD, thus only this rudimentary check, sorry).

  22. ivmai commented on Oct 8, 2017

    @ivmai
    Member

    Here is the fix (39ca0b5) of my previous commit (49d7fe7). Now it works for kFreeBSD.
    Please check 49d7fe7+39ca0b5 - if works for you then I will squash them and push to release-7_6 branch.

  23. dimpase commented on Oct 8, 2017

    @dimpase
    ContributorAuthor

    shouldn't checking for pthread_np.h be done in configure.ac?

  24. dimpase commented on Oct 9, 2017

    @dimpase
    ContributorAuthor

    OK, 49d7fe7+39ca0b5 work for me (on FreeBSD 11.0 and on Linux too), in the master branch as well as on top of 7.6.

  25. ivmai commented on Oct 12, 2017

    @ivmai
    Member

    shouldn't checking for pthread_np.h be done in configure.ac?

    It's up to the design. configure is optional. Historically, multiple build systems are supported and I'm trying not to break them. If possible, the features of the target system should be identified in gcconfig.h - this allows avoid duplicating feature detection in the build scripts. And, not to mention, it allows to compile bdwgc from the command line easily (e.g. gcc -I include tests/test.c extra/gc.c)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions