Closure debugger support by markflorisson · Pull Request #4 · cython/cython

markflorisson · 2010-12-28T21:19:48Z

Before I left on vacation I added some quick support for closures, namely setting breakpoints on inner functions. At the time I didn't have much time too look into it, apparently the cname of an Entry object with 'from_closure' set to True is the cname that will be looked up in the scope object, which I believe is Naming.cur_scope_name. My question is, is Naming.cur_scope_name always the right scope (C variable name) to access entry.cname in?

Out of interest, why is the scope an actual Python object instead of a C struct? (not that it really matters)

Another thing, could we initialize automatic Python variables (PyObject *, in this case, Naming.cur_scope_name) to NULL? Because it's really hard to tell from the debugger whether a non-NULL automatic variable is actually valid. For variables used in Cython code the debugger keeps an "initialized on lineno X" marker, but it would be a bummer to special-case the scope. If nobody wants to do it, I could make this change in the code generator in Nodes.py.
Using the scope from the debugger is generally not really a problem, the result is that some variables might be displayed at incorrect addresses listed as "unknown remote". However, for code execution this is a major problem as it almost certainly leads to an instantaneous segmentation fault as the debugger tries to inc- and decref (and insert into a newly allocated dict) an uninitialized pointer.

… and re-fix runtests.py

…atchpoint to enable) as hardware watchpoints are limited.

…reakpoints that are set and deleted instead of disabled)

Debugger: Recognition of module-level Cython code (initmodulename and PyInit_modulename) Fix debug flag (import Parsing when needed)

…use distutils.core.setup()

… have been initialized for closures

robertwb · 2011-01-14T09:01:30Z

FYI, The change to try importing gdb was made because otherwise it failed when importing test_lib{p,c}ython_in_gdb.py to look for doctests. I'm surprised you didn't see this. I've now made it skip these files.

Over 50% of the line changes in this diff were EOL whitespace additions--it would be nice if you set your editor to not add trailing whitespace (especially to un-edited areas) to ease reviewing code. I'm still having issues with gdb 7.2 on my OS X box, but the code looks good to me.

markflorisson · 2011-01-14T09:34:18Z

What kind of issues do you have with gdb 7.2? Does this mean that so far you haven't been able to run the debugger?

I think I didn't notice the doctest problem because I told the testrunner to skip those. I'll run the entire testsuite in the future. About the whitespace, I'll tell my editor. I think there were so many changes because you stripped the files and I merged with the "ignore whitespace option" at some point. I'll pay more attention to this in the future.

robertwb · 2011-01-14T09:58:27Z

I had to compile 7.2 from source, twiddle with "blessing/signing" the binary, then still had some kind of library loading/linking errors when I tried to run it with Python (which I had to compile a new copy as well). I haven't tried again since before Christmas.

Thanks for taking a look at this. I bet you're right about the whitespace errors being from an incomplete merge from my work--that would explain a lot better why there were so many changes on seemingly irrelevant lines. I tried merging with "git merge -srecursive -Xignore-space-at-eol" on my end, but that didn't do the trick for me, what did you use?

markflorisson · 2011-01-14T10:13:41Z

That is unfortunate. gdb does have a lot of dependencies, unfortunately I don't use OS X, so I don't have any pointers. If you want, though, you can paste the error output and I'll have a look as I've compiled it multiple times. There is also the gdb support channel on irc.freenode.net with a lot of savvy people, although most of them probably don't use OS X.

Another option would be to run Linux in a virtual OS, I can recommend fedora 14 for that since it ships with gdbg 7.2 build with Python by default.

I used git merge --strategy-option=ignore-space-at-eol FETCH_HEAD. I was told this option will be new in the upcoming git release, it's currently available from git://git.kernel.org/pub/scm/git/git.git . This means you will have to build git, of course.

robertwb · 2011-01-14T10:47:06Z

Yes, I've been meaning to try it on one of the linux boxes I use (none of them had gdb 7.2, but it shouldn't be as hard to build/upgrade). I actually built the git 1.7.4 release candidate just to try out this feature (merging from one branch to another), but no luck...

markflorisson · 2011-01-14T10:55:06Z

No luck merging with this option? What exactly did you do and what happened?

robertwb · 2011-01-14T11:09:58Z

I was sitting at b4c160b as my master.
git -b gdb
git pull [your repo]
git checkout master
git merge -srecursive -Xignore-space-at-eol gdb # tried several permutations of this

I was expecting it to merge the non-whitespace-at-eol changes, but it was just a normal merge.

markflorisson · 2011-01-14T11:23:03Z

Hmm, my git has no -b option.

Can you try this:

git fetch [my repo]
git merge --strategy-option=ignore-space-at-eol FETCH_HEAD

? I think in your case the pull already tries to merge in the gdb branch (assuming git -b gdb is equivalent to git branch gdb; git checkout gdb), which is at the same HEAD as the master branch, so it will try to merge as normal with the same problems as before (I think).

robertwb · 2011-01-14T11:29:04Z

Same exact behavior. (BTW, that should have been "git checkout -b ...")

markflorisson · 2011-01-14T11:47:59Z

I see, that's weird. It did work for me, though. In any case, I'll try to ensure that nobody will need this option in the future anyway by pulling from mainline when you strip before continuing work on my branch.

robertwb · 2011-01-14T18:09:53Z

Thanks. I usually try not to be too picky about this kind of thing, but it was to the point that the real changes were dwarfed by the whitespace ones. In any case, as you mentioned, it's probably just an unfortunate artifact of merging the whitespace stripping patch, so I doubt anything this drastic would happen again.

Some code I have uses lxml. When I compile and run lxml with LLVM-12's undefined behaviour sanitizer, I see this: ``` lxml/4.6.3/cython_parts=etree.c/etree.c:191228:65: runtime error: applying zero offset to null pointer #0 0x7faedc799e7c in __Pyx_PyList_GetSlice lxml/4.6.3/cython_parts=etree.c/etree.c:191228 cython#1 0x7faedc6b3723 in __pyx_f_4lxml_5etree_9_ErrorLog_copy lxml/4.6.3/cython_parts=etree.c/etree.c:38092 cython#2 0x7faedc77636e in __pyx_f_4lxml_5etree___copyGlobalErrorLog lxml/4.6.3/cython_parts=etree.c/etree.c:40203 cython#3 0x7faedc775a5c in __pyx_pf_4lxml_5etree_9LxmlError___init__ lxml/4.6.3/cython_parts=etree.c/etree.c:21080 cython#4 0x7faedc774dff in __pyx_pw_4lxml_5etree_9LxmlError_1__init__ lxml/4.6.3/cython_parts=etree.c/etree.c:21019 ``` which implies that Cython's generating undefined behaviour. Making an early exit from the modified function (essentially by hoisting an early exit from one level lower in the stack) avoids this.

…lity function (GH-4734) Some code I have uses lxml. When I compile and run lxml with LLVM-12's undefined behaviour sanitizer, I see this: ``` lxml/4.6.3/cython_parts=etree.c/etree.c:191228:65: runtime error: applying zero offset to null pointer #0 0x7faedc799e7c in __Pyx_PyList_GetSlice lxml/4.6.3/cython_parts=etree.c/etree.c:191228 #1 0x7faedc6b3723 in __pyx_f_4lxml_5etree_9_ErrorLog_copy lxml/4.6.3/cython_parts=etree.c/etree.c:38092 #2 0x7faedc77636e in __pyx_f_4lxml_5etree___copyGlobalErrorLog lxml/4.6.3/cython_parts=etree.c/etree.c:40203 #3 0x7faedc775a5c in __pyx_pf_4lxml_5etree_9LxmlError___init__ lxml/4.6.3/cython_parts=etree.c/etree.c:21080 #4 0x7faedc774dff in __pyx_pw_4lxml_5etree_9LxmlError_1__init__ lxml/4.6.3/cython_parts=etree.c/etree.c:21019 ``` which implies that Cython's generating undefined behaviour. Making an early exit from the modified function (essentially by hoisting an early exit from one level lower in the stack) avoids this.

…d of object object means the argument is always non-NULL valid Python object, while PyObject* argument can be generally NULL. If the argument is indeed passed as NULL, and we declare it as object, generated code will crash while trying to incref it. Quoting cython#4822: object.pxd currently declares `newfunc` as follows: ```pyx ctypedef object (*newfunc)(cpython.type.type, object, object) # (type, args, kwargs) ``` which implies that `args` and `kwargs` are always live objects and cannot be NULL. However Python can, and does, call tp_new with either args=NULL, or kwargs=NULL or both. And in such cases this leads to segfault in automatically-generated __Pyx_INCREF for args or kw. The fix is to change `object` to `PyObject*` for both args and kwargs. Please see below for details: ```cython # cython: language_level=3 from cpython cimport newfunc, type as cpytype, Py_TYPE cdef class X: cdef int i def __init__(self, i): self.i = i def __repr__(self): return 'X(%d)' % self.i cdef newfunc _orig_tp_new = Py_TYPE(X(0)).tp_new cdef object _trace_tp_new(cpytype cls, object args, object kw): print('_trace_tp_new', cls, args, kw) return _orig_tp_new(cls, args, kw) Py_TYPE(X(0)).tp_new = _trace_tp_new x = X(123) print(x) ``` ```console (neo) (py3.venv) (g.env) kirr@deca:~/src/tools/go/pygolang$ cythonize -i x.pyx Compiling /home/kirr/src/tools/go/pygolang/x.pyx because it changed. [1/1] Cythonizing /home/kirr/src/tools/go/pygolang/x.pyx running build_ext building 'x' extension ... x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kirr/src/wendelin/venv/py3.venv/include -I/usr/include/python3.9 -c /home/kirr/src/tools/go/pygolang/x.c -o /home/kirr/src/tools/go/pygolang/tmpqkz1r96s/home/kirr/src/tools/go/pygolang/x.o x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 /home/kirr/src/tools/go/pygolang/tmpqkz1r96s/home/kirr/src/tools/go/pygolang/x.o -o /home/kirr/src/tools/go/pygolang/x.cpython-39-x86_64-linux-gnu.so ``` ```console (neo) (py3.venv) (g.env) kirr@deca:~/src/tools/go/pygolang$ python -c 'import x' Ошибка сегментирования (стек памяти сброшен на диск) ``` ```console (neo) (py3.venv) (g.env) kirr@deca:~/src/tools/go/pygolang$ gdb python core ... Reading symbols from python... Reading symbols from /usr/lib/debug/.build-id/f9/02f8a561c3abdb9c8d8c859d4243bd8c3f928f.debug... [New LWP 218557] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `python -c import x'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _Py_INCREF (op=0x0) at /usr/include/python3.9/object.h:408 408 op->ob_refcnt++; (gdb) bt 5 #0 _Py_INCREF (op=0x0) at /usr/include/python3.9/object.h:408 cython#1 __pyx_f_1x__trace_tp_new (__pyx_v_cls=0x7f5ce75e6880 <__pyx_type_1x_X>, __pyx_v_args=(123,), __pyx_v_kw=0x0) at /home/kirr/src/tools/go/pygolang/x.c:1986 cython#2 0x000000000051dd7e in type_call (type=type@entry=0x7f5ce75e6880 <__pyx_type_1x_X>, args=args@entry=(123,), kwds=kwds@entry=0x0) at ../Objects/typeobject.c:1014 cython#3 0x00007f5ce75df8d4 in __Pyx_PyObject_Call (func=<type at remote 0x7f5ce75e6880>, arg=(123,), kw=0x0) at /home/kirr/src/tools/go/pygolang/x.c:3414 cython#4 0x00007f5ce75df276 in __pyx_pymod_exec_x (__pyx_pyinit_module=<optimized out>) at /home/kirr/src/tools/go/pygolang/x.c:3017 (More stack frames follow...) (gdb) f 1 cython#1 __pyx_f_1x__trace_tp_new (__pyx_v_cls=0x7f5ce75e6880 <__pyx_type_1x_X>, __pyx_v_args=(123,), __pyx_v_kw=0x0) at /home/kirr/src/tools/go/pygolang/x.c:1986 1986 __Pyx_INCREF(__pyx_v_kw); ``` -> Change newfunc signature to use PyObject* instead of object to fix it. With this fix, and test example updates to account for object -> PyObject* change as follows ... --- a/x.pyx.kirr +++ b/x.pyx @@ -1,5 +1,5 @@ # cython: language_level=3 -from cpython cimport newfunc, type as cpytype, Py_TYPE +from cpython cimport newfunc, type as cpytype, Py_TYPE, PyObject cdef class X: cdef int i @@ -10,8 +10,12 @@ cdef class X: cdef newfunc _orig_tp_new = Py_TYPE(X(0)).tp_new -cdef object _trace_tp_new(cpytype cls, object args, object kw): - print('_trace_tp_new', cls, args, kw) +cdef object xobject(PyObject* x): + return "null" if x == NULL else \ + <object>x + +cdef object _trace_tp_new(cpytype cls, PyObject* args, PyObject* kw): + print('_trace_tp_new', cls, xobject(args), xobject(kw)) return _orig_tp_new(cls, args, kw) Py_TYPE(X(0)).tp_new = _trace_tp_new ... it works as expected without crashing: $ python -c 'import x' _trace_tp_new <type 'x.X'> (123,) null X(123) Fixes: cython#4822

…d of object (#4823) object means the argument is always non-NULL valid Python object, while PyObject* argument can be generally NULL. If the argument is indeed passed as NULL, and we declare it as object, generated code will crash while trying to incref it. Quoting #4822: object.pxd currently declares `newfunc` as follows: ```pyx ctypedef object (*newfunc)(cpython.type.type, object, object) # (type, args, kwargs) ``` which implies that `args` and `kwargs` are always live objects and cannot be NULL. However Python can, and does, call tp_new with either args=NULL, or kwargs=NULL or both. And in such cases this leads to segfault in automatically-generated __Pyx_INCREF for args or kw. The fix is to change `object` to `PyObject*` for both args and kwargs. Please see below for details: ```cython # cython: language_level=3 from cpython cimport newfunc, type as cpytype, Py_TYPE cdef class X: cdef int i def __init__(self, i): self.i = i def __repr__(self): return 'X(%d)' % self.i cdef newfunc _orig_tp_new = Py_TYPE(X(0)).tp_new cdef object _trace_tp_new(cpytype cls, object args, object kw): print('_trace_tp_new', cls, args, kw) return _orig_tp_new(cls, args, kw) Py_TYPE(X(0)).tp_new = _trace_tp_new x = X(123) print(x) ``` ```console (neo) (py3.venv) (g.env) kirr@deca:~/src/tools/go/pygolang$ cythonize -i x.pyx Compiling /home/kirr/src/tools/go/pygolang/x.pyx because it changed. [1/1] Cythonizing /home/kirr/src/tools/go/pygolang/x.pyx running build_ext building 'x' extension ... x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kirr/src/wendelin/venv/py3.venv/include -I/usr/include/python3.9 -c /home/kirr/src/tools/go/pygolang/x.c -o /home/kirr/src/tools/go/pygolang/tmpqkz1r96s/home/kirr/src/tools/go/pygolang/x.o x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 /home/kirr/src/tools/go/pygolang/tmpqkz1r96s/home/kirr/src/tools/go/pygolang/x.o -o /home/kirr/src/tools/go/pygolang/x.cpython-39-x86_64-linux-gnu.so ``` ```console (neo) (py3.venv) (g.env) kirr@deca:~/src/tools/go/pygolang$ python -c 'import x' Ошибка сегментирования (стек памяти сброшен на диск) ``` ```console (neo) (py3.venv) (g.env) kirr@deca:~/src/tools/go/pygolang$ gdb python core ... Reading symbols from python... Reading symbols from /usr/lib/debug/.build-id/f9/02f8a561c3abdb9c8d8c859d4243bd8c3f928f.debug... [New LWP 218557] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `python -c import x'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _Py_INCREF (op=0x0) at /usr/include/python3.9/object.h:408 408 op->ob_refcnt++; (gdb) bt 5 #0 _Py_INCREF (op=0x0) at /usr/include/python3.9/object.h:408 #1 __pyx_f_1x__trace_tp_new (__pyx_v_cls=0x7f5ce75e6880 <__pyx_type_1x_X>, __pyx_v_args=(123,), __pyx_v_kw=0x0) at /home/kirr/src/tools/go/pygolang/x.c:1986 #2 0x000000000051dd7e in type_call (type=type@entry=0x7f5ce75e6880 <__pyx_type_1x_X>, args=args@entry=(123,), kwds=kwds@entry=0x0) at ../Objects/typeobject.c:1014 #3 0x00007f5ce75df8d4 in __Pyx_PyObject_Call (func=<type at remote 0x7f5ce75e6880>, arg=(123,), kw=0x0) at /home/kirr/src/tools/go/pygolang/x.c:3414 #4 0x00007f5ce75df276 in __pyx_pymod_exec_x (__pyx_pyinit_module=<optimized out>) at /home/kirr/src/tools/go/pygolang/x.c:3017 (More stack frames follow...) (gdb) f 1 #1 __pyx_f_1x__trace_tp_new (__pyx_v_cls=0x7f5ce75e6880 <__pyx_type_1x_X>, __pyx_v_args=(123,), __pyx_v_kw=0x0) at /home/kirr/src/tools/go/pygolang/x.c:1986 1986 __Pyx_INCREF(__pyx_v_kw); ``` -> Change newfunc signature to use PyObject* instead of object to fix it. With this fix, and test example updates to account for object -> PyObject* change as follows ... --- a/x.pyx.kirr +++ b/x.pyx @@ -1,5 +1,5 @@ # cython: language_level=3 -from cpython cimport newfunc, type as cpytype, Py_TYPE +from cpython cimport newfunc, type as cpytype, Py_TYPE, PyObject cdef class X: cdef int i @@ -10,8 +10,12 @@ cdef class X: cdef newfunc _orig_tp_new = Py_TYPE(X(0)).tp_new -cdef object _trace_tp_new(cpytype cls, object args, object kw): - print('_trace_tp_new', cls, args, kw) +cdef object xobject(PyObject* x): + return "null" if x == NULL else \ + <object>x + +cdef object _trace_tp_new(cpytype cls, PyObject* args, PyObject* kw): + print('_trace_tp_new', cls, xobject(args), xobject(kw)) return _orig_tp_new(cls, args, kw) Py_TYPE(X(0)).tp_new = _trace_tp_new ... it works as expected without crashing: $ python -c 'import x' _trace_tp_new <type 'x.X'> (123,) null X(123) Fixes: #4822

Address sanitizers consistently give an error on running the "parallel" test, saying that it ends up getting freed after releasing and reacquiring the gil, and thus we can't keep using the same threadstate pointer. Sanitizer finding: ==28037==ERROR: AddressSanitizer: heap-use-after-free on address 0x6130000325b8 at pc 0x7f9cb3123c0d bp 0x7f9cad6ddc10 sp 0x7f9cad6ddc08 READ of size 8 at 0x6130000325b8 thread T41 #0 0x7f9cb3123c0c in __Pyx__ExceptionReset <path>/TEST_TMP/run/c/parallel/parallel.c:42891 #1 0x7f9cb3172593 in __pyx_f_8parallel_parallel_exc_nogil_swallow._omp_fn.0 <path>/TEST_TMP/run/c/parallel/parallel.c:30050 #2 0x7f9cc66cdd2d (/lib64/libgomp.so.1+0x21d2d) (BuildId: 94e48b16f615cdab2143a0c2b3f15d0b8d81d0e6) #3 0x7f9cc688ff43 in start_thread (/lib64/libc.so.6+0x8ff43) (BuildId: 099807798c1de6cbe241dc4e3dd67a7326a2c29f) #4 0x7f9cc69184cb in __clone3 (/lib64/libc.so.6+0x1184cb) (BuildId: 099807798c1de6cbe241dc4e3dd67a7326a2c29f)

markflorisson added 19 commits December 14, 2010 00:45

Locate C source code properly and lex C source code in listings

0eb42da

Faster Python stepping using a watchpoint approach (f->f_lasti)

98c5444

Merge branch 'master' of https://github.com/cython/cython

bd41e48

Refactor inferior execution control code, better gdb message handling…

117ef04

… and re-fix runtests.py

Generic exception support

da24fe3

Detect Python exceptions

e325fbe

Disable watchpoint stepping by default (use cy step -w or cy step --w…

0544449

…atchpoint to enable) as hardware watchpoints are limited.

Fix Python stepping watchpoints + cy step repeatability

f0dd51b

Fix breakpoint approach for stepping (make stepping repeatable with b…

af9fbdb

…reakpoints that are set and deleted instead of disabled)

Made libpython.py compatible with gdb 7.1

db664ae

Remove tempfile

e5bb833

Debugger closure support

9140d0f

Debugger: Fix closure support + tests

7d98e4a

Debugger: Recognition of module-level Cython code (initmodulename and PyInit_modulename) Fix debug flag (import Parsing when needed)

Take first Cython step into function before reading variable in test

76567d0

Merge https://github.com/cython/cython

8193f74

Debugger: Disable optimization for test cases manually as they don't …

f4166d5

…use distutils.core.setup()

Debugger: Fix accessing cell variables in outer scopes

09c4d9a

Debugger: Detect whether the scope object and cell and free variables…

49623f8

… have been initialized for closures

Merge https://github.com/cython/cython

05533ff

This was referenced Aug 16, 2016

Closure support in gdb debugger robertwb/issues-import-test#802

Closed

Closure support in gdb debugger #1201

Closed

soreau mentioned this pull request Sep 10, 2017

Cython crash on 32bit #1854

Closed

wkerzendorf mentioned this pull request Jul 18, 2018

Segfault when cythonizing #2505

Closed

website-reader mentioned this pull request Jan 27, 2022

[BUG] Assigning an incorrect variable type from a pointer of a different type #4599

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comments

Closure debugger support#4

Closure debugger support#4
19 commits merged intocython:masterfrom
markflorisson:master

markflorisson commented Dec 28, 2010

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Comments

Conversation

markflorisson commented Dec 28, 2010

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

markflorisson commented Jan 14, 2011

Uh oh!

robertwb commented Jan 14, 2011

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants