Serious incompatibility problems w/ newer mingw-64 on Ubuntu #8653

sstone · 2016-09-02T13:23:41Z

you may have to select the posix version of gcc-mingw-w64 and g++-mingw-w64 to cross-build the latest code on ubuntu

paveljanik · 2016-09-02T13:45:07Z

doc/build-windows.md

    ./configure --prefix=`pwd`/depends/x86_64-w64-mingw32
    make

+On Ununtu 15.10 and above, if your build fails because the cross-compiler does not seem to support C++11 threads (you get errors such as "httpserver.cpp:69:10: error: ‘mutex’ in namespace ‘std’ does not name a type) you may have to select the posix versions of gcc-mingw-w64 and g++-mingw-w64:


NicolasDorier · 2016-09-06T09:02:38Z

tested ACK it solved my issue #8511

sstone · 2016-09-06T16:06:37Z

There might still be a problem with bitcoin-qt.exe which I'm investigating. bitcoind.exe builds and works fine (tested on a VM and several windows boxes). Sorry about that.
Update: I can build and run both bitcoin-qt.exe and bitcoind.exe if I use:

configure --disable-hardening --prefix=`pwd`/depends/x86_64-w64-mingw32

So far I have been able to test on Windows 10 only, so more feedback is welcome

When cross compiling for Windows on Ubuntu 15.10 and above, you may need to disable hardening if bitcoin-qt.exe fails to start properly

laanwj · 2016-09-13T08:21:47Z

doc/build-windows.md

+
+And select the posix version of the compiler.
+
+You may also need to disable "hardening" of compiled binaries if your bitcoin-qt.exe does not start properly by adding the --disable-hardening option when calling the configure script:


What, are you sure about this?

Can you be more specific about this problem? I'd prefer figuring out what the issue is and never telling people to disable hardening. The situation on windows is bad enough as it is (#8248). Note that the gitian executables are built with hardening enabled.

Sorry for not being specific enough: with hardening enabled, bitcoint-qt.exe does not start (it crashes on startup), but bitcoind.exe seems to work fine.
With hardening disabled, bitcoin-qt.exe starts and seems to work fine AFAICT.

Do remove this recommendation. You should never tell anyone in good conscience to disable hardening on a bitcoin wallet. Those security precautions exists with a good reason. Oh absolutely it will seem to "work fine" but you lose any protection against exploitation, if, say, another buffer overflow turns up in UPnP or openssl or any other library.

If it doesn't work with hardening then you need to debug why it doesn't work.

You're right. I was thinking mostly about people who might need to cross-build quickly for testing purposes and overlooked the fact that people might use the binaries as their real wallet instead of downloading an official release or using gitian to build. I should probably close this PR and open a new specific issue ?

laanwj · 2016-09-13T08:22:07Z

Very strong NACK on telling people to disable hardening, ACK on the posix alternatives.

jonasschnelli · 2016-09-13T08:47:35Z

Agree with @laanwj. Also, the disable hardening recommendation comes without a specific reason. Does the posix compatible mingw compiler breaks the Qt build? If so, why exactly?

btcdrak · 2016-09-13T08:51:53Z

doc/build-windows.md

+
+You may also need to disable "hardening" of compiled binaries if your bitcoin-qt.exe does not start properly by adding the --disable-hardening option when calling the configure script:
+
+    ./configure --disable-hardening --prefix=`pwd`/depends/x86_64-w64-mingw32


Why --disable-hardening ?

laanwj · 2016-09-14T10:39:34Z

To see if I could reproduce I did this:

Create a brand-new Ubuntu Xenial (16.04) VM, from today's (2015-09-14) cloud image
Install the necessary dependencies, and build requirements from these documents:

sudo apt-get install g++-mingw-w64-i686 mingw-w64-i686-dev g++-mingw-w64-x86-64 mingw-w64-x86-64-dev curl
sudo apt-get install build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils

While doing this I indeed saw some logging output about gcc alternatives: I wonder which one was selected by default. Apparently the -win32 variant, not the POSIX one.

Follow the instructions for building for windows 64-bit:

    cd depends
    make HOST=x86_64-w64-mingw32 -j4
    cd ..
    ./autogen.sh
    ./configure --prefix=`pwd`/depends/x86_64-w64-mingw32
    make

It fails with the following error, the same error that you get:

/usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/bits/unique_ptr.h:49:28: note: declared here
   template<typename> class auto_ptr;
                            ^
httpserver.cpp:69:10: error: ‘mutex’ in namespace ‘std’ does not name a type
     std::mutex cs;
          ^
httpserver.cpp:70:10: error: ‘condition_variable’ in namespace ‘std’ does not name a type
     std::condition_variable cond;

I changed the alternatives to -posix under sudo update-alternatives --config ... as suggested in this patch.
Cleaned and re-built bitcoin (but not the dependencies - maybe I should), and this time it passed.
Status (Windows 10):

bitcoin-test.exe: crashes on start
bitcoin-qt.exe: crashes on start
bitcoind.exe: seems to work, but throws bad_cast on exit
bitcoin-cli.exe: seems to work, but cannot connect to server?
bitcoin-tx.exe: seems to work

Going to clean and re-build the dependencies, as well as bitcoin entirely with the -posix compiler.
This didn't change anything, still same status as above.

laanwj · 2016-09-14T13:19:56Z

Whoa this even worse than I imagined. Above I already report that bitcoin-cli doesn't seem to work. Well here's why. TCPdump output of what comes back from the server:

15:17:02.157987 IP Acer-PC.lan.8332 > vm07.57962: Flags [P.], seq 1:446, ack 151, win 260, options [nop,nop,TS val 10525409 ecr 2673459], length 445
[email protected]. ..j..m.$|.............
.....(.3HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 14 Sep 2016 13:17:06 GMT
Content-Length: zu
Connection: close

{"result":{"version":139900,"protocolversion":70014,"blocks":418786,"timeoffset":-25,"connections":8,"proxy":"","difficulty":209453158595.381,"testnet":false,"relayfee":0.00001000,"errors":"This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"},"error":null,"id":1}

Yes, you read that correctly: the content length is 'zu'. ZU!

Also, bitcoind isn't stable, it crashed at least once. Unfortunately I didn't get a useful traceback.

~~I wouldn't recommend using POSIX emulation mode on Windows as a workaround here. It introduces more problems than it solves~~ (see later, is not related at all)

sstone · 2016-09-14T13:53:15Z

The issue with bitcoin-cli looks like https://sourceforge.net/p/levent/bugs/363/ which is linked to mingw (but not its POSIX mode specifically)

laanwj · 2016-09-14T14:42:27Z

OK I've done some research. In principle, std::mutex is supported in the -win32 compiler variant (which we've always used, and are using in Trusty). No POSIX emulation layer is needed for c++11 threads. However the problem with <mutex> and friends on newer Ubuntu's mingw-w64 seems to be the following:

Firstly, there is a /usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/mutex. However the definition of the actual class is guarded by _GLIBCXX_HAS_GTHREADS.
However it requires building glibcxx with a certain library (not to be confused with GLIB's "libgthread"):
- _GLIBCXX_HAS_GTHREADS is supposed to be defined in /usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/x86_64-w64-mingw32/bits/c++config.h when this has been done

This is an Ubuntu-specific problem: the "gthreads" (GCC threads) library is not linked into libcxx.

On Trusty this is not an issue as gthreads is provided:

/usr/include/c++/4.8/x86_64-w64-mingw32/bits/c++config.h defines _GLIBCXX_HAS_GTHREADS.

There is a https://github.com/meganz/mingw-std-threads which some people use to work around this problem. But in principle I think Ubuntu should solve this upstream.

laanwj · 2016-09-14T14:53:07Z

I have verified that after reverting the C++11 threads changes (https://github.com/laanwj/bitcoin/tree/2016_09_windows_hell), it builds normally with the default -win32 compiler. ~~bitcoin-qt then works, without having to disable hardening.~~
False positive: I was building without wallet, with the wallet it still crashes. What a shit.

However:

bitcoin-test.exe: crashes on start
bitcoin-qt.exe: crashes on start
bitcoind.exe: seems to work
bitcoin-cli.exe: seems to work, but cannot connect to server?
bitcoin-tx.exe: seems to work

The issue with bitcoin-cli looks like https://sourceforge.net/p/levent/bugs/363/ which is linked to mingw (but not its POSIX mode specifically)

This issue doesn't go away either, unfortunately.

sstone · 2016-09-14T16:02:46Z

FWIW, applying the libevent patch above does seem to fix bitcoin-cli.exe

laanwj · 2016-09-14T16:05:37Z

Looks like -fstack-protector-all is the specific hardening flag that exposes problems. Apparently it affects use of BerkeleyDB and boost::test. The tests pass (at least in wine) without --stack-protector-all. It also makes Bitcoin-Qt with wallet start again.

So apparently there are three, at most tangentially related issues regarding Windows:

The std::mutex std::condition_variable etc problem on Ubuntu 15.10+. Can apparently be solved by enabling POSIX compatibility, or by using something like https://github.com/meganz/mingw-std-threads.
-fstack-protector-all issue (-fstack-protector doesn't have this problem, neither do the other hardening flags like nxcompat) - causes the tests to crash at startup (before even getting to our code in BasicTestingSetup!), as well as bitcoin-qt to crash at startup if the wallet is enabled.
libevent 'zu' header issue - causes bitcoin-cli and bitcoind to be unable to connect over RPC (which as mentioned by @sstone, looks like https://sourceforge.net/p/levent/bugs/363/) (fixed in depends: Add libevent compatibility patch for windows #8730)

I wonder when each of these issues started. My only recommendation for now is: use an Ubuntu Trusty VM for windows cross-compilation

laanwj · 2016-09-14T16:13:12Z

FWIW, applying the libevent patch above does seem to fix bitcoin-cli.exe

Interesting. I wonder what happened in the meantime, I just checked and the 0.13.0 executables don't suffer from this issue. There have been no libevent version changes in depends since then (we've always been using 2.0.22). So it must be an issue with the combination of the new compiler and libevent.

I'm a bit afraid that the patch, if we were to apply it to depends, will introduce a similar issue (but backwards) on Trusty instead. But we can try and test.

achow101 · 2016-09-14T16:14:23Z

@laanwj where can I get a copy of your patches? I can build them in Ubuntu 16.04 and then test them on Windows 10.

@sstone

Add a patch that seems to be necessary for compatibilty of libevent 2.0.22 with recent mingw-w64 gcc versions (at least GCC 5.3.1 from Ubuntu 16.04). Without this patch the Content-Length in the HTTP header ends up as `Content-Length: zu`, causing communication between the RPC client and server to break down. See discussion in bitcoin#8653. Source: https://sourceforge.net/p/levent/bugs/363/ Thanks to @sstone for the suggestion.

theuni · 2016-09-19T16:27:36Z

FYI, I'm working on a large-scale toolchain refactor for depends so that we have more control over these things. Will report here when it's ready for testing.

fanquake · 2016-10-08T09:10:50Z

Closing this, I've copied all the important info into the #8732

@donaloconnor

…and Ubuntu 17.04 696ce46 [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 (fanquake) 4f890ba Add new step to clean $PATH var by removing /mnt specific Window's %PATH% paths that cause issues with the make system (Donal OConnor) Pull request description: This updates the Windows build documentation with the workaround required to build using Ubuntu 17.04 on WSL, and makes it's explicit that building on Ubuntu 16.04 is broken, and not recommended. This includes a commit from @donaloconnor in #11244, and is mostly the investigative work of @laanwj throughout #8732, #8653 and quite a few other issues. I tested building on 14.04, 16.04.3 and 17.04 [here](#11244 (comment)) and got the results we expect. --- Built master at c22a53c on a Windows 10 VM (Version 1607, OS Build 14393.1593) using WSL with Ubuntu 14.04. ![windows](https://user-images.githubusercontent.com/863730/30195033-867f1f24-9489-11e7-932c-e87b8764a627.png) Upgraded WSL to 16.04.3, and tried building c22a53c using these instructions. The result is as expected. ![ubuntu 16 04 3](https://user-images.githubusercontent.com/863730/30235670-b9bf36bc-953d-11e7-8c1d-4debf7113032.png) Upgraded WSL to 17.04 and tried building 3255d63 using these instructions. ![ubuntu 17 04](https://user-images.githubusercontent.com/863730/30235669-b7473434-953d-11e7-8ea3-d05a319ae2d4.png) If someone else could also verify that builds are working on both 14.04 and 17.04 with these instructions, that would be great. Tree-SHA512: 866f1003eb45d208d8ae849504f54fc2f27c32240129d2124ce5a2ee7167bcbf062d29f23b1745123f532ffd0253a8611e719b2a316d1331d3c3924f91e7775d

7383d77 Updated instructions for Windows 10 Fall Creators Update. (Aaron Clauson) e0fc4a7 Updated Windows build doc for WSL/Xenial workarounds. (Aaron Clauson) Pull request description: An update to the Windows build document that provides workarounds for the broken 64 bit mingw32 cross compiler on WSL/Xenial. This update is an alternative to pull request #11437. While that pull request takes a valid approach by stating building on WSL should be avoided I think it is more useful to give Windows developers a workaround option. The instructions have been tested on: - Ubuntu 14.04 and 64 bit mingw32 tool chain - Ubuntu 16.04 and 64 bit mingw32 tool chain - Ubuntu 17.04 and 64 bit mingw32 tool chain - Windows Subsystem for Linux (Windows 10 OS Build 15063.608) and 32 bit mingw32 tool chain - Windows Subsystem for Linux (Windows 10 OS Build 15063.608) and 64 bit mingw32 tool chain Related items: - Serious incompatibility problems w/ newer mingw-64 on Ubuntu #8653 - `-fstack-protector-all` triggers crashes in mingw-w64 5.3.1 #8732 - Windows build appears broken on WSL (buntu okay) #10269 - Compilation error for windows target #11437 Tree-SHA512: 7c937e37ed7120ae5dcf61aba50e5228a7ed6f729647c724b8f48e7cbbd81366c1a83a818618766a8fe0418425e05ba2eba2b14f2616621c58606585444f45fc

@sstone

* depends: fix fontconfig with newer glibc See comment for more detail * Create dependencies.md, and link dependencies file from README & build docs * [Docs] Fix broken Markdown table in dependencies.md. Cleanups. Use the correct capitalization for the dependencies Sort dependencies Fix header formatting. Minor style cleanups. * Add required package dependencies for depends cross compilation [skip-ci] * [depends] expat 2.2.5 * [depends] ccache 3.4.1 * [depends] miniupnpc 2.0.20180203 * Update mac_alias to 2.0.7 * fixme: depends: Add D_DARWIN_C_SOURCE to miniupnpc CFLAGS * update config.guess and config.sub to latest version * [depends] Fix Qt compilation with Xcode 8 * [depends] ZeroMQ 4.2.2 * [depends] Qt 5.7.1 * [depends] Don't build libevent sample code * [depends] native_ds_store 1.1.2 * depends: fix zmq build with mingw < 4.0 * depends: fix libzmq's needless linking against libstdc++ This is broken for a number of reasons, including: - g++ understands "-static-libstdc++ -lstdc++" to mean "link against whatever libstdc++ exists, probably shared", which in itself is buggy. - another stdlib (libc++ for example) may be in use * [depends] Allow depends system to support armv7l * depends: fix qt translations build Their buildsystem insists on using the installed ltranslate, but gets confused about how to find it. Since we manually control the build order, just drop the dependency. * depends: qt: disable printer for all platforms, not just osx This also fixes the native osx build. * depends: add a zlib build qt5.7 changed the location of some of its symbols, creating a circular dependency in Qt5Core. Rather than trying to fix that up, build our own zlib rather than having it built for us. * qt: fix build with zlib for target This contains a few hacks very specific to Qt's buildsystem. These can be reverted once we split the build between native and target builds. Qt's build contains a circular dependency when not using a system zlib. By far the easiest fix is to switch to a system zlib, rather than Qt's own. However, that confuses Qt's cross build which assumes that when using a system zlib, it should also find a system (native) zlib for native tools. The build breaks if that zlib is not present. To solve this: 1. Always use a system zlib rather than the one provided by qt 2. Set force_bootstrap, which instructs the build tools to be built as though we're cross-compiling (build != target) 3. For build tools, use qt's internal zlib so that a native zlib is not required. Step 3 means that if any zlib headers are found by the native build, it will confuse Qt's internal zlib build. So we also need to make sure that the target headers/libs aren't found. To do so, specify that our cflags/cxxflags/cppflags/ldflags only apply for non-host builds. * [depends] dbus 1.10.18 * [Depends] Fix Qt build with Xcode 9.2 * depends: zeromq 4.2.3 * depends: patch pthread_set_name_np out of zeromq * depends: Fix Qt build with XCode 9.3 * depends: Add libevent compatibility patch for windows Add a patch that seems to be necessary for compatibilty of libevent 2.0.22 with recent mingw-w64 gcc versions (at least GCC 5.3.1 from Ubuntu 16.04). Without this patch the Content-Length in the HTTP header ends up as `Content-Length: zu`, causing communication between the RPC client and server to break down. See discussion in bitcoin#8653. Source: https://sourceforge.net/p/levent/bugs/363/ Thanks to @sstone for the suggestion. * [depends] Set OSX_MIN_VERSION to 10.8 * depends: use new variable layout for qt sdk * [depends] Boost 1.64.0 * [depends] libevent 2.1.8-stable * depends: Add 'make clean' and 'make clean-all' rules It's useful to have a standard way to clean up the work done by the depends system when testing changes to it. The `make clean-all` rule removes build artifacts for all supported architectures (in addition to sources/), while `make clean` only removes artifacts for current architecture (`BUILD`). * [depends] bump nativce_ccache to 3.4.2 * [depends] remove libevent patches (not needed since 2.1.7rc) * [depends] Fix zeromq make step * [depends] Remove OBJCXX define from config.site.in * [depends] fontconfig 2.12.1 * [depends] FreeType 2.7.1 * Update doc/dependencies.md

@donaloconnor

…ng WSL and Ubuntu 17.04 696ce46 [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 (fanquake) 4f890ba Add new step to clean $PATH var by removing /mnt specific Window's %PATH% paths that cause issues with the make system (Donal OConnor) Pull request description: This updates the Windows build documentation with the workaround required to build using Ubuntu 17.04 on WSL, and makes it's explicit that building on Ubuntu 16.04 is broken, and not recommended. This includes a commit from @donaloconnor in bitcoin#11244, and is mostly the investigative work of @laanwj throughout bitcoin#8732, bitcoin#8653 and quite a few other issues. I tested building on 14.04, 16.04.3 and 17.04 [here](bitcoin#11244 (comment)) and got the results we expect. --- Built master at bitcoin@c22a53c on a Windows 10 VM (Version 1607, OS Build 14393.1593) using WSL with Ubuntu 14.04. ![windows](https://user-images.githubusercontent.com/863730/30195033-867f1f24-9489-11e7-932c-e87b8764a627.png) Upgraded WSL to 16.04.3, and tried building bitcoin@c22a53c using these instructions. The result is as expected. ![ubuntu 16 04 3](https://user-images.githubusercontent.com/863730/30235670-b9bf36bc-953d-11e7-8c1d-4debf7113032.png) Upgraded WSL to 17.04 and tried building bitcoin@3255d63 using these instructions. ![ubuntu 17 04](https://user-images.githubusercontent.com/863730/30235669-b7473434-953d-11e7-8ea3-d05a319ae2d4.png) If someone else could also verify that builds are working on both 14.04 and 17.04 with these instructions, that would be great. Tree-SHA512: 866f1003eb45d208d8ae849504f54fc2f27c32240129d2124ce5a2ee7167bcbf062d29f23b1745123f532ffd0253a8611e719b2a316d1331d3c3924f91e7775d

doc (trivial): add tip for cross-builds on ubuntu

f17d19e

you may have to select the posix version of gcc-mingw-w64 and g++-mingw-w64 to cross-build the latest code on ubuntu

paveljanik reviewed Sep 2, 2016
View reviewed changes

fanquake added the Docs label Sep 2, 2016

fix typo

be5b9f4

sstone mentioned this pull request Sep 4, 2016

Compilation error for windows target #8511

Closed

sstone changed the title ~~doc (trivial): add tip for cross-builds on ubuntu~~ [Trivial] doc: add tip for cross-builds on ubuntu Sep 5, 2016

fanquake added the Windows label Sep 9, 2016

cross-compiling: disable hardening if needed

d70597f

When cross compiling for Windows on Ubuntu 15.10 and above, you may need to disable hardening if bitcoin-qt.exe fails to start properly

laanwj reviewed Sep 13, 2016
View reviewed changes

btcdrak reviewed Sep 13, 2016
View reviewed changes

laanwj changed the title ~~[Trivial] doc: add tip for cross-builds on ubuntu~~ Serious incompatibility problems w/ newer mingw-64 on Ubuntu Sep 14, 2016

This was referenced Sep 14, 2016

depends: Add libevent compatibility patch for windows #8730

Merged

-fstack-protector-all triggers crashes in mingw-w64 5.3.1 #8732

Closed

This was referenced Sep 21, 2016

Enable (and check for) 64-bit ASLR on Windows #8249

Merged

[Doc] Add missing autogen to example builds #8787

Merged

fanquake closed this Oct 8, 2016

laanwj mentioned this pull request Aug 22, 2017

compilation error #11104

Closed

fanquake mentioned this pull request Oct 2, 2017

[Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 #11437

Merged

sipsorcery mentioned this pull request Oct 2, 2017

Updated Windows build doc for WSL/Xenial workaround #11438

Merged

sstone deleted the wip-doccrosscompile branch May 28, 2021 16:24

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022


		And select the posix version of the compiler.

		You may also need to disable "hardening" of compiled binaries if your bitcoin-qt.exe does not start properly by adding the --disable-hardening option when calling the configure script:


		You may also need to disable "hardening" of compiled binaries if your bitcoin-qt.exe does not start properly by adding the --disable-hardening option when calling the configure script:

		./configure --disable-hardening --prefix=`pwd`/depends/x86_64-w64-mingw32

Serious incompatibility problems w/ newer mingw-64 on Ubuntu #8653

Serious incompatibility problems w/ newer mingw-64 on Ubuntu #8653

Uh oh!

Conversation

sstone commented Sep 2, 2016

Uh oh!

paveljanik Sep 2, 2016

Choose a reason for hiding this comment

Uh oh!

NicolasDorier commented Sep 6, 2016

Uh oh!

sstone commented Sep 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj Sep 13, 2016

Choose a reason for hiding this comment

Uh oh!

sstone Sep 13, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj Sep 14, 2016

Choose a reason for hiding this comment

Uh oh!

sstone Sep 14, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj commented Sep 13, 2016

Uh oh!

jonasschnelli commented Sep 13, 2016

Uh oh!

btcdrak Sep 13, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sstone commented Sep 14, 2016

Uh oh!

laanwj commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sstone commented Sep 14, 2016

Uh oh!

laanwj commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

achow101 commented Sep 14, 2016

Uh oh!

theuni commented Sep 19, 2016

Uh oh!

fanquake commented Oct 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

sstone commented Sep 6, 2016 •

edited

Loading

laanwj commented Sep 14, 2016 •

edited

Loading

laanwj commented Sep 14, 2016 •

edited

Loading

laanwj commented Sep 14, 2016 •

edited

Loading

laanwj commented Sep 14, 2016 •

edited

Loading

laanwj commented Sep 14, 2016 •

edited

Loading

laanwj commented Sep 14, 2016 •

edited

Loading