net: Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s #12569

practicalswift · 2018-02-28T20:34:29Z

Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s.

Before this patch:

$ src/bitcoind -printtoconsole
…
2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo)
2018-02-28 18:37:52 connect() 10.11.21.34:18333 failed after select(): Connection refused (111)
2018-02-28 18:43:22 connect() to 10.11.43.14:18333 failed after select(): Network is unreachable (101)
2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo)
2018-02-28 18:46:54 connect() to [2001:0:9d38:78ff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101)
2018-02-28 18:48:56 connect() to [2001:0:9d38:6aff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101)
2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo)

After this patch:

$ src/bitcoind -printtoconsole
…
2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo)
2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo)
2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo)

Please note that "manual connect():s" (invoked via -connect, -proxy or addnode) are still reported at the default log level as these messages are likely to be relevant to end-users:

$ src/bitcoind -printtoconsole -connect=127.0.0.1:1234
…
2018-02-28 18:31:13 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111)

$ src/bitcoind -printtoconsole -proxy=127.0.0.1:1234
…
2018-02-28 18:32:32 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111)

$ src/bitcoind -printtoconsole &
$ src/bitcoin-cli addnode 127.0.0.1:1234 onetry
…
2018-02-28 18:33:40 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111)

randolf · 2018-02-28T21:38:08Z

Just to confirm: With a higher debug level set, all those "connect" errors can still be logged?

practicalswift · 2018-02-28T21:42:42Z

@randolf Exactly :-)

Empact · 2018-03-01T00:18:01Z

src/net.cpp

FYI the style guide prefers the prior style:
"Variable and namespace names are all lowercase, and may use _ to separate words (snake_case)"

Empact · 2018-03-01T00:22:44Z

src/netbase.cpp

Maybe extract a LogConnectFailure method to DRY these two up?

practicalswift · 2018-03-09T15:17:02Z

@Empact Thanks for reviewing. Updated in light of review. Please re-review :-)

instagibbs · 2018-03-09T20:15:40Z

src/netbase.cpp

not totally clear to me this should be true?

I admit I don't know the networking code as well.

This is the connection to the proxy server. Since the proxy server is specified by the user it is handled as a "manual" connection since any connection errors are likely of relevance to the end-user and thus should be logged. Makes sense? :-)

eklitzke

Concept ACK. Regarding argument forwarding, there's related code for FatalError() in #11857 that I think you could re-use here.

eklitzke · 2018-03-11T08:29:16Z

src/netbase.cpp

I know this is annoying, but can you use argument forwarding or make this variadic? That would avoid allocation when the message is not going to be logged.

…en logging failed non-manual connect():s Before this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28T18:37:52Z connect() 10.11.21.34:18333 failed after select(): Connection refused (111) 2018-02-28 18:43:22 connect() to 10.11.43.14:18333 failed after select(): Network is unreachable (101) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:46:54 connect() to [2001:0:9d38:78ff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:48:56 connect() to [2001:0:9d38:6aff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` After this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` Please note that "manual connect():s" (invoked via `-connect`, `-proxy` or `addnode`) are still reported as usual: ``` $ src/bitcoind -printtoconsole -connect=10.11.12.13 … 2018-02-28 18:33:13 connect() to 10.11.12.13:18333 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole -proxy=10.11.12.13 … 2018-02-28 18:32:32 connect() to 10.11.12.13:9050 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole & $ src/bitcoin-cli addnode "10.11.12.13" onetry … 2018-02-28 18:34:40 connect() to 10.11.12.13:18333 failed after select(): Connection refused (111) ```

practicalswift · 2018-03-11T09:50:00Z

@eklitzke Good point! Updated. Please re-review :-)

eklitzke · 2018-03-13T05:19:27Z

That change looks good, but one other thing occurred to me just now. I think we already have special logic for handling peers added using addnode, and that should definitely be treated as a "manual" connection case. Can you take a look at that?

practicalswift · 2018-03-13T06:58:16Z

@eklitzke It already is:

$ src/bitcoind -printtoconsole &
$ src/bitcoin-cli addnode 127.0.0.1:1234 onetry
…
2018-02-28 18:33:40 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111)

:-)

sipa · 2018-03-17T00:21:19Z

utACK cba2800

eklitzke · 2018-03-17T01:16:14Z

utACK cba2800

minhhoang123

Good

practicalswift · 2018-03-28T02:38:42Z

@laanwj Would it be possible to get this PR tagged with a 0.17.0 milestone? :-)

ajtowns · 2018-04-12T09:15:00Z

utACK cba2800

…usting log level when logging failed non-manual connect():s cba2800 Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s (practicalswift) Pull request description: Increase signal-to-noise ratio in `debug.log` by adjusting log level when logging failed non-manual `connect()`:s. Before this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:37:52 connect() 10.11.21.34:18333 failed after select(): Connection refused (111) 2018-02-28 18:43:22 connect() to 10.11.43.14:18333 failed after select(): Network is unreachable (101) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:46:54 connect() to [2001:0:9d38:78ff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:48:56 connect() to [2001:0:9d38:6aff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` After this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` Please note that "manual `connect()`:s" (invoked via `-connect`, `-proxy` or `addnode`) are still reported at the default log level as these messages are likely to be relevant to end-users: ``` $ src/bitcoind -printtoconsole -connect=127.0.0.1:1234 … 2018-02-28 18:31:13 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole -proxy=127.0.0.1:1234 … 2018-02-28 18:32:32 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole & $ src/bitcoin-cli addnode 127.0.0.1:1234 onetry … 2018-02-28 18:33:40 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) ``` Tree-SHA512: 92e3c1e4b54ce8ccdd7ec31de147c8505710cd799ceb2bbc8576a086709967802403c9184df364b3cfa59bd98859f6ac8feb27fb09b9324194c6c47a042fc6d3

… by adjusting log level when logging failed non-manual connect():s cba2800 Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s (practicalswift) Pull request description: Increase signal-to-noise ratio in `debug.log` by adjusting log level when logging failed non-manual `connect()`:s. Before this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:37:52 connect() 10.11.21.34:18333 failed after select(): Connection refused (111) 2018-02-28 18:43:22 connect() to 10.11.43.14:18333 failed after select(): Network is unreachable (101) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:46:54 connect() to [2001:0:9d38:78ff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:48:56 connect() to [2001:0:9d38:6aff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` After this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` Please note that "manual `connect()`:s" (invoked via `-connect`, `-proxy` or `addnode`) are still reported at the default log level as these messages are likely to be relevant to end-users: ``` $ src/bitcoind -printtoconsole -connect=127.0.0.1:1234 … 2018-02-28 18:31:13 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole -proxy=127.0.0.1:1234 … 2018-02-28 18:32:32 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole & $ src/bitcoin-cli addnode 127.0.0.1:1234 onetry … 2018-02-28 18:33:40 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) ``` Tree-SHA512: 92e3c1e4b54ce8ccdd7ec31de147c8505710cd799ceb2bbc8576a086709967802403c9184df364b3cfa59bd98859f6ac8feb27fb09b9324194c6c47a042fc6d3 fix compilation Signed-off-by: pasta <[email protected]>

…okup 95bb364 BugFix: activemasternode::init, DMN connectivity check was missing to set the error cause. (furszy) 6e9b6ee trivial: activemasternode clang-tidy cleanups (furszy) 6f4e17c increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s (practicalswift) 889ce4a Fix CActiveMasternodeManager connectivity test (furszy) 34363a5 Fix CActiveMasternodeManager::GetLocalAddress to prefer IPv4 if multiple local addresses are known (UdjinM6) Pull request description: Correcting two issues on the Deterministic Masternode operator side: 1) The DMN service address connectivity check was not being performed (due a non-initialized socket), failing all-the-time on testnet and mainnet. Erring out at startup. 2) `CActiveMasternodeManager::GetLocalAddress` will prefer IPv4 if there are multiple addresses available locally. --> dashpay#3304. 3) `CActiveMasternodeManager::Init` connectivity check was missing to set the error cause. Plus, now that i'm here, ported bitcoin#12569 as well. ACKs for top commit: random-zebra: ACK 95bb364 Fuzzbawls: ACK 95bb364 Tree-SHA512: 1b1616ddae8f73832d0e5d9b7a30714e2bee223a4348a453d8802371becab073421c105d47974698ddb19a087967892ac703a25413942139170e098f68827203

… by adjusting log level when logging failed non-manual connect():s cba2800 Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s (practicalswift) Pull request description: Increase signal-to-noise ratio in `debug.log` by adjusting log level when logging failed non-manual `connect()`:s. Before this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:37:52 connect() 10.11.21.34:18333 failed after select(): Connection refused (111) 2018-02-28 18:43:22 connect() to 10.11.43.14:18333 failed after select(): Network is unreachable (101) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:46:54 connect() to [2001:0:9d38:78ff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:48:56 connect() to [2001:0:9d38:6aff:1234:1234:1234:1234]:18333 failed: Network is unreachable (101) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` After this patch: ``` $ src/bitcoind -printtoconsole … 2018-02-28 18:42:51 UpdateTip: new best=0000000000005448b10a219683d34b770a28044e1cc421032dea1a79ff548948 height=1286903 version=0x20000000 log2_work=69.791313 tx=17408546 date='2018-02-28 18:42:46' progress=1.000000 cache=0.0MiB(173txo) 2018-02-28 18:44:49 UpdateTip: new best=000000000000029a521ff2803e1441b09413b876accff5084a4cccf7747d798b height=1286904 version=0x20000000 log2_work=69.791345 tx=17408559 date='2018-02-28 18:44:51' progress=1.000000 cache=0.1MiB(502txo) 2018-02-28 18:49:11 UpdateTip: new best=000000000000000206b79eb235e5dd907b6369de0e5d764330bf40ec0d460311 height=1286905 version=0x20000000 log2_work=69.791377 tx=17408577 date='2018-02-28 18:49:12' progress=1.000000 cache=1.0MiB(5245txo) ``` Please note that "manual `connect()`:s" (invoked via `-connect`, `-proxy` or `addnode`) are still reported at the default log level as these messages are likely to be relevant to end-users: ``` $ src/bitcoind -printtoconsole -connect=127.0.0.1:1234 … 2018-02-28 18:31:13 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole -proxy=127.0.0.1:1234 … 2018-02-28 18:32:32 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) $ src/bitcoind -printtoconsole & $ src/bitcoin-cli addnode 127.0.0.1:1234 onetry … 2018-02-28 18:33:40 connect() to 127.0.0.1:1234 failed after select(): Connection refused (111) ``` Tree-SHA512: 92e3c1e4b54ce8ccdd7ec31de147c8505710cd799ceb2bbc8576a086709967802403c9184df364b3cfa59bd98859f6ac8feb27fb09b9324194c6c47a042fc6d3 fix compilation Signed-off-by: pasta <[email protected]>

practicalswift changed the title ~~Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s~~ net: Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s Feb 28, 2018

randolf approved these changes Feb 28, 2018

View reviewed changes

Empact reviewed Mar 1, 2018

View reviewed changes

src/netbase.cpp Outdated

Copy link

Contributor

Empact Mar 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe extract a LogConnectFailure method to DRY these two up?

fanquake added the Utils/log/libs label Mar 1, 2018

practicalswift force-pushed the logging-signal-to-noise branch 2 times, most recently from 1bcdf2d to 84b4fe3 Compare March 9, 2018 15:15

instagibbs reviewed Mar 9, 2018

View reviewed changes

eklitzke reviewed Mar 11, 2018

View reviewed changes

practicalswift force-pushed the logging-signal-to-noise branch from 84b4fe3 to cba2800 Compare March 11, 2018 09:49

minhhoang123 suggested changes Mar 28, 2018

View reviewed changes

eklitzke approved these changes Mar 28, 2018

View reviewed changes

laanwj merged commit cba2800 into bitcoin:master Apr 13, 2018

practicalswift deleted the logging-signal-to-noise branch April 10, 2021 19:33

furszy mentioned this pull request Feb 10, 2022

Fix DMN operator connectivity check and local address lookup PIVX-Project/PIVX#2742

Merged

bitcoin locked as resolved and limited conversation to collaborators Aug 18, 2022

net: Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s #12569

net: Increase signal-to-noise ratio in debug.log by adjusting log level when logging failed non-manual connect():s #12569

Uh oh!

Conversation

practicalswift commented Feb 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

randolf commented Feb 28, 2018

Uh oh!

practicalswift commented Feb 28, 2018

Uh oh!

Empact Mar 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Empact Mar 1, 2018

Choose a reason for hiding this comment

Uh oh!

practicalswift commented Mar 9, 2018

Uh oh!

instagibbs Mar 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

practicalswift Mar 10, 2018

Choose a reason for hiding this comment

Uh oh!

eklitzke left a comment

Choose a reason for hiding this comment

Uh oh!

eklitzke Mar 11, 2018

Choose a reason for hiding this comment

Uh oh!

practicalswift commented Mar 11, 2018

Uh oh!

eklitzke commented Mar 13, 2018

Uh oh!

practicalswift commented Mar 13, 2018

Uh oh!

sipa commented Mar 17, 2018

Uh oh!

eklitzke commented Mar 17, 2018

Uh oh!

minhhoang123 left a comment

Choose a reason for hiding this comment

Uh oh!

practicalswift commented Mar 28, 2018

Uh oh!

ajtowns commented Apr 12, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

practicalswift commented Feb 28, 2018 •

edited

Loading

Empact Mar 1, 2018 •

edited

Loading

instagibbs Mar 9, 2018 •

edited

Loading