/retest

🔨 rebuilding ci/circleci: coverage (failed build)

build failures seem unrelated.

TSAN failure:

ERROR: no such package '@com_github_grpc_grpc//bazel': java.io.IOException: Error downloading [https://github.com/grpc/grpc/archive/d8f4928fa779f6005a7fe55a176bdb373b0f910f.tar.gz] to /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/external/com_github_grpc_grpc/d8f4928fa779f6005a7fe55a176bdb373b0f910f.tar.gz: GET returned 504 Gateway Time-out

Coverage failure:

clang: error: unable to execute command: Killed
clang: error: clang frontend command failed due to signal (use -v to see invocation)

while compiling //test/common/router:router_test_lib_internal_only

/retest

🔨 rebuilding ci/circleci: coverage (failed build)

/azp run envoy-linux

Unfortunately I think the Windows CI failure is real. cc @wrowe

/wait

I had to change the patch to have this compile for me:

 bazel/foreign_cc/cares-win32-nameser.patch | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/bazel/foreign_cc/cares-win32-nameser.patch b/bazel/foreign_cc/cares-win32-nameser.patch
index 756c3933e..80694b747 100644
--- a/bazel/foreign_cc/cares-win32-nameser.patch
+++ b/bazel/foreign_cc/cares-win32-nameser.patch
@@ -1,12 +1,12 @@
---- CMakeLists.txt.orig        2020-02-19 14:42:47.978299400 -0500
-+++ CMakeLists.txt     2020-02-19 14:45:18.925903400 -0500
-@@ -652,6 +652,9 @@
- # Headers installation target
+--- CMakeLists.txt     2020-03-13 18:20:06.000000000 +1100
++++ CMakeLists_patch.txt       2020-03-16 08:50:04.370342688 +1100
+@@ -693,6 +693,9 @@
+ # Headers and Man Pages installation target
  IF (CARES_INSTALL)
        SET (CARES_HEADERS ares.h ares_version.h ares_dns.h "${PROJECT_BINARY_DIR}/ares_build.h" ares_rules.h)
 +      IF (WIN32)
 +              SET (CARES_HEADERS ${CARES_HEADERS} nameser.h)
 +      ENDIF()
-       INSTALL (FILES ${CARES_HEADERS} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
- ENDIF ()
+       INSTALL (FILES ${CARES_HEADERS} COMPONENT Devel DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})

+       # ManPages

@moderation Did you also update c-ares?

@derekargueta Yes. For some reason the c-ares and curl upgrades became intermingled due to the Windows PRs but I could be wrong.

For some reason the c-ares and curl upgrades became intermingled due to the Windows PRs

curl has always had a c-ares dependency, that's not windows specific.

I had to change the patch to have this compile for me:

On cursory review, that is the correct patch when c-ares is refreshed; the net behavior of the patch remains the same.

Unfortunately I think the Windows CI failure is real.

Confirmed, the new version of curl is not honoring our 'Bazel' build target. We are investigating why this is, from the build flags used by c-ares and listed in the curl CMakeCache.txt there is no indication of how this is newly broken.

We can clarify that curl build is behaving correctly.

The c-ares build was and continues to be invalid even after updating to 1.16.0. We are investigating and will report back shortly.

Thanks @wrowe! Without a Windows computer I'm completely blind here. It'd probably be best to update c-ares first then. Did a cursory attempt and got segfaults in tests 🙃so there's some non-Windows-specific debugging to do as well.

After resolving some other odds and ends here including resolution of the bad c-ares flags, we've determined that the invalid "-MD" flag is being injected in the curl component source file compilation commands, at the very end following the "-fPIC" toggle. This is tricky because we need to use "/MT", and the problem flag is generally mis-injected as "/MD"

We haven't been able to determine where/how this flag is injected yet. It doesn't initially appear to come from bazel, according to the rules_foreign_cc invocation scripts of cmake. There are cases in curl where -MD is used to emit dependencies at compilation time, but these should not be toggled or triggered except on very specific build environments.

Further updates Friday morning. With luck we can "pre-clean" some of these issues, at which point this c-ares update should go smoothly.

from a git bisect, looks like this is the bad commit from curl: curl/curl@fc9312f

Specifically this change: curl/curl@fc9312f#diff-af3b638bc2a3e6c650974192a53c7291R41 is the issue, we're investigating if updating cmake versions etc. can help and what exactly may be the difference in cmake policies etc.

We'll have a look at #10469 and see what the build system does with this build. Difficult to figure out how restricting our cmake 3.15.0-rc2 local environment to versions 3.0...3.16 exclusive suddenly broke all of the assumptions about the way that CFLAG's are merged, but there we are. The PR adds the cares and some other resolutions to reduce the chance of future breakage.

FYI - bumping c-ares is a very bad idea.

https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/35151/logs/68

I pointed out it breaks tests. It needs fixing but I don't know how that makes it a very bad idea.

The ASAN violation is concerning but the dns resolver relies on constructing/destructing quite a bit so I'm suspecting that it's correlated with that.

cc @junr03 who can look at the c-ares issues. It might also be a bug in our code. Can you do a PR to just bump that dep or maybe sync up with Jose on slack and he can do it? Thank you.

c-ares can certainly be bumped in 1.17.0 or whenever it's regressions are corrected. We can't have a broken master branch, though.

Bumping curl alone appears to work, #10469 resolves the windows issues, we've backed off breaking changes to OS/X and backed out the c-ares bump.