TST: including C source line coverage for CI / codecov - #11610

I think I'm stuck with Travis problems when compiling _configtest.c after adjusting the C compiler flags to include --coverage. Seems to work just fine locally.

Any suggestions from the experienced NumPy devs? Strange things that sometimes have to be adjusted when playing with compiler flags?

The error message is quite sensible, undefined reference to __gcov_init, so it looks like I'm part of the way there since there's recognition that I'm trying to use gcov, but something is also still missing, and adjusting to include --coverage in link-time flags had no apparent effect.

Setting LDFLAGS environment variable overrides default Python flags, see gh-11525, which probably is the cause for all problems. Finding out what the "default Python flags" are is somewhat complex and annoying, as you probably found out.

This is the reason why in Scipy, the gcov setup is done by setting CC/CXX/F77/F90 compiler executables and not touch the *FLAGS environment variables. With the recent numpy distutils change you can in principle do it via the *FLAGS, but setting the compilers may be more robust.

Of course, I'm wrong above, since the Scipy stanza for this reads

            import distutils.sysconfig
            cvars = distutils.sysconfig.get_config_vars()
            env['OPT'] = '-O0 -ggdb'
            env['FOPT'] = '-O0 -ggdb'
            env['CC'] = cvars['CC'] + ' --coverage'
            env['CXX'] = cvars['CXX'] + ' --coverage'
            env['F77'] = 'gfortran --coverage '
            env['F90'] = 'gfortran --coverage '
            env['LDSHARED'] = cvars['LDSHARED'] + ' --coverage'
            env['LDFLAGS'] = " ".join(cvars['LDSHARED'].split()[1:]) + ' --coverage'

You may be able to do it in an easier way by setting export NPY_DISTUTILS_APPEND_FLAGS=1, so that it's not necessary to figure out what the old flags were.

So, the CI is finally green and we can see the coverage report for C source files now. Looks like another 40,000 lines of code are tracked now, with about a 1 % drop from the pure Python cov report.

I need to clean up a bit here still, and I'm somewhat concerned that I had to retrieve the above link from the CI job "terminal" in Travis rather than being able to click on the links at the bottom of the PR (maybe because this is the first activation for those new files?) to see the new C files with highlights.

Other discussion point -- any concern about manipulating the "full coverage" CI job with coverage compiler flags (i.e., is it preferred to expand the matrix for a seperate C cov job or combining is ok?).

is it preferred to expand the matrix for a seperate C cov job or combining is ok?

combining seems fine to me. we already have a very large build matrix, and unless there's a reason to grow its size then it's better not to

I've removed debug code & cleaned up the commit history, but I'll leave the WIP tag for now because I'm still concerned about having to use the "Travis terminal link" to access the newly-covered files; perhaps one more slight adjustment is needed to get it just right for those links.

I've removed the WIP tag--I don't see any evidence of a specific issue with the added C coverage report now. The codecov links at the bottom of the page are for the diffs between commit hashes at head of master and the current branch--naturally, you can't diff on a file that isn't tracked in the master coverage report yet; clicking on the newer commit from this branch properly shows the coverage for C files.

you can't diff on a file that isn't tracked in the master coverage report yet; clicking on the newer commit from this branch properly shows the coverage for C files.

For those like me who have not seen this before, it is pretty neat. @tylerjereddy took me through the results here, for example to see src/umath/loops.c.src, do this (I think you need to be logged in on a desktop to see the "Show all checks" link:

• click on Show all checks to expand the test runs
• click on details for either of the codecov runs
• At the top note the Compare 8eed36a ... +0 ... 654a4c0, @tylerjereddy noted that the base commit has no coverage yet so click on the 654a4c0 changeset
• Then drill down Files->numpy->core->src->umath->loops.c.src. Once you get past the macro definitions, you begin to see where our tests do not cover code paths.

Rebased on latest master & force pushed to make sure this plays well with the setup.py ignore stuff for codecov (can't imagine it wouldn't though).

LGTM, any demurrers?

Well, let's see how this goes... Thanks @tylerjereddy .

Clicking through the badge on our main readme / landing page on github now shows the C source line coverage as well.

It's missing some files though, see
https://codecov.io/gh/numpy/numpy/tree/master/numpy/core/src/private
e.g. mem_overlap.c is not there.