[CIVIS-2892] datadog_cov native extension for per test code coverage by anmarchenko · Pull Request #137 · DataDog/datadog-ci-rb

anmarchenko · 2024-03-14T15:48:01Z

What does this PR do?
Adds datadog_cov native extension for per test code coverage.

Usage of Datadog::CI::ITR::Coverage::DDCov:

# track files coverage in the current folder
cov = Datadog::CI::ITR::Coverage::DDCov.new(root: Dir.pwd)
cov.start_coverage
Calculator.new.add(1, 2) # => 3
cov.stop_coverage # => { "/path/to/current-folder/calculator.rb" => true }

# track lines coverage
cov = Datadog::CI::ITR::Coverage::DDCov.new(root: Dir.pwd, mode: :lines)
cov.start_coverage
Calculator.new.add(1, 2) # => 3
cov.stop_coverage # => { "/path/to/current-folder/calculator.rb" => {12 => true, 13 => true} }

Additional Notes
Current limitations:

UTF-8 in source file paths is not supported because of additional overhead of parsing UTF-8 strings: will be added later as an opt-in via environment variable as non-ASCII characters in paths and filenames are rare
Lines coverage is barebones and not optimized: ITR will use only files coverage in the first stage

How to test the change?
Unit tests are provided in spec/datadog/ci/cov_spec.rb

Scenarios tested:

coverage with different root values
root folder path is longer than covered files' paths
coverage is not collected after stop_coverage call
coverage of mixed in code from helpers
multi threaded coverage (coverage is collected per thread)
UTF-8 in file names does not fail (but produces incorrect result)

codecov-commenter · 2024-03-14T16:24:54Z

Codecov Report

❌ Patch coverage is 99.57447% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.22%. Comparing base (6a994b6) to head (e4e86f3).

Files with missing lines	Patch %	Lines
lib/datadog/ci/utils/git.rb	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              1.0     #137      +/-   ##
==========================================
+ Coverage   99.19%   99.22%   +0.02%     
==========================================
  Files         163      171       +8     
  Lines        7240     7468     +228     
  Branches      302      310       +8     
==========================================
+ Hits         7182     7410     +228     
  Misses         58       58

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

juan-fernandez

🎉 very nice!

tonyredondo

LGTM, I left a couple of nits, and I have a question regarding what happens if a test spawn new threads, is that possible? how the code coverage works in that case?

ivoanjo

Left a few notes! :)

ivoanjo

Ahaha clearly my head kept reviewing the code as an async background task and when I took a break to go get some tea the async task resolved and pointed out these two issues 🤣

ivoanjo

Left a few more notes, I think you're very close ;)

One thing that occurred to me is that we have a weird setup for packaging dd-trace-rb when used by single step instrumentation (sometimes known as "lib injection" in dd-trace-rb).

TL;DR we handle gems with native extensions in a specific way, so if this native gem is still going to become a dependency of ddtrace 1.x, it's probably worth checking nothing is broken there.

I'm not entirely sure of all the details, @TonyCTHsu is the right person to ask "if this new native extension going to break or not".

Git.root returns Dir.pwd if git is not available

…in later

… ls-files output

anmarchenko · 2024-03-20T15:53:48Z

This pull request is now rebased on top of 1.0 branch and the base is set to 1.0.

From now on, all ITR functionality will go into 1.0 branch so that we don't add additional risks for ddtrace 1.x gem.

…s pass

… is ITR-only functionality

anmarchenko · 2024-03-21T09:02:12Z

@ivoanjo this is ready for another pass

ivoanjo

👍 Native bits look good to me! I didn't re-review the other things that came in from the rebase/changing the branch :)

anmarchenko marked this pull request as ready for review March 15, 2024 14:00

anmarchenko requested review from a team as code owners March 15, 2024 14:00

anmarchenko requested a review from nikita-tkachenko-datadog March 15, 2024 14:00

anmarchenko force-pushed the anmarchenko/ddcov_per_test_coverage branch from c1ab331 to 29a064b Compare March 18, 2024 15:16

juan-fernandez approved these changes Mar 19, 2024

View reviewed changes

tonyredondo approved these changes Mar 19, 2024

View reviewed changes

Comment thread ext/datadog_cov/datadog_cov.c

Comment thread .gitignore

ivoanjo reviewed Mar 19, 2024

View reviewed changes

Comment thread ext/datadog_cov/datadog_cov.c Outdated

Comment thread ext/datadog_cov/datadog_cov.c Outdated

anmarchenko force-pushed the anmarchenko/ddcov_per_test_coverage branch from 2f21495 to 09c27e5 Compare March 20, 2024 11:36

ivoanjo reviewed Mar 20, 2024

View reviewed changes

Comment thread ext/datadog_cov/datadog_cov.c

Comment thread ext/datadog_cov/datadog_cov.c Outdated

Comment thread ext/datadog_cov/datadog_cov.c Outdated

anmarchenko added 18 commits March 20, 2024 16:47

Add 0.8.3 to CHANGELOG.md

f989c49

Bump version 0.8.2 to 0.8.3

c1cb0c3

Update lockfiles for datadog-ci 0.8.3

228ad44

datadog_cov native extension

cfc5ebc

test datadog_cov extension

717cb0d

fix bool type compilation error

7c51a7a

exclude native binary artifacts from gemspec

6cdaa9f

don't run code coverage specs on JRuby

3bc3725

do not require native extension on JRuby

fb9e4aa

try again

b0a1847

don't use class name in top-level describe

f60bb79

ITR::Runner has thread local coverage collector

cfe2f9c

Git.root returns Dir.pwd if git is not available

gemfile updates

7e87934

ITR code coverage start/stop in Recorder

6d5380c

freeze source file name to reduce object allocations

5ef7b1e

frozen strings do not make it faster

cc6fcf2

require native extension on ITR configuration step

8aa5261

add Datadog::CI::Cov placeholder

80a987e

anmarchenko added 16 commits March 20, 2024 16:47

fix jruby tests

bdc6430

multithreaded code coverage and mixins tests

40a7ee5

note that UTF-8 paths are not supported yet, will be added as an opt-…

84b7469

…in later

specs for ITR::Runner#start_coverage

911c4fe

add more specs, try to fix release spec with forcing encoding for git…

f2817f6

… ls-files output

exclude cov_spec in JRUby completely

8a55463

move cov tests and skip them completely on JRuby

f9b8917

do not compile native extension on unsupported platforms

19bf537

check that dd_cov_data->root pointer is not NULL to avoid segfault

0abf879

minor fixes

0d40a56

store root as RString; convert to char* before using it for prefix check

5bad369

initialize dd_cov_data->mode field when allocating dd_cov_data struct

a1562d6

use Queue to synchronize threads in multithreaded coverage spec

5eacfbe

add a simple test for lines coverage

5a1fb34

store covered lines in hash

bf00d6f

use ruby 2.7 syntax in ITR::Runner

2d4f694

anmarchenko force-pushed the anmarchenko/ddcov_per_test_coverage branch from 483aff8 to 2d4f694 Compare March 20, 2024 15:51

anmarchenko changed the base branch from main to 1.0 March 20, 2024 15:51

add gem datadog explicitly to integration test's Gemfile to make test…

b010927

…s pass

anmarchenko added this to the 1.0 milestone Mar 21, 2024

anmarchenko added 4 commits March 21, 2024 09:35

filter out invalid line numbers when tracking lines coverage

74ef237

use RSTRING_LEN to avoid computing root's length on every event

f979f58

mark dd_cov_data->root as movable and update pointer on compaction

a291646

move coverage collector to Datadog::CI::ITR::Coverage namespace as it…

e4e86f3

… is ITR-only functionality

ivoanjo reviewed Mar 21, 2024

View reviewed changes

anmarchenko merged commit 61be62e into 1.0 Mar 21, 2024

anmarchenko deleted the anmarchenko/ddcov_per_test_coverage branch March 21, 2024 10:28

Uh oh!

Conversation

anmarchenko commented Mar 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Mar 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

juan-fernandez left a comment

Choose a reason for hiding this comment

Uh oh!

tonyredondo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anmarchenko commented Mar 20, 2024

Uh oh!

anmarchenko commented Mar 21, 2024

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anmarchenko commented Mar 14, 2024 •

edited

Loading

codecov-commenter commented Mar 14, 2024 •

edited

Loading