Build and package datadog-profiling in CI by morrisonlevi · Pull Request #1663 · DataDog/dd-trace-php

morrisonlevi · 2022-07-20T04:39:00Z

Description

This adds jobs to the build_packages workflow in CircleCI that build
the datadog-profiling extension on x86_64-unknown-linux-gnu and
x86_64-alpine-linux-musl. It adjusts the datadog-setup.php install
script to use extension = datadog-profiling.so instead of
zend_extension = datadog-profiling.so and checks for profiling
dependencies that weren't there before, which depending on your OS
includes libgcc_s, libdl, libpthread, and librt.

This also updated the Alpine images to v3.16, which caused a few things
to break in CI. I fixed one of them already, but there's still an issue
with the warning return-local-addr. This was beyond my skill to just
drop in and fix quickly, so I dropped in a
-Wno-error=return-local-addr that should be removed once the fix is
in place. In the testing I did, the tracer built from this workflow can
run on Alpine 3.11 and earlier for x86-64, but I suspect for arm it
will not be possible to due to the libmusl v1.2 upgrade that happens
on Alpine 3.13.

Readiness checklist

Changelog has been added to the release document.
Tests added for this feature/bug.

Reviewer checklist

Appropriate labels assigned.
Milestone is set.

This is because my Internet connection sucks at the moment and I need to minimize total size _and_ individual layer size. As a bonus, the extra time spent here means that more layers get shared than before. Also temporarily allow `-Wno-error=return-local-addr`; these started showing up after the Alpine 3.16 upgrade. It needs a tracer engineer to look at it, so for now I'm just allowing it so I can work in parallel.

This is simpler to implement in Rust because it avoid global mutable state in favor of lazy initialization. It does have the unfortunate side-effect that if the profiler is installed but disabled and the user clicks on the Code Hotspots tab, then the app will try to find associated profiles even though there won't be any.

This shaves off about 40s of link time. I haven't profiled performance yet _at all_ so this is fine.

This saves some time in the CI, as this is on the slowest path to the "package extension" job.

Overall this will cost more CI resources but cuts off around 90s of build time.

r1viollet · 2022-07-28T07:29:23Z

+      - restore_cache:
+          name: Restore Cargo Package Cache
+          keys:
+            - profiling-cargo-cache-<< parameters.triplet >>-{{ checksum "profiling/Cargo.toml" }}


This key looks nice to me

r1viollet · 2022-07-28T08:05:19Z

-curl -L -o $tmp_folder_profiling_archive $profiling_url
+
+# Profiling: C version
+#curl -L -o $tmp_folder_profiling_archive $profiling_url


This does make it easy to revert back. Just flagging it so I remember.

r1viollet

Some nice optimisations to get the Rust build times down.
I think we could have even more caching. To be discussed for next PR, this is already big !
Congrats on putting all of this together. It must have been some amount of iterations.

bwoebi

Thanks, this looks great to me. I'm not sure whether we need that extra package_profiling step, but it does no harm, so, let's move on :-)

morrisonlevi · 2022-07-29T19:55:43Z

I took at look at "package profiling" and I think it does about 1 minute of harm! It spends about 1 minute downloading the container, attaching the workspace, re-uploading the workspace, etc for just 10ish seconds of work. We can just do those things in "package extension". Good eye!

bwoebi

That seems to shave off a couple seconds. Guess there's nothing left to optimize now.

r1viollet

Good thinking on these last optims !
LGTM

Base automatically changed from levi/profiling-ci to master July 20, 2022 19:28

Build profiler in CI

eb82471

morrisonlevi force-pushed the levi/profiling-build branch from 142e56b to eb82471 Compare July 20, 2022 22:25

morrisonlevi added 7 commits July 21, 2022 08:22

Build profiling on Alpine

9163be1

Add CARGO_HOME to Alpine too

92b298d

Fix cache per OS

60bfa6e

Fixup alpine jobs

7b0d963

Drop cargo from workspace, use cache only

04bd756

Try integrating profiler into built packages

0e8e32e

morrisonlevi force-pushed the levi/profiling-build branch 2 times, most recently from 1c95b20 to af061e3 Compare July 22, 2022 04:56

Try fixing -Werror=address-of-packed-member

2903a62

morrisonlevi force-pushed the levi/profiling-build branch from af061e3 to 2903a62 Compare July 22, 2022 05:11

morrisonlevi added 3 commits July 22, 2022 08:44

Handle zend_extension -> extension

0c749a8

Display sha256 hashes on artifacts

e02c8a1

morrisonlevi force-pushed the levi/profiling-build branch from 4454302 to e02c8a1 Compare July 23, 2022 00:38

morrisonlevi added 2 commits July 25, 2022 09:42

Merge branch 'master' into levi/profiling-build

c177c87

Be more explicit about profiler deps

74efc13

morrisonlevi force-pushed the levi/profiling-build branch from d75d9ed to 286923f Compare July 25, 2022 21:34

Add note about cargo lib licensing

dabc095

morrisonlevi force-pushed the levi/profiling-build branch 2 times, most recently from d2de1c8 to 09ff06b Compare July 25, 2022 22:24

Adjust profiling install-uninstall-reinstall test

3de065c

morrisonlevi force-pushed the levi/profiling-build branch from 09ff06b to 3de065c Compare July 25, 2022 22:46

Merge branch 'master' into levi/profiling-build

51d11e5

morrisonlevi force-pushed the levi/profiling-build branch 3 times, most recently from 2965a0e to 320c831 Compare July 26, 2022 20:34

morrisonlevi force-pushed the levi/profiling-build branch from 9224be5 to bb53bcc Compare July 27, 2022 16:53

Try building using ramdisk

3f7f355

morrisonlevi force-pushed the levi/profiling-build branch from bb53bcc to 3f7f355 Compare July 27, 2022 20:07

morrisonlevi added 7 commits July 27, 2022 15:05

Add helpers and :fingerscrossed: fix test_uninstall.sh

06090b4

Hopefully fix the last of the installer tests

ad3c1da

Switch from fat LTO to thin

8a758de

This shaves off about 40s of link time. I haven't profiled performance yet _at all_ so this is fine.

Use smaller image for packaging the profiler

d1262d3

This saves some time in the CI, as this is on the slowest path to the "package extension" job.

Adjust cargo dependency caching

cd68b2d

Split out testing to save time on critical path

02c37bf

Overall this will cost more CI resources but cuts off around 90s of build time.

Re-organize cargo test jobs a bit

acbabf6

morrisonlevi added this to the 0.77.0 milestone Jul 28, 2022

morrisonlevi added dev/testing profiling Relates to the Continuous Profiler labels Jul 28, 2022

morrisonlevi changed the title ~~Build and package profiler~~ Build and package datadog-profiling in CI Jul 28, 2022

morrisonlevi marked this pull request as ready for review July 28, 2022 04:29

morrisonlevi requested review from a team as code owners July 28, 2022 04:29

r1viollet reviewed Jul 28, 2022

View reviewed changes

r1viollet previously approved these changes Jul 28, 2022

View reviewed changes

bwoebi previously approved these changes Jul 29, 2022

View reviewed changes

Merge 'package profiler' into 'package extension'

dbcf89d

morrisonlevi dismissed stale reviews from bwoebi and r1viollet via dbcf89d July 29, 2022 20:03

bwoebi approved these changes Jul 29, 2022

View reviewed changes

r1viollet approved these changes Jul 31, 2022

View reviewed changes

morrisonlevi merged commit d8daab4 into master Aug 1, 2022

morrisonlevi deleted the levi/profiling-build branch August 1, 2022 00:32

morrisonlevi mentioned this pull request Aug 2, 2022

Release 0.77.0-beta.1 #1684

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build and package datadog-profiling in CI#1663

Build and package datadog-profiling in CI#1663
morrisonlevi merged 28 commits into
masterfrom
levi/profiling-build

morrisonlevi commented Jul 20, 2022 •

edited

Loading

Uh oh!

r1viollet Jul 28, 2022

Uh oh!

r1viollet Jul 28, 2022

Uh oh!

r1viollet left a comment

Uh oh!

bwoebi left a comment

Uh oh!

morrisonlevi commented Jul 29, 2022 •

edited

Loading

Uh oh!

bwoebi left a comment

Uh oh!

r1viollet left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

morrisonlevi commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Readiness checklist

Reviewer checklist

Uh oh!

r1viollet Jul 28, 2022

Choose a reason for hiding this comment

Uh oh!

r1viollet Jul 28, 2022

Choose a reason for hiding this comment

Uh oh!

r1viollet left a comment

Choose a reason for hiding this comment

Uh oh!

bwoebi left a comment

Choose a reason for hiding this comment

Uh oh!

morrisonlevi commented Jul 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwoebi left a comment

Choose a reason for hiding this comment

Uh oh!

r1viollet left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

morrisonlevi commented Jul 20, 2022 •

edited

Loading

morrisonlevi commented Jul 29, 2022 •

edited

Loading