Skip to content

Replace fpm with nfpm#5905

Merged
andrewlock merged 11 commits into
masterfrom
andrew/ci/remove-fpm
Aug 28, 2024
Merged

Replace fpm with nfpm#5905
andrewlock merged 11 commits into
masterfrom
andrew/ci/remove-fpm

Conversation

@andrewlock

@andrewlock andrewlock commented Aug 15, 2024

Copy link
Copy Markdown
Member

Summary of changes

Replaces the ruby tool fpm with nfpm and tar

Reason for change

fpm is a ruby tool, which means we have to install ruby. This has historically caused a bunch of issues at various points when dependencies change (e.g. we add/update anything or change dockerfiles).

nfpm is a go tool created specifically to do pretty much the same thing as fpm, but without requiring ruby. The only slightly annoying thing is that it doesn't support raw "tar" so we have to use the built-in tar for that (although that's not a big hardship).

Implementation details

  • Install nfpm in the docker images and remove the fpm install (and removing all the other ruby-related code)
  • Call tar to pack the tar images, and nfpm to pack the deb/rpm packages
    • nfpm is a "config" based tool, so we dump the yaml config to a file, and pass that to the execution
    • I have validated the nfpm and fpm deb output are essentially identical, there are just a few nuances
      • fpm adds a License field to the .deb header, but that's non standard. The correct approach is to include a specific license file. See this issue for more details.
      • Similarly, fpm adds a Vendor field that is non standard (but Maintainer is standard and is added).
      • fpm adds Relocations: /var/datadog to the .rpm header, so I have replicated that for nfpm, but I don't know that we actually want that... If we let people relocate the headers, then the documentation is wrong for the profiler env vars etc. Left it as-is anyway, to reduce chance of breaking people, just something I spotted.
      • The "Apache License 2.0" license string used for .rpm previously is not a valid value, as defined by fedora, so switched it to the supported moniker here.
    • The tar files were "accidentally" including the before/after scripts, so I've made sure to still include them, incase anyone was relying on those (they run the createLogPath.sh file, and create a symlink to dd-dotnet in /usr/bin)

Test coverage

Ran a full test with all installers here - ignore the couple of failures in the dotnet tool smoke tests, those are failures on master I need to fix 😅

Will manually compare the output of this run too:

  • tar (tar -tzvf *.tar.gz)
    • The same, except the before/after scripts weren't being made executable. Fixed
  • deb (dpkg --info *.deb and dpkg -c *.deb)
    • Minor differences:
      • nfpm adds the directories as entries
      • nfpm doesn't have the non-standard License field (unlike fpm)
      • nfpm adds the license in the required copyright path
      • nfpm doesn't include a dummy changelog.gz file (unlike fpm)
  • rpm (rpm -qipl --scripts *.rpm)
    • Minor differences, nfpm adds the directories as entries, nothing significant though

Other details

Stacked on

because we need to change the dockerfiles, and we need the Centos7 fixes in there at the very least. We'll need to update the VMSS images though after making these changes so I've been delaying until that makes sense up to this point.

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Aug 15, 2024
@andrewlock
andrewlock requested a review from a team as a code owner August 15, 2024 13:59
@datadog-ddstaging

datadog-ddstaging Bot commented Aug 15, 2024

Copy link
Copy Markdown

Datadog Report

Branch report: andrew/ci/remove-fpm
Commit report: 0dcd714
Test service: dd-trace-dotnet

✅ 0 Failed, 297628 Passed, 1604 Skipped, 12h 11m 2s Total Time

@andrewlock

andrewlock commented Aug 15, 2024

Copy link
Copy Markdown
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

@andrewlock

andrewlock commented Aug 15, 2024

Copy link
Copy Markdown
Member Author

Benchmarks Report for tracer 🐌

Benchmarks for #5905 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.141
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.83μs 41.8ns 229ns 0.012 0.00399 0 5.43 KB
master StartStopWithChild netcoreapp3.1 10.1μs 52.3ns 277ns 0.0247 0.00987 0 5.62 KB
master StartStopWithChild net472 15.9μs 45.9ns 165ns 1.02 0.301 0.0949 6.07 KB
#5905 StartStopWithChild net6.0 7.52μs 42.2ns 277ns 0.015 0.00751 0 5.43 KB
#5905 StartStopWithChild netcoreapp3.1 9.73μs 52.5ns 302ns 0.0288 0.0144 0.0048 5.61 KB
#5905 StartStopWithChild net472 16.3μs 66.3ns 257ns 0.997 0.284 0.0729 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 467μs 326ns 1.26μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 639μs 361ns 1.4μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 837μs 469ns 1.82μs 0.417 0 0 3.3 KB
#5905 WriteAndFlushEnrichedTraces net6.0 469μs 365ns 1.42μs 0 0 0 2.7 KB
#5905 WriteAndFlushEnrichedTraces netcoreapp3.1 630μs 375ns 1.45μs 0 0 0 2.7 KB
#5905 WriteAndFlushEnrichedTraces net472 852μs 537ns 2.01μs 0.425 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 187μs 919ns 4.01μs 0.184 0 0 18.45 KB
master SendRequest netcoreapp3.1 210μs 1.15μs 8.57μs 0.217 0 0 20.61 KB
master SendRequest net472 0.000247ns 0.000155ns 0.00056ns 0 0 0 0 b
#5905 SendRequest net6.0 191μs 1.05μs 6.61μs 0.187 0 0 18.45 KB
#5905 SendRequest netcoreapp3.1 206μs 1.08μs 5.2μs 0.208 0 0 20.61 KB
#5905 SendRequest net472 7.7E‑05ns 7.7E‑05ns 0.000278ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5905

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.61 KB 41.92 KB 306 B 0.74%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 564μs 2.25μs 8.73μs 0.581 0 0 41.47 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 670μs 3.11μs 11.6μs 0.342 0 0 41.61 KB
master WriteAndFlushEnrichedTraces net472 848μs 3.4μs 13.2μs 8.42 2.66 0.443 53.3 KB
#5905 WriteAndFlushEnrichedTraces net6.0 558μs 2.55μs 9.87μs 0.566 0 0 41.48 KB
#5905 WriteAndFlushEnrichedTraces netcoreapp3.1 684μs 3.38μs 14μs 0.338 0 0 41.92 KB
#5905 WriteAndFlushEnrichedTraces net472 859μs 4.06μs 15.7μs 8.36 2.64 0.44 53.33 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.2μs 1.04ns 3.89ns 0.0145 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.68μs 1.1ns 4.12ns 0.0133 0 0 1.02 KB
master ExecuteNonQuery net472 2.02μs 2.08ns 8.06ns 0.156 0 0 987 B
#5905 ExecuteNonQuery net6.0 1.32μs 1.5ns 5.81ns 0.0145 0 0 1.02 KB
#5905 ExecuteNonQuery netcoreapp3.1 1.75μs 1.23ns 4.78ns 0.0131 0 0 1.02 KB
#5905 ExecuteNonQuery net472 1.91μs 1.65ns 6.39ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.21μs 0.493ns 1.91ns 0.0133 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.55μs 0.529ns 1.98ns 0.0132 0 0 976 B
master CallElasticsearch net472 2.45μs 1.51ns 5.84ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.27μs 1.24ns 4.62ns 0.0133 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.6μs 0.662ns 2.48ns 0.0137 0 0 1.02 KB
master CallElasticsearchAsync net472 2.54μs 2.69ns 10.4ns 0.167 0 0 1.05 KB
#5905 CallElasticsearch net6.0 1.14μs 0.632ns 2.45ns 0.0136 0 0 976 B
#5905 CallElasticsearch netcoreapp3.1 1.49μs 0.39ns 1.46ns 0.0127 0 0 976 B
#5905 CallElasticsearch net472 2.45μs 1.39ns 5.39ns 0.157 0 0 995 B
#5905 CallElasticsearchAsync net6.0 1.32μs 1.24ns 4.8ns 0.0132 0 0 952 B
#5905 CallElasticsearchAsync netcoreapp3.1 1.59μs 0.88ns 3.29ns 0.0135 0 0 1.02 KB
#5905 CallElasticsearchAsync net472 2.61μs 2.08ns 8.07ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.22μs 0.768ns 2.97ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.66μs 0.944ns 3.65ns 0.0124 0 0 952 B
master ExecuteAsync net472 1.77μs 4.48ns 17.3ns 0.145 0 0 915 B
#5905 ExecuteAsync net6.0 1.29μs 0.768ns 2.97ns 0.0132 0 0 952 B
#5905 ExecuteAsync netcoreapp3.1 1.56μs 0.294ns 1.1ns 0.0131 0 0 952 B
#5905 ExecuteAsync net472 1.72μs 0.817ns 3.17ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.33μs 1.55ns 5.57ns 0.0304 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.13μs 1.3ns 5.03ns 0.0358 0 0 2.76 KB
master SendAsync net472 7.75μs 1.47ns 5.69ns 0.5 0 0 3.15 KB
#5905 SendAsync net6.0 4.09μs 2.6ns 9.74ns 0.0308 0 0 2.22 KB
#5905 SendAsync netcoreapp3.1 5.02μs 1.68ns 6.27ns 0.0375 0 0 2.76 KB
#5905 SendAsync net472 7.73μs 1.78ns 6.9ns 0.499 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.57μs 0.627ns 2.34ns 0.0229 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.28μs 0.822ns 2.96ns 0.0219 0 0 1.64 KB
master EnrichedLog net472 2.67μs 1.98ns 7.68ns 0.249 0 0 1.57 KB
#5905 EnrichedLog net6.0 1.5μs 0.439ns 1.58ns 0.0233 0 0 1.64 KB
#5905 EnrichedLog netcoreapp3.1 2.13μs 6.66ns 24.9ns 0.0223 0 0 1.64 KB
#5905 EnrichedLog net472 2.69μs 1.03ns 3.85ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 116μs 218ns 845ns 0 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 118μs 199ns 717ns 0 0 0 4.28 KB
master EnrichedLog net472 147μs 254ns 982ns 0.654 0.218 0 4.46 KB
#5905 EnrichedLog net6.0 115μs 346ns 1.34μs 0.057 0 0 4.28 KB
#5905 EnrichedLog netcoreapp3.1 120μs 327ns 1.27μs 0 0 0 4.28 KB
#5905 EnrichedLog net472 148μs 315ns 1.22μs 0.664 0.221 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.04μs 0.63ns 2.44ns 0.0305 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.21μs 10.7ns 41.4ns 0.0296 0 0 2.2 KB
master EnrichedLog net472 5.03μs 2.59ns 10ns 0.32 0 0 2.02 KB
#5905 EnrichedLog net6.0 2.96μs 1.19ns 4.61ns 0.0311 0 0 2.2 KB
#5905 EnrichedLog netcoreapp3.1 4.27μs 0.994ns 3.85ns 0.0299 0 0 2.2 KB
#5905 EnrichedLog net472 4.97μs 1.81ns 7.01ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.38μs 1.18ns 4.59ns 0.0158 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.76μs 0.717ns 2.78ns 0.0149 0 0 1.14 KB
master SendReceive net472 2.16μs 2.48ns 9.61ns 0.183 0.00109 0 1.16 KB
#5905 SendReceive net6.0 1.35μs 0.605ns 2.34ns 0.0162 0 0 1.14 KB
#5905 SendReceive netcoreapp3.1 1.88μs 1.25ns 4.86ns 0.015 0 0 1.14 KB
#5905 SendReceive net472 2.25μs 1.61ns 6.02ns 0.183 0.00113 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.76μs 1.17ns 4.36ns 0.0221 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.89μs 1.53ns 5.94ns 0.0213 0 0 1.65 KB
master EnrichedLog net472 4.43μs 2.83ns 11ns 0.323 0 0 2.04 KB
#5905 EnrichedLog net6.0 2.71μs 0.587ns 2.2ns 0.0217 0 0 1.6 KB
#5905 EnrichedLog netcoreapp3.1 4.05μs 3.34ns 12.9ns 0.0218 0 0 1.65 KB
#5905 EnrichedLog net472 4.3μs 3.11ns 11.6ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5905

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.142 652.79 571.51

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 398ns 0.184ns 0.711ns 0.008 0 0 576 B
master StartFinishSpan netcoreapp3.1 592ns 0.386ns 1.5ns 0.00778 0 0 576 B
master StartFinishSpan net472 653ns 0.175ns 0.676ns 0.0917 0 0 578 B
master StartFinishScope net6.0 547ns 0.231ns 0.832ns 0.00956 0 0 696 B
master StartFinishScope netcoreapp3.1 737ns 0.518ns 2.01ns 0.00918 0 0 696 B
master StartFinishScope net472 879ns 0.903ns 3.5ns 0.105 0 0 658 B
#5905 StartFinishSpan net6.0 406ns 0.768ns 2.87ns 0.00799 0 0 576 B
#5905 StartFinishSpan netcoreapp3.1 614ns 0.419ns 1.51ns 0.00774 0 0 576 B
#5905 StartFinishSpan net472 572ns 0.938ns 3.63ns 0.0917 0 0 578 B
#5905 StartFinishScope net6.0 552ns 0.493ns 1.91ns 0.0099 0 0 696 B
#5905 StartFinishScope netcoreapp3.1 773ns 0.952ns 3.69ns 0.00955 0 0 696 B
#5905 StartFinishScope net472 918ns 4.05ns 15.7ns 0.105 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5905

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 1.139 1,011.99 888.54

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 647ns 0.413ns 1.6ns 0.0098 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 1.01μs 0.299ns 1.12ns 0.00962 0 0 696 B
master RunOnMethodBegin net472 1.07μs 0.593ns 2.3ns 0.104 0 0 658 B
#5905 RunOnMethodBegin net6.0 605ns 0.564ns 2.19ns 0.00983 0 0 696 B
#5905 RunOnMethodBegin netcoreapp3.1 888ns 0.325ns 1.17ns 0.00932 0 0 696 B
#5905 RunOnMethodBegin net472 1.03μs 0.987ns 3.56ns 0.104 0 0 658 B

@lucaspimentel lucaspimentel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks

@lucaspimentel
lucaspimentel requested a review from a team August 20, 2024 17:23
@andrewlock
andrewlock force-pushed the andrew/ci/try-fix-builds branch from fe1376b to 9001630 Compare August 23, 2024 09:46
@andrewlock
andrewlock force-pushed the andrew/ci/remove-fpm branch from 85e665e to 3d85dec Compare August 23, 2024 09:46
Comment thread Datadog.Trace.Minimal.slnf
@andrewlock

Copy link
Copy Markdown
Member Author

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5905) (11.456M)   : 0, 11456165
    master (11.567M)   : 0, 11566509
    benchmarks/2.9.0 (11.883M)   : 0, 11882814

    section Automatic
    This PR (5905) (7.740M)   : 0, 7739659
    master (7.841M)   : 0, 7841471
    benchmarks/2.9.0 (8.446M)   : 0, 8445613

    section Trace stats
    master (8.125M)   : 0, 8124886

    section Manual
    master (11.708M)   : 0, 11708359

    section Manual + Automatic
    This PR (5905) (7.198M)   : 0, 7198061
    master (7.216M)   : 0, 7216432

    section DD_TRACE_ENABLED=0
    master (10.773M)   : 0, 10773417

Loading
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5905) (9.662M)   : 0, 9661526
    master (9.828M)   : 0, 9828340
    benchmarks/2.9.0 (9.760M)   : 0, 9759697

    section Automatic
    This PR (5905) (6.573M)   : 0, 6572556
    master (6.642M)   : 0, 6641935

    section Trace stats
    master (6.912M)   : 0, 6911773

    section Manual
    master (9.501M)   : 0, 9500687

    section Manual + Automatic
    This PR (5905) (6.032M)   : 0, 6032343
    master (6.162M)   : 0, 6161652

    section DD_TRACE_ENABLED=0
    master (8.890M)   : 0, 8889645

Loading
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5905) (10.345M)   : 0, 10345255

    section Automatic
    This PR (5905) (6.895M)   : 0, 6895351

    section Manual + Automatic
    This PR (5905) (6.310M)   : 0, 6310476

Loading

Base automatically changed from andrew/ci/try-fix-builds to master August 28, 2024 13:42
@andrewlock
andrewlock force-pushed the andrew/ci/remove-fpm branch from f91433b to 0dcd714 Compare August 28, 2024 13:44
@andrewlock
andrewlock merged commit 86c5843 into master Aug 28, 2024
@andrewlock
andrewlock deleted the andrew/ci/remove-fpm branch August 28, 2024 15:52
@github-actions github-actions Bot added this to the vNext-v3 milestone Aug 28, 2024
andrewlock added a commit that referenced this pull request Aug 28, 2024
## Summary of changes

Adds support for the tracer on alpine on ARM64

## Reason for change

We currently support x64 on glibc/alpine but only glibc on arm64. This
closes that matrix which makes it easier to understand

## Implementation details

There's several layers, but at a high level
- Anywhere where we detect platform, make sure to include
`linux-musl-arm64` as well as `linux-arm64`
- Build the tracer + profiler on alpine arm64

There were several difficulties
- The ruby-based fpm bundler we use doesn't work on alpine arm64, hence
#5905
- I couldn't get our existing alpine.build.dockerfile to compile on
arm64 (where we currently compile llvm/clang16 on alpine14).
- To work around it, created a separate alpine.build.arm64.dockerfile
image which uses alpine:18 (the first version that includes clang16) as
the base.
- Obviously this gives us different requirements in arm64, but I think
that's fine because a) this is a new feature b) we already have
differences on the glibc requirements for arm64 c) We only support .NET
5+ anyway on arm64 (.NET 6 really given .NET 5 is mostly gone) and they
ship alpine:18 images for .NET 6+
- I created a multi arch base image from the two base images ([as some
bloke with a blog describes
here](https://andrewlock.net/combining-multiple-docker-images-into-a-multi-arch-image/))
- I had difficulty compiling the profiler on arm64 initially (we require
it for some crash tracking functionality), as we were getting `undefined
reference to `_Uaarch64_get_accessors_int'` when trying to compile
libunwind. @gleocadie tracked it down to [this
issue](libunwind/libunwind#788), and fixed it
by just not building the libunwind tests 🙇‍♂️
- The sqlite IAST instrumentation tests on alpine arm64 were failing in
.NET 5. I _think_ that's because the sqlite package we're using there
doesn't support it (which makes sense) so just disabled the tests for
that scenario. They pass in .NET 6+ so I'm not worried about it.
- Removed the "noop" smoke tests stage (there's nothing to run in it
currently)
- We can't even install .NET Core 2.1 using the dotnet install script on
alpine arm64, so bailing out. We can probably bail out of more too, but
that's not _necessary_ so haven't done it yet.

There's still one outstanding issue I can't get my head around, on
_debian x64_ (i.e. not something that should have changed) The
`OnEolFrameworkInSsi_WhenForwarderPathExists_CallsForwarderWithExpectedTelemetry`
tests is failing by crashing. It's bizarre, I can't figure it out, if
anyone has any ideas, please let me know 😅

## Test coverage

I've run full installer + full TFM tests
[here](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=162869&view=results)
🤞

## Other details

Stacked on 
- #5770
- #5905

as they both change key parts of the build + will mean we need to
rebuild the VMs

Fixes #3850

---------

Co-authored-by: Gregory LEOCADIE <[email protected]>
andrewlock added a commit that referenced this pull request Aug 30, 2024
## Summary of changes

Replaces the ruby tool `fpm` with `nfpm` and `tar`

## Reason for change

`fpm` is a ruby tool, which means we have to install ruby. This has
historically caused a bunch of issues at various points when
dependencies change (e.g. we add/update anything or change dockerfiles).

[`nfpm` is a go tool](https://github.com/goreleaser/nfpm) created
specifically to do pretty much the same thing as `fpm`, but without
requiring ruby. The only slightly annoying thing is that it doesn't
support raw "tar" so we have to use the built-in `tar` for that
(although that's not a big hardship).

## Implementation details

- Install `nfpm` in the docker images and remove the `fpm` install (and
removing all the other ruby-related code)
- Call `tar` to pack the `tar` images, and `nfpm` to pack the `deb/rpm`
packages
- `nfpm` is a "config" based tool, so we dump the yaml config to a file,
and pass that to the execution
- I have validated the nfpm and fpm deb output are essentially
identical, there are just a few nuances
- `fpm` adds a `License` field to the `.deb` header, but that's non
standard. [The correct
approach](https://www.debian.org/doc/debian-policy/ch-docs.html#copyright-information)
is to include a specific license file. See [this
issue](goreleaser/nfpm#847) for more details.
- Similarly, `fpm` adds a `Vendor` field that is non standard (but
`Maintainer` is standard and is added).
- `fpm` adds `Relocations: /var/datadog` to the `.rpm` header, so I have
replicated that for nfpm, but I don't know that we actually _want_
that... If we let people relocate the headers, then the documentation is
wrong for the profiler env vars etc. Left it as-is anyway, to reduce
chance of breaking people, just something I spotted.
- The `"Apache License 2.0"` license string used for `.rpm` previously
is not a valid value, as defined by fedora, so switched it to the
supported moniker
[here](https://docs.fedoraproject.org/en-US/legal/allowed-licenses/).
- The `tar` files were "accidentally" including the before/after
scripts, so I've made sure to still include them, incase anyone was
relying on those (they run the `createLogPath.sh` file, and create a
symlink to dd-dotnet in `/usr/bin`)

## Test coverage

[Ran a full test with all installers
here](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=162539&view=results)
- ignore the couple of failures in the dotnet tool smoke tests, those
are failures on `master` I need to fix 😅

Will manually compare the output of this run too:
- [x] tar (`tar -tzvf *.tar.gz`)
- The same, except the before/after scripts weren't being made
executable. Fixed
- [x] deb (`dpkg --info *.deb` and `dpkg -c *.deb`)
  - Minor differences: 
    - nfpm adds the directories as entries
    - nfpm doesn't have the non-standard License field (unlike fpm)
    - nfpm adds the license in the required copyright path
    - nfpm doesn't include a dummy changelog.gz file (unlike fpm)
- [x] rpm (`rpm -qipl --scripts *.rpm`)
- Minor differences, nfpm adds the directories as entries, nothing
significant though

## Other details

Stacked on
- #5770

because we need to change the dockerfiles, and we need the Centos7 fixes
in there at the very least. We'll need to update the VMSS images though
after making these changes so I've been delaying until that makes sense
up to this point.
andrewlock added a commit that referenced this pull request Aug 30, 2024
## Summary of changes

The `release/2.x` branch is broken since I updated the VMs

## Reason for change

I updated VMs yesterday, which cleared the build cache. I knew that
would slow down release/2.x builds but I forgot it would completely
break them due to the centos7 mess.

## Implementation details

Backported some PRs to fix the layer caching etc

- #5770
- #5905
- #5933 (just the changes
in the alpine dockerfile)

## Test coverage

This is the test, hopefully it works 🤞
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants