Skip to content

Add continous profiler checks to dd-trace #3167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevingosse merged 13 commits into from
Sep 19, 2022
Merged

Add continous profiler checks to dd-trace #3167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9191dc8
Add diagnostic checks for the profiler
kevingosse Sep 1, 2022
7912078
Start adding tests
kevingosse Sep 5, 2022
6a44933
More tests
kevingosse Sep 5, 2022
fd844fa
Fix assertion
kevingosse Sep 5, 2022
531ee10
Fix paths
kevingosse Sep 5, 2022
a6480e9
Fix result of profiler tests
kevingosse Sep 8, 2022
6c305fd
Fix module path
kevingosse Sep 8, 2022
85a358b
Fix path on Alpine
kevingosse Sep 8, 2022
a96ef7c
Fix path²
kevingosse Sep 8, 2022
3b874fa
Update tracer/src/Datadog.Trace.Tools.Runner/Checks/ProcessBasicCheck.cs
Sep 9, 2022
c9de3e7
Validate that LD_PRELOAD points to the right file
kevingosse Sep 9, 2022
cc19b5d
Fix test case
kevingosse Sep 9, 2022
1757dc4
Fix test
kevingosse Sep 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tracer/src/Datadog.Trace.Coverage.collector/CoverageCollector.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ internal void ProcessFolder(string folder, SearchOption searchOption)
Directory.EnumerateFiles(folder, "*.*", searchOption),
file =>
{
var path = Path.GetDirectoryName(file);
var path = Path.GetDirectoryName(file)!;
var fileWithoutExtension = Path.GetFileNameWithoutExtension(file);
// Skip the Datadog.Trace assembly
if (tracerAssemblyName == fileWithoutExtension)
Expand Down
164 changes: 153 additions & 11 deletions tracer/src/Datadog.Trace.Tools.Runner/Checks/ProcessBasicCheck.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Reflection.PortableExecutable;
using System.Runtime.InteropServices;
using Spectre.Console;

Expand Down Expand Up @@ -41,11 +40,38 @@ public static bool Run(ProcessInfo process, IRegistryService? registryService =
runtime = ProcessInfo.Runtime.NetFx;
}

var profilerModule = FindProfilerModule(process);
bool isContinuousProfilerEnabled;

if (profilerModule == null)
if (process.EnvironmentVariables.TryGetValue("DD_PROFILING_ENABLED", out var profilingEnabled))

@zacharycmontoya zacharycmontoya Sep 9, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to run all of the ContinuousProfiler checks after identifying the loader module and the native tracer module because these console messages aren't actionable if the profiler isn't a part of the product they installed

{
Utils.WriteWarning(ProfilerNotLoaded);
if (ParseBooleanConfigurationValue(profilingEnabled))
{
AnsiConsole.WriteLine(ContinuousProfilerEnabled);
isContinuousProfilerEnabled = true;
}
else
{
AnsiConsole.WriteLine(ContinuousProfilerDisabled);
isContinuousProfilerEnabled = false;
}
}
else
{
AnsiConsole.WriteLine(ContinuousProfilerNotSet);
isContinuousProfilerEnabled = false;
}

var loaderModule = FindLoader(process);
var nativeTracerModule = FindNativeTracerModule(process, loaderModule != null);

if (loaderModule == null)

@zacharycmontoya zacharycmontoya Sep 9, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the user is running a pretty old (pre-loader DLL), then the loader isn't relevant and the user could get concerned over this message. What about strengthening the check like this?

Suggested change
if (loaderModule == null)
if (loaderModule == null && (nativeTracerModule?.StartsWith("Datadog.Tracer.Native", StringComparison.OrdinalIgnoreCase) ?? true))

{
AnsiConsole.WriteLine(LoaderNotLoaded);
}

if (nativeTracerModule == null)
{
Utils.WriteWarning(NativeTracerNotLoaded);
ok = false;
}
else
Expand All @@ -54,11 +80,16 @@ public static bool Run(ProcessInfo process, IRegistryService? registryService =
// .so modules don't have version metadata
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
var version = FileVersionInfo.GetVersionInfo(profilerModule);
var version = FileVersionInfo.GetVersionInfo(nativeTracerModule);
AnsiConsole.WriteLine(ProfilerVersion(version.FileVersion ?? "{empty}"));
}
}

if (isContinuousProfilerEnabled)
{
ok &= CheckContinousProfiler(process, loaderModule);
}

var tracerModules = FindTracerModules(process).ToArray();

if (tracerModules.Length == 0)
Expand Down Expand Up @@ -136,7 +167,7 @@ public static bool Run(ProcessInfo process, IRegistryService? registryService =
ok &= CheckProfilerPath(process, runtime == ProcessInfo.Runtime.NetCore ? "CORECLR_PROFILER_PATH_32" : "COR_PROFILER_PATH_32", requiredOnLinux: false);
ok &= CheckProfilerPath(process, runtime == ProcessInfo.Runtime.NetCore ? "CORECLR_PROFILER_PATH_64" : "COR_PROFILER_PATH_64", requiredOnLinux: false);

if (RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows))
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
if (!CheckRegistry(registryService))
{
Expand All @@ -147,6 +178,51 @@ public static bool Run(ProcessInfo process, IRegistryService? registryService =
return ok;
}

internal static bool CheckContinousProfiler(ProcessInfo process, string? loaderModule)
{
bool ok = true;

var continuousProfilerModule = FindContinousProfilerModule(process);

if (continuousProfilerModule == null)
{
Utils.WriteWarning(ContinuousProfilerNotLoaded);
ok = false;
}

if (loaderModule == null)
{
Utils.WriteError(ContinuousProfilerWithoutLoader);
ok = false;
}

if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
{
process.EnvironmentVariables.TryGetValue("LD_PRELOAD", out var ldPreload);

if (ldPreload == null)
{
Utils.WriteError(LdPreloadNotSet);
ok = false;
}
else
{
if (Path.GetFileName(ldPreload) != "Datadog.Linux.ApiWrapper.x64.so")
{
Utils.WriteError(WrongLdPreload(ldPreload));
ok = false;
}
else if (!File.Exists(ldPreload))
{
Utils.WriteError(ApiWrapperNotFound(ldPreload));
ok = false;
}
}
}

return ok;
}

internal static bool CheckRegistry(IRegistryService? registry = null)
{
registry ??= new Windows.RegistryService();
Expand Down Expand Up @@ -219,7 +295,7 @@ private static bool CheckProfilerPath(ProcessInfo process, string key, bool requ
}
else if (requiredOnLinux)
{
if (!RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows))
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
Utils.WriteError(EnvironmentVariableNotSet(key));
ok = false;
Expand Down Expand Up @@ -258,7 +334,7 @@ private static bool IsExpectedProfilerFile(string fullPath)
{
var fileName = Path.GetFileName(fullPath);

if (RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows))
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
return "Datadog.Trace.ClrProfiler.Native.dll".Equals(fileName, StringComparison.OrdinalIgnoreCase) ||
"Datadog.Tracer.Native.dll".Equals(fileName, StringComparison.OrdinalIgnoreCase) ||
Expand All @@ -270,18 +346,67 @@ private static bool IsExpectedProfilerFile(string fullPath)
// Paths are case-sensitive on Linux
return "Datadog.Trace.ClrProfiler.Native.so".Equals(fileName, StringComparison.Ordinal) ||
"Datadog.Tracer.Native.so".Equals(fileName, StringComparison.Ordinal) ||
// Check for legacy names
// Check for legacy names
"Datadog.AutoInstrumentation.NativeLoader.so".Equals(fileName, StringComparison.Ordinal);
}

private static string? FindProfilerModule(ProcessInfo process)
private static string? FindLoader(ProcessInfo process)
{
foreach (var module in process.Modules)
{
var fileName = Path.GetFileName(module);

if (fileName.Equals("Datadog.Trace.ClrProfiler.Native.dll", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals("Datadog.Trace.ClrProfiler.Native.so", StringComparison.OrdinalIgnoreCase))
|| fileName.Equals("Datadog.Trace.ClrProfiler.Native.so", StringComparison.Ordinal))
{
// This could be either the native tracer or the loader.
// If it's the loader then there should be a loader.conf file next to it.
var folder = Path.GetDirectoryName(module)!;

if (File.Exists(Path.Combine(folder, "loader.conf")))
{
return module;
}
}
else if (fileName.Equals("Datadog.AutoInstrumentation.NativeLoader.x64.dll", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals("Datadog.AutoInstrumentation.NativeLoader.x86.dll", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals("Datadog.AutoInstrumentation.NativeLoader.so", StringComparison.Ordinal))
{
return module;
}
}

return null;
}

private static string? FindContinousProfilerModule(ProcessInfo process)
{
foreach (var module in process.Modules)
{
var fileName = Path.GetFileName(module);

if (fileName.Equals("Datadog.Profiler.Native.dll", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals("Datadog.Profiler.Native.so", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals("Datadog.AutoInstrumentation.Profiler.Native.x64.dll", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals("Datadog.AutoInstrumentation.Profiler.Native.x86.dll", StringComparison.OrdinalIgnoreCase))
{
return module;
}
}

return null;
}

private static string? FindNativeTracerModule(ProcessInfo process, bool foundLoader)
{
var expectedFileName = foundLoader ? "Datadog.Tracer.Native" : "Datadog.Trace.ClrProfiler.Native";

foreach (var module in process.Modules)
{
var fileName = Path.GetFileName(module);

if (fileName.Equals($"{expectedFileName}.dll", StringComparison.OrdinalIgnoreCase)
|| fileName.Equals($"{expectedFileName}.so", StringComparison.OrdinalIgnoreCase))
{
return module;
}
Expand All @@ -302,5 +427,22 @@ private static IEnumerable<string> FindTracerModules(ProcessInfo process)
}
}
}

private static bool ParseBooleanConfigurationValue(string value)
{
var trimmedValue = value.Trim();

return trimmedValue is "true"
or "True"
or "TRUE"
or "yes"
or "Yes"
or "YES"
or "t"
or "T"
or "Y"
or "y"
or "1";
}
Comment thread
andrewlock marked this conversation as resolved.
}
}
15 changes: 14 additions & 1 deletion tracer/src/Datadog.Trace.Tools.Runner/Checks/Resources.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ internal static class Resources
public const string NetCoreRuntime = "Target process is running with .NET Core";
public const string RuntimeDetectionFailed = "Failed to detect target process runtime, assuming .NET Framework";
public const string BothRuntimesDetected = "The target process is running .NET Framework and .NET Core simultaneously. Checks will be performed assuming a .NET Framework runtime.";
public const string ProfilerNotLoaded = "The native library is not loaded into the process";
public const string LoaderNotLoaded = "The native loader library is not loaded into the process";
public const string NativeTracerNotLoaded = "The native tracer library is not loaded into the process";
public const string TracerNotLoaded = "Tracer is not loaded into the process";
public const string AgentDetectionFailed = "Could not detect the agent version. It may be running with a version older than 7.27.0.";
public const string IisProcess = "The target process is an IIS process. The detection of the configuration might be incomplete, it's recommended to use dd-trace check iis <site name> instead.";
Expand All @@ -32,6 +33,18 @@ internal static class Resources
public const string AspNetCoreProcessNotFound = "Could not find the ASP.NET Core applicative process.";
public const string VersionConflict = "Tracer version 1.x can't be loaded simultaneously with other versions and will produce orphaned traces. Make sure to synchronize the Datadog.Trace NuGet version with the installed automatic instrumentation package version.";

public const string ContinuousProfilerEnabled = "DD_PROFILING_ENABLED is set.";
public const string ContinuousProfilerDisabled = "The continuous profiler is explicitly disabled through DD_PROFILING_ENABLED.";
public const string ContinuousProfilerNotSet = "DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.";
public const string ContinuousProfilerNotLoaded = "The continuous profiler library is not loaded into the process.";
public const string ContinuousProfilerWithoutLoader = "The continuous profiler needs the Datadog.Trace.ClrProfiler.Native module and the loader.conf file to work. Try reinstalling the tracer in version 2.14+.";

public const string LdPreloadNotSet = "The environment variable LD_PRELOAD is not set. Check the Datadog .NET Profiler documentation to set it properly.";

public static string ApiWrapperNotFound(string path) => $"The environment variable LD_PRELOAD is set to '{path}' but the file could not be found. Check the Datadog .NET Profiler documentation to set it properly.";

public static string WrongLdPreload(string path) => $"The environment variable LD_PRELOAD is set to '{path}' but it should point to Datadog.Linux.ApiWrapper.x64.so instead. Check the Datadog .NET Profiler documentation to set it properly.";

public static string ProfilerVersion(string version) => $"The native library version {version} is loaded into the process.";

public static string TracerVersion(string version) => $"The tracer version {version} is loaded into the process.";
Expand Down
2 changes: 1 addition & 1 deletion tracer/src/Datadog.Trace.Tools.Runner/Utils.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ internal static void WriteSuccess(string message)
AnsiConsole.MarkupLine($"[green]{message.EscapeMarkup()}[/]");
}

private static bool IsAlpine()
internal static bool IsAlpine()
{
try
{
Expand Down
Loading