Add dd-autoinstrumentation CLI with LLM-native structured output#8313
Merged
Conversation
bouwkast
added
the
AI Generated
bouwkast
force-pushed
the
steven/auto-instrumentation-gen-cli-v2
branch
from
March 16, 2026 19:33
78661d7 to
fc5fd18
Compare
BenchmarksBenchmark execution time: 2026-07-09 13:14:50 Comparing candidate commit 8adb8f4 in PR branch Found 2 performance improvements and 1 performance regressions! Performance is the same for 69 metrics, 0 unstable metrics, 66 known flaky benchmarks, 60 flaky benchmarks without significant changes.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8313) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (70ms) : 67, 74
master - mean (70ms) : 68, 72
section Bailout
This PR (8313) - mean (74ms) : 72, 76
master - mean (74ms) : 73, 76
section CallTarget+Inlining+NGEN
This PR (8313) - mean (1,085ms) : 1036, 1135
master - mean (1,084ms) : 1029, 1139
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (110ms) : 105, 115
master - mean (109ms) : 106, 112
section Bailout
This PR (8313) - mean (110ms) : 108, 112
master - mean (110ms) : 107, 113
section CallTarget+Inlining+NGEN
This PR (8313) - mean (780ms) : 757, 803
master - mean (780ms) : 752, 809
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (96ms) : 94, 99
master - mean (101ms) : 94, 107
section Bailout
This PR (8313) - mean (98ms) : 96, 100
master - mean (98ms) : 94, 102
section CallTarget+Inlining+NGEN
This PR (8313) - mean (940ms) : 900, 979
master - mean (934ms) : 894, 973
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (108ms) : 91, 125
master - mean (94ms) : 92, 97
section Bailout
This PR (8313) - mean (109ms) : crit, 94, 124
master - mean (95ms) : 93, 97
section CallTarget+Inlining+NGEN
This PR (8313) - mean (822ms) : 729, 916
master - mean (817ms) : 783, 851
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (202ms) : 197, 207
master - mean (202ms) : 196, 208
section Bailout
This PR (8313) - mean (204ms) : 200, 208
master - mean (205ms) : 202, 208
section CallTarget+Inlining+NGEN
This PR (8313) - mean (1,208ms) : 1168, 1248
master - mean (1,206ms) : 1162, 1251
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (289ms) : 285, 294
master - mean (291ms) : 284, 299
section Bailout
This PR (8313) - mean (290ms) : 285, 295
master - mean (291ms) : 283, 298
section CallTarget+Inlining+NGEN
This PR (8313) - mean (970ms) : 944, 995
master - mean (974ms) : 949, 999
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (284ms) : 277, 291
master - mean (283ms) : 275, 290
section Bailout
This PR (8313) - mean (283ms) : 277, 288
master - mean (283ms) : 278, 287
section CallTarget+Inlining+NGEN
This PR (8313) - mean (1,173ms) : 1137, 1208
master - mean (1,169ms) : 1133, 1206
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8313) - mean (280ms) : 274, 287
master - mean (281ms) : 275, 287
section Bailout
This PR (8313) - mean (281ms) : 273, 289
master - mean (278ms) : 271, 285
section CallTarget+Inlining+NGEN
This PR (8313) - mean (1,047ms) : 997, 1098
master - mean (1,043ms) : 991, 1095
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
bouwkast
force-pushed
the
steven/auto-instrumentation-gen-cli-v2
branch
from
April 1, 2026 12:21
fc5fd18 to
dd7cbfc
Compare
vandonr
reviewed
Apr 15, 2026
|
|
||
| > This document is a complete reference for AI agents using the `dd-autoinstrumentation` CLI to generate CallTarget auto-instrumentation code. It covers all commands, options, JSON schemas, error codes, and recommended workflows. | ||
|
|
||
| ## Quick Start |
Contributor
There was a problem hiding this comment.
do AI agents really need a "quick start" ? Or is it meant to be both for humans and AIs ?
| |---|---| | ||
| | `--no-method-begin` | Skip `OnMethodBegin` handler | | ||
| | `--no-method-end` | Skip `OnMethodEnd` handler | | ||
| | `--async-method-end` | Generate `OnAsyncMethodEnd` handler | |
Contributor
There was a problem hiding this comment.
shouldn't this be automatic for async methods ?
| { | ||
| Console.WriteLine("Available configuration keys for --set (camelCase, same as --json output):"); | ||
| Console.WriteLine(); | ||
| Console.WriteLine(" {0,-45} {1,-10} {2}", "Key", "Type", "Default"); |
Contributor
There was a problem hiding this comment.
nit: I assume this -45 and -10 is for whitespace padding. If you want to make it nice always, rather than hardcoding a value, you can iterate once to get the max length before starting to print.
OR you can kee it as is, but truncate long values so that they don't stick out of the array.
Comment on lines
+36
to
+49
| metadata = result.Metadata is { } meta | ||
| ? new | ||
| { | ||
| meta.AssemblyName, | ||
| meta.TypeName, | ||
| meta.MethodName, | ||
| meta.ReturnTypeName, | ||
| meta.ParameterTypeNames, | ||
| meta.MinimumVersion, | ||
| meta.MaximumVersion, | ||
| meta.IntegrationName, | ||
| meta.IntegrationClassName, | ||
| meta.IsInterface, | ||
| } |
Contributor
There was a problem hiding this comment.
why do we need this ? Can't we just pass the result.Metadata ?
bouwkast
force-pushed
the
steven/auto-instrumentation-gen-cli-v2
branch
from
April 28, 2026 18:16
dd7cbfc to
a4c28bd
Compare
Contributor
|
I have a feature request, which is to dump all the methods in a dll. This is what the ai toolkit we're developing works with in other languages, it doesn't do back and forth with the tool to get methods. |
bouwkast
marked this pull request as ready for review
June 1, 2026 12:32
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d93c81dda
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+347
to
+349
| var outputText = jsonMode | ||
| ? JsonOutputFormatter.Format(result, config) | ||
| : result.SourceCode!; |
There was a problem hiding this comment.
When callers combine --json (which the new agent workflow recommends) with --output path/to/Integration.cs, outputText becomes the JSON envelope and the later File.WriteAllText(output.FullName, outputText) writes that JSON into the .cs file instead of the generated source. This makes the documented output-file workflow produce an unusable integration file whenever structured stdout is requested; keep the file contents as result.SourceCode and emit JSON only to stdout (or reject the combination).
Useful? React with 👍 / 👎.
Add dd-autoinstrumentation CLI that uses the Core library to generate integration boilerplate from the command line. Includes --set key=value configuration, --config-file support, --json output, and Nuke build integration via RunInstrumentationGeneratorCli target. Co-Authored-By: Claude Opus 4.6 <[email protected]>
… assembly inspection Add global --json flag, unified JSON error envelope with machine-readable error codes, inspect subcommand for type/method discovery, and structured --list-keys output. Co-Authored-By: Claude Opus 4.6 <[email protected]>
… workflow Update InstrumentationGenerator.md with inspect command, structured error handling, and LLM usage sections. Add dedicated LLM reference doc with full JSON schemas, error recovery patterns, and step-by-step autonomous workflow. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Wrap AssemblyBrowser construction in try-catch so corrupted or non-.NET DLLs return a structured error instead of crashing the CLI. Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Remove --async-method-end flag (auto-detect handles async methods, including disabling OnMethodEnd which the flag alone did not do) - Compute --list-keys column widths from data instead of hardcoding - Pass result.Metadata directly to the JSON serializer - Rename "Quick Start" to "Invocation" in the LLM-only doc
System.CommandLine binds Option<string[]> to a single token by default, so the documented `--parameter-types System.String System.Int32` form left the second type as an unmatched argument. Set AllowMultipleArgumentsPerToken=true so all space-separated types after the option are bound, fixing both the documented invocation and the RunInstrumentationGeneratorCli Nuke target which appends each type as a separate token.
When --json was set and generator.Generate() returned Success=false
(e.g., static method on a value type, generic interface), the failure
was serialized via JsonOutputFormatter as a stripped-down
{success, errorMessage} object and the command still exited 0. This
made automation paths — including the recommended LLM workflow and the
RunInstrumentationGeneratorCli Nuke target — treat failed generation
as success.
Route the failure through OutputHelper.WriteError so it emits the same
unified envelope as other errors (command + errorCode + errorMessage)
and exits 1. Drop the now-unreachable failure branch from
JsonOutputFormatter.Format.
ListMethods filtered out every IsSpecialName member, which silently dropped constructors (.ctor), property accessors (get_*/set_*), event accessors, and operators. CallTarget integrations in this repo target all of those (search for MethodName = ".ctor" / "get_" / "set_" under ClrProfiler/AutoInstrumentation), so any inspect-driven workflow could not discover them and had no way to recover without already knowing the name. Drop the special-name filter and keep only the compiler-generated check; update the LLM doc to match.
In `generate --json --output <file>`, stdout received "Output written to: <path>" as plain text, breaking the documented "all output on stdout is JSON" contract for callers combining file output with machine-readable parsing. Mirror the JSON envelope to stdout in JSON mode so automation can always parse stdout, while keeping the human-readable confirmation in plain mode.
The Generator.Cli project only targets net10.0, so the documented `dotnet run --framework net8.0` invocation fails before reaching the generator. Omit the flag so dotnet-run picks the project's only TFM.
When the requested --type was absent, ResolveMethod returned null and ListOverloads returned empty, so the missing-type case fell through to METHOD_NOT_FOUND. JSON clients are documented to recover from a missing type by switching to `inspect --list-types`, so they need TYPE_NOT_FOUND to make that decision. Add AssemblyBrowser.TypeExists and check it before the overload lookup. Behavior is now: type missing -> TYPE_NOT_FOUND, type present but method missing -> METHOD_NOT_FOUND, otherwise AMBIGUOUS_OVERLOAD.
Direct rootCommand.Parse(args).Invoke() let System.CommandLine print its default help/error text on parse failures (unknown option, missing required argument, unparseable option value), so JSON-mode callers got unparseable stdout/stderr despite the --json contract. Inspect parseResult.Errors before invoking. When --json is set and errors are present, emit OutputHelper.WriteError with INVALID_ARGUMENT and the resolved subcommand name so automation can recover. Plain mode still falls through to Invoke() and prints the default text.
The StartsWith check on "System.Threading.Tasks.Task" classified TaskScheduler, TaskCompletionSource, TaskFactory, TaskCanceledException, TaskCreationOptions, TaskStatus, etc. as async return types because they share the prefix. CallTarget's async handling only applies to Task / Task<T> / ValueTask / ValueTask<T>, so callers acting on the isAsync flag could pick the wrong handler shape for those returns. Add GenerationConfiguration.IsAsyncReturnType that matches only the exact non-generic names and the dnlib generic forms (Task\`1<...> and ValueTask\`1<...>), and use it from both the inspect output and the auto-detect path in CreateForMethod so the two stay in sync. Update the LLM doc note to reflect the precise rule.
Splitting GeneratorArgs on every space corrupted quoted CLI values:
inline JSON for --config (e.g. '{"k":true}') and paths under
"Program Files" got shredded into separate tokens before the generator
ran, so the wrapper rejected or misapplied otherwise valid invocations.
Add a small shell-style tokenizer that respects single and double
quotes and use it in place of the naive split.
The Generator.Cli .csproj uses <TargetFrameworks> (plural) so dotnet run rejects the project with "targets multiple frameworks" even though only net10.0 is listed. Inheriting from tracer/src/Directory .Build.props makes switching to <TargetFramework> singular pull in net6.0 instead, so the simpler fix is to keep the flag in the documented invocation. The for-ai LLM reference already includes it.
Wrapping arguments only when they contain spaces lost embedded quotes and trailing backslashes, so inline JSON for --config arrived at the generator with the quotes stripped or paths ending in "\\" misparsed. Replace the naive join with a PasteArguments-style escape (matches .NET's ProcessStartInfo.ArgumentList behavior on Windows): wrap when the arg contains whitespace, quotes, or backslashes, double the backslashes that precede each embedded quote, and double the trailing backslashes before the closing quote.
bouwkast
force-pushed
the
steven/auto-instrumentation-gen-cli-v2
branch
from
June 11, 2026 10:44
9d93c81 to
0def04c
Compare
This comment has been minimized.
This comment has been minimized.
andrewlock
reviewed
Jun 18, 2026
Contributor
|
@codex review |
vandonr
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Adds a new CLI tool (
dd-autoinstrumentation) for auto-instrumentation code generation, designedfor scriptable and AI-agent-driven workflows with structured JSON output.
Reason for change
Creating integrations currently requires either the GUI or manual boilerplate. A CLI with structured
JSON output enables LLM agents to autonomously discover types/methods in an assembly, generate
instrumentation code, and recover from errors — all without human intervention.
Implementation details
New
Datadog.AutoInstrumentation.Generator.Cliproject with two commands:inspect— assembly discovery--list-types: lists all non-compiler-generated types with visibility, kind, method count--list-methods <type>: lists instrumentable methods with parameters, async detection, overloadindex/count
generate— code generation--overload-index,--parameter-types)--no-auto-detect)--set key=value,--config '{json}', or--config-file path.json--no-method-begin,--no-method-end,--async-method-end--jsonflag (global) — all output (success and error) as structured JSON on stdout with:FILE_NOT_FOUND,TYPE_NOT_FOUND,AMBIGUOUS_OVERLOAD,BAD_ASSEMBLY, etc.)AMBIGUOUS_OVERLOADincludes full overload list in response data for immediate retryinspectoutput includesoverloadIndex/overloadCountso agents know upfront if disambiguationis needed
Nuke integration —
RunInstrumentationGeneratorClitarget with direct parameters and--generator-argspassthrough.Documentation:
docs/development/InstrumentationGenerator.md— comprehensive reference for both GUI and CLIdocs/development/for-ai/InstrumentationGenerator-CLI.md— LLM-focused reference with full JSONschemas, error recovery patterns, and recommended autonomous workflow
Test coverage
No automated tests in this PR — this is a developer/AI tooling utility, not a production code path.
Manual testing performed against real assemblies for all commands, error paths, and JSON output.
Other details
#8312
🤖 Generated with Claude Code