## Summary of changes

Adds JSON helper APIs to ensure we use the array pool where possible

## Reason for change

There are various "helper" APIs, which are wrappers around
Newtonsoft.JSON's `JsonSerializer` and `JsonReader`/`JsonWriter` APIs.
In #8228 we updated the explicit usages to use an array pool
implementation for `JsonReader`/`JsonWriter` calls, but they're used
internally without a pool in the helper cases.

This PR creates alternative implementations which _do_ use the pool,
updates existing code to use them, and adds the existing APIs to the
"banned API" list.

> There's another possible approach, in which we update the vendored
code to _always_ use the array pool. I was torn as to which is the
better option, and went for this approach in the end, but I'm not wedded
to it, so happy to take the alternative approach if people think it's
preferable?

## Implementation details

- Use 🤖 to find all the potential places that we need to convert.
- Create "array pool" versions of the APIs
- Update the call sites to use the new APIs
- Add the original APIs to the list of "banned" APIs to avoid using them
accidentally in the future

## Test coverage

Covered by all our existing tests, added some unit tests to confirm the
new tests behave as expected

## Other details

https://datadoghq.atlassian.net/browse/LANGPLAT-940

Discovered this while exploring remote config optimizations, but should
help lots of areas.

## Summary of changes

Stop sending the fixed `RootVersion = 1` with every remote config
request

## Reason for change

Currently we're sending a fixed value of `RootVersion = 1` for all our
remote config requests, but doing so causes the agent to repeatedly send
us all the root certificates, significantly increasing the payload size,
because it thinks we haven't seen them. Sending the "final" version,
acknowledges it, and stops all the extra data, saving ~35kB per call.

## Implementation details

- Deserialize the roots in `GetRcmResponse`, leaving them as base64
encoded `string`s (which is how they are sent in the payload)
- When processing the response, de-encode the _last_ root object, and
deserialize it
- Using the `Base64DecodingStream` introduced in #8226 to avoid extra
allocations
  - Using the `IArrayPool` from #8228
- Introduces a `MinimalTufRoot` (in contrast to `TufRoot`) so that we
only materialize what we need (the `roots.signed.version` key)

This implementation avoids ~35kB per call for subsequent remote config
requests.

## Test coverage

Added unit test, and did manual test with the real agent, to confirm the
expected behaviour (reduction in data sent)

## Other details

https://datadoghq.atlassian.net/browse/LANGPLAT-940

All part of the Remote Config perf stack

…#8228)

## Summary of changes

Adds a simple `IArrayPool<char>` for use by Newtonsoft.JSON, and uses it
everywhere we can

## Reason for change

Newtonsoft.JSON fundamentally works with .NET's `char` type (UTF-16),
(as opposed to System.Text.Json which works with UTF-8 where it can). To
do so, it needs to create a bunch of `char[]` instances to use as
buffers.

The `JsonTextReader` and `JsonTextWriter` abstractions allow plugging in
an `IArrayPool<char>` implementation, and luckily this matches (pretty
much exactly) the API exposed by the `ArrayPool` in corelib (+
vendored), so it's easy to implement.

This should help alleviate some GC pressure, as we currently do a fair
amount of serializing and deserializing.

## Implementation details

Pretty simple:
- Implement `IArrayPool<char>` using `ArrayPool<char>.Shared`
- Get 🤖 to find everywhere that we could use it (`JsonTextReader` and
`JsonTextWriter`) and initialize
- Fix some cases where these weren't being disposed.

> [!WARNING]
> It's important that we _do_ dispose these, so that the arrays are
correctly returned to the pool, so that we don't leak memory

There are actually other places we can update too, as this PR doesn't
cover the common `JsonConvert.Serialize()` etc, but I'll follow up with
those in a separate PR.

## Test coverage

All the existing tests should pass. I worked on this as part of general
perf work on remote config, so the results are a bit fuzzy (as I can't
remember if it includes the savings from #8226 as well), but the results
are pretty conclusive, especially for big payloads 😅


| Method | size | Mean | Error | Allocated |
| ---------------------------- | ----- | ----------: | ------------: |
---------: |
| DeserializeResponse_Original | Small | 22.71 ns | 2,076.771 ns | 23.83
KB |
| DeserializeResponse_Updated | Small | 13.70 us | 0.186 us | 17.23 KB |
| | | | | |
| DeserializeResponse_Original | Big | 1,953.04 us | 58,665.219 ns
|2,343.26 KB |
| DeserializeResponse_Updated | Big | 614.46 us | 11.988 us | 252.37 KB
|


## Other details


https://datadoghq.atlassian.net/browse/LANGPLAT-940

Discovered this while exploring remote config optimizations

## Summary of changes

Adds JSON helper APIs to ensure we use the array pool where possible

## Reason for change

There are various "helper" APIs, which are wrappers around
Newtonsoft.JSON's `JsonSerializer` and `JsonReader`/`JsonWriter` APIs.
In #8228 we updated the explicit usages to use an array pool
implementation for `JsonReader`/`JsonWriter` calls, but they're used
internally without a pool in the helper cases.

This PR creates alternative implementations which _do_ use the pool,
updates existing code to use them, and adds the existing APIs to the
"banned API" list.

> There's another possible approach, in which we update the vendored
code to _always_ use the array pool. I was torn as to which is the
better option, and went for this approach in the end, but I'm not wedded
to it, so happy to take the alternative approach if people think it's
preferable?

## Implementation details

- Use 🤖 to find all the potential places that we need to convert.
- Create "array pool" versions of the APIs
- Update the call sites to use the new APIs
- Add the original APIs to the list of "banned" APIs to avoid using them
accidentally in the future

## Test coverage

Covered by all our existing tests, added some unit tests to confirm the
new tests behave as expected

## Other details

https://datadoghq.atlassian.net/browse/LANGPLAT-940

Discovered this while exploring remote config optimizations, but should
help lots of areas.

## Summary of changes

A variety of minor performance improvements to remote config

## Reason for change

I did some initial benchmarking of remote config, as well as running a
test app with ASM/Debugger enabled (which use RCM), and the results
weren't great. Given we make RCM requests every 5s, smallish changes
here should add up, though the real-world effect will be tricky to
gauge.

## Implementation details

Most of the individual changes are small. In summary:
- Making work lazy where possible (delaying creating collections)
  - Don't re-create dictionaries inside loops if not required
- Avoid creating empty dictionaries and collections for no-op RCM
responses
- Avoid creating empty collections in JSON objects when they're not in
the JSON
- Cache things that change rarely
- The `ExtraServicesProvider` will rarely see new service names, so
cache the array (as the collection is append-only)
- Don't create a new request each time, just update any values that may
have changed
  - Cache the capabilities array which will rarely change
- There's actually a potential threading bug around the use of
`BigInteger`. We probably could/should use `ulong` instead but that I'll
look at that in a separate PR
- Use abstractions introduced earlier in the stack (e.g. #8226, #8228)

Additionally, I did a little bit of cleanup:
- Remove unused members from the `IRcmSubscriptionManager` interface
- Add `#nullable enable` to the RCM types (and fix nullability where
required)

## Test coverage

Mostly covered by existing unit tests, also did some manual testing.
Finally, ran a few benchmarks, but it's a bit tricky to check reliably.
This benchmark is benchmarking `_manager.SendRequest(_rcmTracer, _ =>
_steadyStateResponseTask)` and passing the same response every time. In
practice, the response changes every time, so this isn't strictly
representative, but with the changes to request caching, this should
actually mean our improvements are _better_ than the original would be.
The regression in the .NET duration is curious, but I'm not massively
concerned, and the allocations are obviously down a lot at least


| Method | Runtime | Mean | Error | Allocated |
| ------------------------ | ------------------ | ----------: |
-----------: | --------: |
| PollSteadyState_Original | .NET 10.0 | 14,235.8 ns | 275.09 ns | 12.04
KB |
| PollSteadyState_Updated | .NET 10.0 | 20,472.0 ns | 1,352.567 ns |
3.90 KB |
| | | | | |
| PollSteadyState_Original | .NET 6.0 | 33,915.7 ns | 3,339.42 ns |
12.27 KB |
| PollSteadyState_Updated | .NET 6.0 | 14,979.7 ns | 447.784 ns | 3.99
KB |
| | | | | |
| PollSteadyState_Original | .NET Core 2.1 | 21,488.2 ns | 365.39 ns |
12.85 KB |
| PollSteadyState_Updated | .NET Core 2.1 | 17,847.2 ns | 693.510 ns |
4.04 KB |
| | | | | |
| PollSteadyState_Original | .NET Core 3.1 | 18,925.2 ns | 304.16 ns |
12.23 KB |
| PollSteadyState_Updated | .NET Core 3.1 | 15,407.7 ns | 313.26 ns |
4.01 KB |
| | | | | |
| PollSteadyState_Original | .NET Framework 4.8 | 20,946.8 ns | 320.06
ns | 14.21 KB |
| PollSteadyState_Updated | .NET Framework 4.8 | 16,635.2 ns | 266.526
ns | 4.40 KB |


## Other details
https://datadoghq.atlassian.net/browse/LANGPLAT-940

All part of the Remote Config perf stack. I think this is probably about
the end of it for now