Problem with .AddStandardResilienceHandler(); when used in integration test with custom web factory and when there is more than one test.

I have a service that uses HttpClients to 3rd party endpoints. On this one, I use the.AddStandardResilienceHandler();
I set this up in my startup.cs then I have an integration test where I use the WebApplicationFactory

All works fine, but after some lib updates, I start getting random exceptions from my tests.
If I run one test at a time with no problem, it seems to be shared resource issues when they run in parallel.

The tests randomly give me:

  Message: 
System.ObjectDisposedException : Cannot access a disposed object.
Object name: 'IServiceProvider'.

And:

System.InvalidCastException: Unable to cast object of type 'System.TimeSpan' to type 'System.String'.
   at __OptionValidationGeneratedAttributes.<Validators_g>F8B1F3D42962A35D8FF1B1489612AEF6C36F3713335EFF79DB68A25973333C495____SourceGen__RangeAttribute.IsValid(Object value)
   at System.ComponentModel.DataAnnotations.ValidationAttribute.IsValid(Object value, ValidationContext validationContext)
   at System.ComponentModel.DataAnnotations.ValidationAttribute.GetValidationResult(Object value, ValidationContext validationContext)
   at System.ComponentModel.DataAnnotations.Validator.TryValidate(Object value, ValidationContext validationContext, ValidationAttribute attribute, ValidationError& validationError)
   at System.ComponentModel.DataAnnotations.Validator.GetValidationErrors(Object value, ValidationContext validationContext, IEnumerable`1 attributes, Boolean breakOnFirstError)
   at System.ComponentModel.DataAnnotations.Validator.TryValidateValue(Object value, ValidationContext validationContext, ICollection`1 validationResults, IEnumerable`1 validationAttributes)
   at Microsoft.Extensions.Http.Resilience.__HttpTimeoutStrategyOptionsValidator__.

It does not happen all the time but mostly every time.

I have tried fixture, using the same client that's created on all test methods but it seems like a disposal is happening.
But next time there is some expectation that the resilience services shall exist somehow. Not all services are not resettled.
Some singleton was added once I guess.

 public HttpClient CreateClient(ITestOutputHelper testOutputHelper = null)
 {
     _waf = new WebApplicationFactory<TProgram>()
        .WithWebHostBuilder(builder =>
        {

If I remove the .AddStandardResilienceHandler(); from my setup of the HTTP client all works fine.

I also tried to remove the added services and override them in

 builder.ConfigureTestServices(services =>
 {

But that will not work because it's still have added the resilience stuff before I reach this area.

So it seams like:
First it creates my client with resilience, then I ran the fist FACT test. Then a second one.
I have 2 tests one for success and one for failure. If I remove one of the test all works fine. But as soon I share the client in many test this happens.

Not sure if there is som clean-up, or dispose issues with the resilience setup? I have followed the basic guidelines to setup integration tests in .net

Have I missed something? Or is this a known bug?
I can see there is a bug reported where it seams to be a shared setting for all services atm so we can't override settings on different once. Maybe that's related here?