WebApplicationBuilder currently copies sources from its Configuration property in a _hostBuilder.ConfigureHostConfiguration( callback. This works well for sources that are configured by that point, but that means lower-level services like loggers that use the _hostBuilder's configuration don't observe sources that are added are removed after the ConfigureHostConfiguration runs.
As part of this, I think we should try to re-add app configuration during WebApplicationBuilder.Build() so we can allow users to change the environment after constructing the WebApplicationBuilder but before actually building it. See WebApplicationFunctionalTests.EnvironmentSpecificLoggingConfigurationSectionPassedToLoggerByDefault() for one of the broken scenarios.