Repository navigation
Web Server returning utf8 (not utf-8) for ContentType header throws an error in ReadBufferAsString method #133319
Description
Description
Whilst wrapping some code around a developer API in PowerShell, I was getting an error thrown about the incorrect use of utf8 as returned by in the ContentType Header by that server.
I have a workaround in a private build of PowerShell & a draft PR that I've just closed as wasn't really the best approach.
I feel it makes more sense to fix this in the System.Net.Http dotnet package so that other users including PowerShell could get the benefit of the fix & not come across this issue.
I think I have a interim fix, (not yet built & tested locally that's on the list) which is relatively small that I think makes sense for as part of inclusion in the System.Net.Http dotnet package. This would negate my own draft PR in the PowerShell repo that is currently failing as well as my need for keeping a private build.
Reproduction Steps
set up a webserver that throws a contentType header of application/json; charset=utf8
in PowerShell 7.6.5 that sets the ContentType Header to utf8
function Start-TestWebServer {
Add-Type -AssemblyName System.Net
$listener = [System.Net.HttpListener]::new()
$listener.Prefixes.Add("http://localhost:8088/")
$listener.Start()
Write-Host "Listening on http://localhost:8088/ (Ctrl+C to stop)"
$stop = $false
# Ctrl+C handler
$null = Register-EngineEvent -SourceIdentifier ConsoleCancelEvent -Action {
Write-Host "Ctrl+C detected, stopping..."
$stop = $true
}
while (-not $stop -and $listener.IsListening) {
try {
$context = $listener.GetContext() # blocking, but safe inside Task
$response = $context.Response
# Intentionally broken charset
$response.ContentType = "application/json; charset=utf8"
$body = "Hello from a broken charset server"
$bytes = [System.Text.Encoding]::UTF8.GetBytes($body)
$response.ContentLength64 = $bytes.Length
$response.OutputStream.Write($bytes, 0, $bytes.Length)
$response.OutputStream.Close()
}
catch {
break
}
}
# Main loop waits for Ctrl+C
try {
while (-not $stop) {
Start-Sleep -Milliseconds 200
}
}
finally {
Write-Host "Ctrl+ C pressed - Stopping listener..."
$listener.Stop()
$listener.Close()
$listener.Dispose()
Write-Host "Listener stopped & disposed."
}
}
Start-TestWebServerThen run from another PowerShell 7.6.5 window
Invoke-WebRequest http://localhost:8088 | Select -expand Headers This returns showing the broken header
Invoke-RestMethod http://localhost:8088 -Verbose -DebugThis returns exception showing that the utf8 not being supported
System.ArgumentException: 'utf8' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method. (Parameter 'name')
at System.Text.EncodingTable.InternalGetCodePageFromName(String name)
at System.Text.EncodingTable.GetCodePageFromName(String name)
at System.Text.Encoding.GetEncoding(String name)
at System.Net.Http.HttpContent.ReadBufferAsString(LimitArrayPoolWriteStream stream, HttpContentHeaders headers)
Expected behavior
Ideally a graceful conversion that is either contained within this method or pulled in from another package that does a map nicely from utf8 to utf-8 or other encodings
Actual behavior
Errors as shown in the repro steps
Regression?
Unknown
Known Workarounds
In the private build of PowerShell I have & subsequent failing draft PR I got around this with a catch all approach defaulting the Encoding to UTF8 which is a hacky hack & isn't really a sensible fix.
Configuration
Name Value
PSVersion 7.6.5
PSEdition Core
GitCommitId 7.6.5
OS Microsoft Windows 10.0.26200
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.4
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Other information
No response
Activity
dotnet-policy-service commentedon Sep 6, 2026
Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.
MihaZupan commentedon Sep 6, 2026
Are you able to ask the server authors to fix their logic?
In C# this is rather easy to workaround as you can overwrite the header yourself.
I'm not convinced the default HttpClient logic should be trying to catch such invalid inputs. If this were very common, it seems like something
Encoding.GetEncodingwould want to consider rather than consuming components.For PowerShell, the recommendation on the PR to reuse PowerShell's existing mapping support (PowerShell/PowerShell#27789 (comment)) seems like a reasonable way to solve this.
dotnet-policy-service commentedon Sep 7, 2026
This issue has been marked
needs-author-actionand may be missing some important information.kilasuit commentedon Sep 9, 2026
Yes I intend to do so, & whilst it doesn't feel like it's a common thing, I get the feeling that we shouldn't really need to adapt downstream (where possible, like in via the recommendation to use what we have in PowerShell which may well end up being what I look at doing there if we don't fix it here) when people, or in future 'AI' tooling, may decide to strip out
-etc. I know we always strive to get people to "accept & adopt the standards" as they currently are.Yeah, same in PowerShell or F# though shouldn't need to workaround downstream when it's a case of being like 90% of what's expected.
That may well be a better place to surface a change like this than the attempt I've made via #133322 but this has at least given me the opportunity to understand a bit more about the current codebase, try and build it locally and then propose a "fix" even if that fix isn't what ends up being what gets taken.
MihaZupan commentedon Sep 11, 2026
Given I don't remember seeing similar reports about utf-8 vs utf8, I doubt it's a very widespread thing.
Normalizing this particular example seems rather harmless, but as I said earlier I'm not convinced the HTTP stack is the right layer for something like this.
@tarekgh does normalization like this seem like something you'd consider for
Encoding.GetEncoding?If not, I'm inclined to close this as won't-fix. Since PowerShell has prior art of doing normalization like this elsewhere, I think it makes sense for their helpers to do something similar in this case.
dotnet-policy-service commentedon Sep 11, 2026
Tagging subscribers to this area: @dotnet/area-system-text-encoding
See info in area-owners.md if you want to be subscribed.
tarekgh commentedon Sep 11, 2026
Although the
utf8is not standard name as IANA define it asutf-8, but it makes sense to supportutf8as alias for that encoding. Other platforms already support that as alias toutf-8and .NET should support it too to avoid issues as the one reported here. I'll try to open a PR soon to do that.Add utf8, utf16, utf32 as accepted Encoding.GetEncoding aliases (#133710