HttpLoggingMiddleware is overly aggressive at redacting headers #36156
Description
In doing a quick experiment with YARP to see if the Logging Middleware would help with diagnostics, I came to the conclusion that potentially yes, but its overly aggressive redacting of headers currently hinders its usefulness.
For example:
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/foo - -
info: Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware[1]
Request:
Protocol: HTTP/1.1
Method: GET
Scheme: http
PathBase:
Path: /foo
QueryString:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Connection: keep-alive
Host: localhost:5000
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.84
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Upgrade-Insecure-Requests: [Redacted]
sec-ch-ua: [Redacted]
sec-ch-ua-mobile: [Redacted]
DNT: [Redacted]
Sec-Fetch-Site: [Redacted]
Sec-Fetch-Mode: [Redacted]
Sec-Fetch-User: [Redacted]
Sec-Fetch-Dest: [Redacted]
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'minimumroute'
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
Proxying to http://www.example.com/foo
info: Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware[2]
Response:
StatusCode: 404
Content-Type: text/html; charset=utf-8
Date: [Redacted]
Server: [Redacted]
Accept-Ranges: [Redacted]
Age: [Redacted]
Cache-Control: [Redacted]
Content-Encoding: [Redacted]
Expires: [Redacted]
Last-Modified: [Redacted]
Vary: [Redacted]
Content-Length: 648
X-Cache: [Redacted]
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'minimumroute'
info: Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware[4]
ResponseBody: ▼ ?§?] ♥}TMs? ►??Wl?K2#$'i→?-i??i☼i☼i☼=↕??P☺??t??B?#7????]x???↕???▬???)f?c?\¶3?_?o????m►>↓t???:←?(?∟a#(?Y▲y???:@?????<?%nZc?d?F
?\?a?♣?▲?,9|?♫?r0???↕w▌????S.a?►???'t?pKx;????♀/Q☺?y??9??⌂E‼?&↓?↓+?2??∟?\▲☺?-s5↨fC??▬.?o↑??↔???%????♀_♫?????I?bB?1??s?b??p?L`i,??►Km4NA?)¶?É?[???j??t☻~??QU???D?c??$d??
τ[??f?I$?<1v??!]?O3*n?↔H►? ??Ht|m:☼??¶?4t?R1????S???-Z%?#G???!??U▲
A???x◄►?4???{???A?♂(?|o?♦
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://localhost:5000/foo - - - 404 648 text/html;+charset=utf-8 9.5025ms
There should be nothing security related in these headers, yet they are redacted:
- Date: [Redacted]
- Server: [Redacted]
- Accept-Ranges: [Redacted]
- Age: [Redacted]
- Cache-Control: [Redacted]
- Content-Encoding: [Redacted]
- Expires: [Redacted]
- Last-Modified: [Redacted]
- Vary: [Redacted]
I agree that the headers shown should be on an allow-list basis so that custom headers that may contain keys etc are hidden. But you should go over the IANA list and extend the allow lost with common ones that are likely safe.