You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Invoke-WebRequest should save the file name with the name in Content-Disposition #11671
How to get a file name from the server when downloading a file?
The server has a Content-Disposition property, how to get it, and is used when downloading files
Steps to reproduce
$URL=@'https://skydrive.live.com/download?resid=A78ACCAEBB24EDD7!37948&authkey=!AKjXpDoxgONjhUw'@
Iwr $URL-outfile ./
Or
$a=Iwr $URL-outfile ./sss
Expected behavior
Autoaddfilename
Or
PS >$a.headers
Key Value
--- -----
x-amz-id-2 {XAI956ZLVN5q12SAfXINuiiU6rUrLD0w88mtCcMt1uLcwwv…
x-amz-request-id {3EE4C53FE4416D0E}
Date {Fri, 24 Jan 2020 08:36:04 GMT}
ETag {"c4ac8566c3831011ab981c9662b33183"}
Accept-Ranges {bytes}
Server {AmazonS3}
Last-Modified {Wed, 18 Dec 2019 08:27:16 GMT}
Content-Disposition {attachment; filename=OpenSSH-Win32.zip}
Content-Type {application/octet-stream}
Content-Length {3018600}
Actual behavior
Error or null
Environment data
Reacted by Michael Klement, Vitor Rosa, Dave Tapley, John Steel, Mark A. Ziesemer, WindR, elector-bg, cin, Mr Beedell, Roke Julian Lockhart, Matthew A Cargile and 10 more
If I understand correctly you want to set only target directory and use file name from Content-Disposition.
What behavior (in a script, in interactive session) should be if this header is not present?
I agree that the following would make perfect sense and constitutes a simple enhancement to Invoke-WebRequest :
Allow passing a mere directory path to Invoke-WebRequest's -OutFile parameter; e.g., . to ask the downloaded file to be placed in the current directory.
If the -OutFile argument is indeed a directory path, look for the server-side file name in the Content-Disposition response-header field, and use that as the local file name for the downloaded file; if no such header field is present or if it contains an invalid file name, report a statement-terminating error.
This has been asked for twice before - #6618 and #6537 - but shot down on what appears to me to be flawed reasoning: this is a simple enhancement to an extant feature that makes its use more convenient, not an attempt at "feature creep".
Steve Lee (@SteveL-MSFT), given that and that this is now being asked for for the third time - can we revisit this?
Reacted by Mr Beedell, Roke Julian Lockhart, loxia01, Sergio Alonso Pascual, Keith Hill and mpql
P.S.: Given that the file name would then be controlled by the server of origin, there's an even more pressing need to implement #3174 and to use -LiteralPath rather than -OutFile.
Need to get 'Content-Disposition' approach, there is currently no method.To get it you must store "iwr" as a variable. It will take up a lot of memory.
Through links, sometimes it's hard to get a file name,
he852100 , yes, the current workaround is both inconvenient and inefficient (not sure if how the header field is parsed below is robust enough):
$url='https://github.com/PowerShell/Win32-OpenSSH/releases/download/v8.1.0.0p1-Beta/OpenSSH-Win32.zip'$response= iwr $url# stores the full file content in memory$filename=$response.Headers.'Content-Disposition'-replace'.*\bfilename=(.+)(?: |$)','$1'$outDir=Convert-Path$pwd
[IO.File]::WriteAllBytes("$outDir/$filename",$response.Content)
Yes, Ilya (@iSazonov), but that is a separate matter - a dedicated downloading cmdlet with all the bells and whistles is also nice to have.
As stated, this is simply about making an existing feature more convenient - arguably, it should have worked that way from the beginning.
Reacted by ZenoArrow, Synertry, Matthew A Cargile and mpql
Michael Klement (@mklement0) The RFC comes from a fact that web cmdlets are already very complex - they have over 30 parameters! And now we discuss that -OutFile will have additional semantics. This not only opens the way to new errors (how to distinguish path and file? What is user skip final slash? It is very bad UX for script language.) and typos, but it also confuses users by complicity.
So it's a good thing that this enhancement doesn't require a new parameter.
how to distinguish path and file? What is user skip final slash?
The semantics would be the exact same as with Copy-Item with a file source path, for instance: if the target path refers to an existing directory, copy to that directory - whether or not there is a trailing \ or /; if the target path doesn't exist, treat it as file.
No additional complexity, just long-established and well-known patterns.
I should again notice that web cmdlets is very complex. We should avoid to add more complicity. For file processing scenarios we have Mark Kraus (@markekraus) 's RFC.
As noted, enhancing Invoke-WebRequest as proposed and the RFC aren't mutually exclusive.
The proposed feature adds little complexity, from what I can tell, and only makes the download feature that is already there and won't go away complete.
he852100: The existing -PassThru switch provides similar functionality in conjunction with -OutFile, but it passes the entire response through - including the whole file content in property .Content.
I don't see a need for a new parameter that passes just the header through - the proposed enhancement (arguably: fix) requires no new parameters and is the most straightforward solution to the problem.
In my opinion the usage of this new feature (let's call it autonaming) intrinsecally exposes the user to a some risk wether we use the last segment or the content-disposition. The last segment of the original uri is safer but often less informative and so less useful.
I wouldn't use an opt in, too many parameters and discoverability problem.
I don't think the users need to remember these rules, they are delegating the choice of the name to PowerShell. If they wanted to control the output they would have added a name string to outfile
Erroring out would be a breaking change, I would prefer to avoid it (we could look for a better naming scheme)
I do see your point, but with the last-input-URI-segment logic that risk is known, unlike with the other two options.
too many parameters and discoverability problem.
I do see the parameter-proliferation problem; but if opt-in is desirable (TBD), there's no other way.
Discoverability could be helped by pointing to -AutoFileName from the description of -OutFile in the documentation.
And, as stated, you should be able to use only-AutoFileName, as shorthand for -AutoFileName -Out-File .
An opt-in would also allow us to report the then-unknown resulting filename by default, instead of the user needing to remember to ask for it via -PassThru.
Erroring out would be a breaking change
Since nothing has been released yet that would allow you to pass a mere directory path to -OutFile, I don't think it would be a breaking change - or am I missing something?
That said, with -AutoFileName it would make sense to fall back to the sanitized hostname, and considering the last segment of the ultimately-redirected-to-URI makes sense then as well.
An error would then only be reported if you don't use -AutoFileName and no suitable last-input-URI-segment is present.
I don't see a problem if we try Content-Disposition first and then make a fallback to last segment. If an user is not satisfied with this he can always use the explicit name as he does now.
CarloToso Note that both the Content-Disposition filename value and the URL decoded last segment may contain invalid chars (especially / and \, which could lead to nasty path traversal issues, and also the usual set of forbidden characters (Path.GetInvalidFileNameChars())) and should also be checked for DOS special files (COM0-9, AUX,...).
The linked Chromium function has quite a lot of various checks, I'm trying to port the relevant parts to C# now (for an unrelated project), I'll post the final code if I get something usable.
This is what I have now (Windows-only, although only minor modifications should be needed to make it work on Linux), seems to work reasonably intuitively from my testing:
Source code
usingSystem;usingSystem.IO;usingSystem.Linq;usingSystem.Net.Http.Headers;usingSystem.Text.RegularExpressions;usingSystem.Web;namespacePog.Utils.Http;/// <summary>/// Utility class for getting the server-provided file name for a downloaded file./// </summary>publicstaticclassHttpFileNameParser{/// <param name="resolvedUri">Resolved URI (after redirects) of the downloaded file.</param>/// <param name="contentDisposition">The Content-Disposition header from the server response, if any.</param>publicstaticstringGetDownloadedFileName(UriresolvedUri,ContentDispositionHeaderValue?contentDisposition){// link to a similar function in Chromium:// https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L219// Chromium also takes into account the mime type of the response, we're not doing that for now// try available names in the order of preference, use the first one found that is validstring?fileName=null;// if Content-Disposition is set and valid, use the specified filenamefileName??=SanitizeDownloadedFileName(GetRawDispositionFileName(contentDisposition));// use the last segment of the resolved URLvarlastSegment=resolvedUri.Segments.LastOrDefault();fileName??=SanitizeDownloadedFileName(HttpUtility.UrlDecode(lastSegment));// use the hostnamefileName??=SanitizeDownloadedFileName(resolvedUri.Host);// fallback default namefileName??="download";returnfileName;}privatestaticstring?GetRawDispositionFileName(ContentDispositionHeaderValue?contentDisposition){if(contentDispositionis not {DispositionType:"attachment"}){returnnull;}varheaderVal=contentDispositionswitch{{FileNameStar: not null}=>contentDisposition.FileNameStar,{FileName: not null}=>contentDisposition.FileName,
_ =>null,};if(headerVal==null){returnnull;}if(headerVal.StartsWith("\"")&&headerVal.EndsWith("\"")){// ContentDispositionHeaderValue parser leaves the quotes in the parsed filename, strip themheaderVal=headerVal.Substring(1,headerVal.Length-2);}returnheaderVal;}privatestaticreadonlyRegexInvalidDosNameRegex=newRegex(@"^(CON|PRN|AUX|NUL|COM\d|LPT\d)(\..+)?$",RegexOptions.Compiled|RegexOptions.IgnoreCase|RegexOptions.CultureInvariant);privatestaticstring?SanitizeDownloadedFileName(string?fileName){// sources for relevant functions in Chromium:// GetFileNameFromURL: https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L119// GenerateSafeFileName: https://github.com/chromium/chromium/blob/bf9e98c98e8d7e79befeb057fde42b0e320d9b19/net/base/filename_util.cc#L163// SanitizeGeneratedFileName: https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L79// list of invalid filenames on Windows: https://stackoverflow.com/a/62888if(fileName==null){returnnull;}// Win32 does not like trailing '.' and ' ', remove itfileName=fileName.TrimEnd('.',' ');// replace any invalid characters with _fileName=string.Join("_",fileName.Split(Path.GetInvalidFileNameChars()));if(InvalidDosNameRegex.IsMatch(fileName)){// is a DOS file name, prefix with _ to lose the special meaningfileName="_"+fileName;}// if fileName is empty or only consists of invalid chars (or _), skip itif(fileName.All(c =>c=='_')){returnnull;}returnfileName;}}
Unit tests
usingSystem.Net.Http.Headers;usingPog.Utils.Http;usingXunit;namespacePog.Tests;// list of test cases for Content-Disposition parsing: http://test.greenbytes.de/tech/tc2231/publicclassHttpFileNameParserTests{[Fact]publicvoidTestDosNames(){TestSingle("_aux","https://example.com/aux");TestSingle("_AUX.txt","https://example.com/AUX.txt");TestSingle("_COM9","https://example.com/COM9");}[Fact]publicvoidTestPriority(){TestSingle("header","https://host/segment","attachment; filename=header");TestSingle("segment","https://host/segment");TestSingle("segment","https://host/segment","attachment; filename=_");TestSingle("host","https://host/");TestSingle("host","https://host/_","attachment; filename=_");}[Fact]publicvoidTestSanitization(){// filename*TestSingle("hello_world.txt","https://host/segment","attachment; filename*=utf-8''hello%2Fworld.txt");// filenameTestSingle("hello_world.txt","https://host/segment","attachment; filename=\"hello/world.txt\"");// segmentTestSingle(".._hello_world.txt","https://host/..%2Fhello%2Fworld.txt");TestSingle("invalid_chars_.txt","https://host/invalid%2fchars%0a.txt");}privatestaticvoidTestSingle(stringexpected,stringurl,string?dispositionHeader=null){vardisposition=dispositionHeader==null?null:ContentDispositionHeaderValue.Parse(dispositionHeader);Assert.Equal(expected,HttpFileNameParser.GetDownloadedFileName(newUri(url),disposition));}}
Reacted by Mr Beedell, Roke Julian Lockhart and CarloToso
Activity
he852100 commentedon Jan 24, 2020
You must now specify it in a variable. When downloading large files will become difficult
iSazonov commentedon Jan 24, 2020
If I understand correctly you want to set only target directory and use file name from
Content-Disposition.What behavior (in a script, in interactive session) should be if this header is not present?
mklement0 commentedon Jan 24, 2020
I agree that the following would make perfect sense and constitutes a simple enhancement to
Invoke-WebRequest:Allow passing a mere directory path to
Invoke-WebRequest's-OutFileparameter; e.g.,.to ask the downloaded file to be placed in the current directory.If the
-OutFileargument is indeed a directory path, look for the server-side file name in theContent-Dispositionresponse-header field, and use that as the local file name for the downloaded file; if no such header field is present or if it contains an invalid file name, report a statement-terminating error.This has been asked for twice before - #6618 and #6537 - but shot down on what appears to me to be flawed reasoning: this is a simple enhancement to an extant feature that makes its use more convenient, not an attempt at "feature creep".
Steve Lee (@SteveL-MSFT), given that and that this is now being asked for for the third time - can we revisit this?
mklement0 commentedon Jan 24, 2020
P.S.: Given that the file name would then be controlled by the server of origin, there's an even more pressing need to implement #3174 and to use
-LiteralPathrather than-OutFile.he852100 commentedon Jan 24, 2020
Need to get 'Content-Disposition' approach, there is currently no method.To get it you must store "iwr" as a variable. It will take up a lot of memory.
Through links, sometimes it's hard to get a file name,
iSazonov commentedon Jan 24, 2020
There is a reference to RFC Download cmdlet PowerShell/PowerShell-RFC#124
mklement0 commentedon Jan 24, 2020
he852100 , yes, the current workaround is both inconvenient and inefficient (not sure if how the header field is parsed below is robust enough):
Yes, Ilya (@iSazonov), but that is a separate matter - a dedicated downloading cmdlet with all the bells and whistles is also nice to have.
As stated, this is simply about making an existing feature more convenient - arguably, it should have worked that way from the beginning.
iSazonov commentedon Jan 24, 2020
Michael Klement (@mklement0) The RFC comes from a fact that web cmdlets are already very complex - they have over 30 parameters! And now we discuss that
-OutFilewill have additional semantics. This not only opens the way to new errors (how to distinguish path and file? What is user skip final slash? It is very bad UX for script language.) and typos, but it also confuses users by complicity.mklement0 commentedon Jan 24, 2020
So it's a good thing that this enhancement doesn't require a new parameter.
The semantics would be the exact same as with
Copy-Itemwith a file source path, for instance: if the target path refers to an existing directory, copy to that directory - whether or not there is a trailing\or/; if the target path doesn't exist, treat it as file.No additional complexity, just long-established and well-known patterns.
he852100 commentedon Jan 24, 2020
Ilya (@iSazonov) Can I transfer
headersto a variable when I have theOutFileparameter?You just need to provide this feature,The user determines the processing method by the variable value.
iSazonov commentedon Jan 24, 2020
I should again notice that web cmdlets is very complex. We should avoid to add more complicity. For file processing scenarios we have Mark Kraus (@markekraus) 's RFC.
mklement0 commentedon Jan 24, 2020
As noted, enhancing
Invoke-WebRequestas proposed and the RFC aren't mutually exclusive.The proposed feature adds little complexity, from what I can tell, and only makes the download feature that is already there and won't go away complete.
mklement0 commentedon Jan 24, 2020
he852100: The existing
-PassThruswitch provides similar functionality in conjunction with-OutFile, but it passes the entire response through - including the whole file content in property.Content.I don't see a need for a new parameter that passes just the header through - the proposed enhancement (arguably: fix) requires no new parameters and is the most straightforward solution to the problem.
14 remaining items
CarloToso commentedon Mar 17, 2023
Michael Klement (@mklement0) Thank you for your thorough review.
I will wait for Steve Lee (@SteveL-MSFT) to decide the best course of action before continuing.
mklement0 commentedon Mar 17, 2023
Thanks, CarloToso.
I do see your point, but with the last-input-URI-segment logic that risk is known, unlike with the other two options.
I do see the parameter-proliferation problem; but if opt-in is desirable (TBD), there's no other way.
Discoverability could be helped by pointing to
-AutoFileNamefrom the description of-OutFilein the documentation.And, as stated, you should be able to use only
-AutoFileName, as shorthand for-AutoFileName -Out-File .An opt-in would also allow us to report the then-unknown resulting filename by default, instead of the user needing to remember to ask for it via
-PassThru.Since nothing has been released yet that would allow you to pass a mere directory path to
-OutFile, I don't think it would be a breaking change - or am I missing something?That said, with
-AutoFileNameit would make sense to fall back to the sanitized hostname, and considering the last segment of the ultimately-redirected-to-URI makes sense then as well.An error would then only be reported if you don't use
-AutoFileNameand no suitable last-input-URI-segment is present.CarloToso commentedon Mar 20, 2023
Steve Lee (@SteveL-MSFT) friendly ping
iSazonov commentedon Mar 21, 2023
I don't see a problem if we try Content-Disposition first and then make a fallback to last segment. If an user is not satisfied with this he can always use the explicit name as he does now.
majkinetor commentedon Apr 20, 2023
Oh, finally, thank you for this CarloToso 🥇
MatejKafka commentedon Aug 20, 2023
CarloToso The provided code looks more-or-less the same as what Chromium is doing, sans some sanitization: https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L219 In general, it would make sense to me to stay close to the behavior of major browsers as much as possible, since that's what most users likely expect as the default.
MatejKafka commentedon Aug 22, 2023
CarloToso Note that both the Content-Disposition filename value and the URL decoded last segment may contain invalid chars (especially / and \, which could lead to nasty path traversal issues, and also the usual set of forbidden characters (
Path.GetInvalidFileNameChars())) and should also be checked for DOS special files (COM0-9, AUX,...).The linked Chromium function has quite a lot of various checks, I'm trying to port the relevant parts to C# now (for an unrelated project), I'll post the final code if I get something usable.
MatejKafka commentedon Aug 23, 2023
This is what I have now (Windows-only, although only minor modifications should be needed to make it work on Linux), seems to work reasonably intuitively from my testing:
Source code
Unit tests
ziesemer commentedon Feb 19, 2024
This remains a valid issue that should be addressed.