Skip to content

Invoke-WebRequest should save the file name with the name in Content-Disposition #11671

Open

Description

@he852100

How to get a file name from the server when downloading a file?

The server has a Content-Disposition property, how to get it, and is used when downloading files

Steps to reproduce

$URL=@'
https://skydrive.live.com/download?resid=A78ACCAEBB24EDD7!37948&authkey=!AKjXpDoxgONjhUw
'@

Iwr $URL -outfile ./
Or
$a=Iwr $URL -outfile ./sss

Expected behavior

Autoaddfilename
Or
PS >$a.headers
  Key                 Value
---                 -----
x-amz-id-2          {XAI956ZLVN5q12SAfXINuiiU6rUrLD0w88mtCcMt1uLcwwv…
x-amz-request-id    {3EE4C53FE4416D0E}
Date                {Fri, 24 Jan 2020 08:36:04 GMT}
ETag                {"c4ac8566c3831011ab981c9662b33183"}
Accept-Ranges       {bytes}
Server              {AmazonS3}
Last-Modified       {Wed, 18 Dec 2019 08:27:16 GMT}
Content-Disposition {attachment; filename=OpenSSH-Win32.zip}
Content-Type        {application/octet-stream}
Content-Length      {3018600}

Actual behavior

Error or null

Environment data


Activity

  1. added
    Issue-Questionideally support can be provided via other mechanisms, but sometimes folks do open an issue to get a
    on Jan 24, 2020
  2. he852100 commented on Jan 24, 2020

    @he852100
    Author

    You must now specify it in a variable. When downloading large files will become difficult

  3. iSazonov commented on Jan 24, 2020

    @iSazonov
    Collaborator

    If I understand correctly you want to set only target directory and use file name from Content-Disposition.
    What behavior (in a script, in interactive session) should be if this header is not present?

  4. mklement0 commented on Jan 24, 2020

    @mklement0
    Contributor

    I agree that the following would make perfect sense and constitutes a simple enhancement to Invoke-WebRequest :

    • Allow passing a mere directory path to Invoke-WebRequest's -OutFile parameter; e.g., . to ask the downloaded file to be placed in the current directory.

    • If the -OutFile argument is indeed a directory path, look for the server-side file name in the Content-Disposition response-header field, and use that as the local file name for the downloaded file; if no such header field is present or if it contains an invalid file name, report a statement-terminating error.

    This has been asked for twice before - #6618 and #6537 - but shot down on what appears to me to be flawed reasoning: this is a simple enhancement to an extant feature that makes its use more convenient, not an attempt at "feature creep".

    Steve Lee (@SteveL-MSFT), given that and that this is now being asked for for the third time - can we revisit this?

  5. mklement0 commented on Jan 24, 2020

    @mklement0
    Contributor

    P.S.: Given that the file name would then be controlled by the server of origin, there's an even more pressing need to implement #3174 and to use -LiteralPath rather than -OutFile.

  6. he852100 commented on Jan 24, 2020

    @he852100
    Author

    Need to get 'Content-Disposition' approach, there is currently no method.To get it you must store "iwr" as a variable. It will take up a lot of memory.

    Through links, sometimes it's hard to get a file name,

  7. iSazonov commented on Jan 24, 2020

    @iSazonov
    Collaborator

    There is a reference to RFC Download cmdlet PowerShell/PowerShell-RFC#124

  8. mklement0 commented on Jan 24, 2020

    @mklement0
    Contributor

    he852100 , yes, the current workaround is both inconvenient and inefficient (not sure if how the header field is parsed below is robust enough):

    $url = 'https://github.com/PowerShell/Win32-OpenSSH/releases/download/v8.1.0.0p1-Beta/OpenSSH-Win32.zip'
    $response = iwr $url  # stores the full file content in memory
    $filename = $response.Headers.'Content-Disposition' -replace '.*\bfilename=(.+)(?: |$)', '$1'
    $outDir = Convert-Path $pwd
    [IO.File]::WriteAllBytes("$outDir/$filename", $response.Content)

    Yes, Ilya (@iSazonov), but that is a separate matter - a dedicated downloading cmdlet with all the bells and whistles is also nice to have.

    As stated, this is simply about making an existing feature more convenient - arguably, it should have worked that way from the beginning.

  9. iSazonov commented on Jan 24, 2020

    @iSazonov
    Collaborator

    Michael Klement (@mklement0) The RFC comes from a fact that web cmdlets are already very complex - they have over 30 parameters! And now we discuss that -OutFile will have additional semantics. This not only opens the way to new errors (how to distinguish path and file? What is user skip final slash? It is very bad UX for script language.) and typos, but it also confuses users by complicity.

  10. mklement0 commented on Jan 24, 2020

    @mklement0
    Contributor

    they have over 30 parameters!

    So it's a good thing that this enhancement doesn't require a new parameter.

    how to distinguish path and file? What is user skip final slash?

    The semantics would be the exact same as with Copy-Item with a file source path, for instance: if the target path refers to an existing directory, copy to that directory - whether or not there is a trailing \ or /; if the target path doesn't exist, treat it as file.

    No additional complexity, just long-established and well-known patterns.

  11. he852100 commented on Jan 24, 2020

    @he852100
    Author

    Ilya (@iSazonov) Can I transfer headers to a variable when I have the OutFile parameter?

    You just need to provide this feature,The user determines the processing method by the variable value.

    $a=iwr f outfile ./dff.vvg
    $a=$a.headers xxx
    rename-item xxx
    
  12. iSazonov commented on Jan 24, 2020

    @iSazonov
    Collaborator

    I should again notice that web cmdlets is very complex. We should avoid to add more complicity. For file processing scenarios we have Mark Kraus (@markekraus) 's RFC.

  13. mklement0 commented on Jan 24, 2020

    @mklement0
    Contributor

    As noted, enhancing Invoke-WebRequest as proposed and the RFC aren't mutually exclusive.
    The proposed feature adds little complexity, from what I can tell, and only makes the download feature that is already there and won't go away complete.

  14. mklement0 commented on Jan 24, 2020

    @mklement0
    Contributor

    he852100: The existing -PassThru switch provides similar functionality in conjunction with -OutFile, but it passes the entire response through - including the whole file content in property .Content.

    I don't see a need for a new parameter that passes just the header through - the proposed enhancement (arguably: fix) requires no new parameters and is the most straightforward solution to the problem.

  15. 14 remaining items

  16. CarloToso commented on Mar 17, 2023

    @CarloToso
    Contributor

    Michael Klement (@mklement0) Thank you for your thorough review.

    • In my opinion the usage of this new feature (let's call it autonaming) intrinsecally exposes the user to a some risk wether we use the last segment or the content-disposition. The last segment of the original uri is safer but often less informative and so less useful.
    • I wouldn't use an opt in, too many parameters and discoverability problem.
    • I don't think the users need to remember these rules, they are delegating the choice of the name to PowerShell. If they wanted to control the output they would have added a name string to outfile
    • Erroring out would be a breaking change, I would prefer to avoid it (we could look for a better naming scheme)

    I will wait for Steve Lee (@SteveL-MSFT) to decide the best course of action before continuing.

  17. mklement0 commented on Mar 17, 2023

    @mklement0
    Contributor

    Thanks, CarloToso.

    intrinsically exposes the user to a some risk

    I do see your point, but with the last-input-URI-segment logic that risk is known, unlike with the other two options.

    too many parameters and discoverability problem.

    I do see the parameter-proliferation problem; but if opt-in is desirable (TBD), there's no other way.
    Discoverability could be helped by pointing to -AutoFileName from the description of -OutFile in the documentation.
    And, as stated, you should be able to use only -AutoFileName, as shorthand for -AutoFileName -Out-File .

    An opt-in would also allow us to report the then-unknown resulting filename by default, instead of the user needing to remember to ask for it via -PassThru.

    Erroring out would be a breaking change

    Since nothing has been released yet that would allow you to pass a mere directory path to -OutFile, I don't think it would be a breaking change - or am I missing something?

    That said, with -AutoFileName it would make sense to fall back to the sanitized hostname, and considering the last segment of the ultimately-redirected-to-URI makes sense then as well.
    An error would then only be reported if you don't use -AutoFileName and no suitable last-input-URI-segment is present.

  18. CarloToso commented on Mar 20, 2023

    @CarloToso
    Contributor
  19. iSazonov commented on Mar 21, 2023

    @iSazonov
    Collaborator

    I don't see a problem if we try Content-Disposition first and then make a fallback to last segment. If an user is not satisfied with this he can always use the explicit name as he does now.

  20. majkinetor commented on Apr 20, 2023

    @majkinetor

    Oh, finally, thank you for this CarloToso 🥇

  21. MatejKafka commented on Aug 20, 2023

    @MatejKafka
    Contributor

    CarloToso The provided code looks more-or-less the same as what Chromium is doing, sans some sanitization: https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L219 In general, it would make sense to me to stay close to the behavior of major browsers as much as possible, since that's what most users likely expect as the default.

  22. MatejKafka commented on Aug 22, 2023

    @MatejKafka
    Contributor

    CarloToso Note that both the Content-Disposition filename value and the URL decoded last segment may contain invalid chars (especially / and \, which could lead to nasty path traversal issues, and also the usual set of forbidden characters (Path.GetInvalidFileNameChars())) and should also be checked for DOS special files (COM0-9, AUX,...).

    The linked Chromium function has quite a lot of various checks, I'm trying to port the relevant parts to C# now (for an unrelated project), I'll post the final code if I get something usable.

  23. MatejKafka commented on Aug 23, 2023

    @MatejKafka
    Contributor

    This is what I have now (Windows-only, although only minor modifications should be needed to make it work on Linux), seems to work reasonably intuitively from my testing:

    Source code
    using System;
    using System.IO;
    using System.Linq;
    using System.Net.Http.Headers;
    using System.Text.RegularExpressions;
    using System.Web;
    
    namespace Pog.Utils.Http;
    
    /// <summary>
    /// Utility class for getting the server-provided file name for a downloaded file.
    /// </summary>
    public static class HttpFileNameParser {
        /// <param name="resolvedUri">Resolved URI (after redirects) of the downloaded file.</param>
        /// <param name="contentDisposition">The Content-Disposition header from the server response, if any.</param>
        public static string GetDownloadedFileName(Uri resolvedUri, ContentDispositionHeaderValue? contentDisposition) {
            // link to a similar function in Chromium:
            // https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L219
            // Chromium also takes into account the mime type of the response, we're not doing that for now
    
            // try available names in the order of preference, use the first one found that is valid
            string? fileName = null;
    
            // if Content-Disposition is set and valid, use the specified filename
            fileName ??= SanitizeDownloadedFileName(GetRawDispositionFileName(contentDisposition));
    
            // use the last segment of the resolved URL
            var lastSegment = resolvedUri.Segments.LastOrDefault();
            fileName ??= SanitizeDownloadedFileName(HttpUtility.UrlDecode(lastSegment));
    
            // use the hostname
            fileName ??= SanitizeDownloadedFileName(resolvedUri.Host);
    
            // fallback default name
            fileName ??= "download";
    
            return fileName;
        }
    
        private static string? GetRawDispositionFileName(ContentDispositionHeaderValue? contentDisposition) {
            if (contentDisposition is not {DispositionType: "attachment"}) {
                return null;
            }
    
            var headerVal = contentDisposition switch {
                {FileNameStar: not null} => contentDisposition.FileNameStar,
                {FileName: not null} => contentDisposition.FileName,
                _ => null,
            };
    
            if (headerVal == null) {
                return null;
            }
    
            if (headerVal.StartsWith("\"") && headerVal.EndsWith("\"")) {
                // ContentDispositionHeaderValue parser leaves the quotes in the parsed filename, strip them
                headerVal = headerVal.Substring(1, headerVal.Length - 2);
            }
    
            return headerVal;
        }
    
        private static readonly Regex InvalidDosNameRegex = new Regex(@"^(CON|PRN|AUX|NUL|COM\d|LPT\d)(\..+)?$",
                RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
    
        private static string? SanitizeDownloadedFileName(string? fileName) {
            // sources for relevant functions in Chromium:
            // GetFileNameFromURL: https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L119
            // GenerateSafeFileName: https://github.com/chromium/chromium/blob/bf9e98c98e8d7e79befeb057fde42b0e320d9b19/net/base/filename_util.cc#L163
            // SanitizeGeneratedFileName: https://github.com/chromium/chromium/blob/11a147047d7ed9d429b53e536fe1ead54fad5053/net/base/filename_util_internal.cc#L79
    
            // list of invalid filenames on Windows: https://stackoverflow.com/a/62888
    
            if (fileName == null) {
                return null;
            }
    
            // Win32 does not like trailing '.' and ' ', remove it
            fileName = fileName.TrimEnd('.', ' ');
            // replace any invalid characters with _
            fileName = string.Join("_", fileName.Split(Path.GetInvalidFileNameChars()));
    
            if (InvalidDosNameRegex.IsMatch(fileName)) {
                // is a DOS file name, prefix with _ to lose the special meaning
                fileName = "_" + fileName;
            }
    
            // if fileName is empty or only consists of invalid chars (or _), skip it
            if (fileName.All(c => c == '_')) {
                return null;
            }
            return fileName;
        }
    }
    Unit tests
    using System.Net.Http.Headers;
    using Pog.Utils.Http;
    using Xunit;
    
    namespace Pog.Tests;
    
    // list of test cases for Content-Disposition parsing: http://test.greenbytes.de/tech/tc2231/
    public class HttpFileNameParserTests {
        [Fact]
        public void TestDosNames() {
            TestSingle("_aux", "https://example.com/aux");
            TestSingle("_AUX.txt", "https://example.com/AUX.txt");
            TestSingle("_COM9", "https://example.com/COM9");
        }
    
        [Fact]
        public void TestPriority() {
            TestSingle("header", "https://host/segment", "attachment; filename=header");
            TestSingle("segment", "https://host/segment");
            TestSingle("segment", "https://host/segment", "attachment; filename=_");
            TestSingle("host", "https://host/");
            TestSingle("host", "https://host/_", "attachment; filename=_");
        }
    
        [Fact]
        public void TestSanitization() {
            // filename*
            TestSingle("hello_world.txt", "https://host/segment", "attachment; filename*=utf-8''hello%2Fworld.txt");
            // filename
            TestSingle("hello_world.txt", "https://host/segment", "attachment; filename=\"hello/world.txt\"");
            // segment
            TestSingle(".._hello_world.txt", "https://host/..%2Fhello%2Fworld.txt");
            TestSingle("invalid_chars_.txt", "https://host/invalid%2fchars%0a.txt");
        }
    
        private static void TestSingle(string expected, string url, string? dispositionHeader = null) {
            var disposition = dispositionHeader == null ? null : ContentDispositionHeaderValue.Parse(dispositionHeader);
            Assert.Equal(expected, HttpFileNameParser.GetDownloadedFileName(new Uri(url), disposition));
        }
    }
  24. ziesemer commented on Feb 19, 2024

    @ziesemer

    This remains a valid issue that should be addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Issue-Enhancementthe issue is more of a feature request than a bugKeepOpenThe bot will ignore these and not auto-closeWG-Cmdletsgeneral cmdlet issuesWG-NeedsReviewNeeds a review by the labeled Working Group

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions