I would love to see SFTP and FTPS on that list. Or at least keep them in mind when considering future extensions to the cmdlet.

@KevinMarquette My plan is to start by fixing the gap created by the API switch from WebRequest to HttpClient where we lost FTP. Use that to mature the code, and then open it up for inclusion of more protocols.

I hope that eventually this could support FTPS, SFTP, and SCP. Assuming we can find good .NET libraries to do those, the interfaces could be used to wrap pretty much any implementation and then plugged into the ProtocolHandlers dictionary. However, I chose to initially only support HTTP, HTTPS, FTP, and FILE to keep it simple, mature the code, and provide a sold set of examples on how to implement protocols.

Then we can all start plugging our own funky protocols in markekraus://markekraus.com:11111/SomeFile.exe. 😄

So the goal here is to start small, but make it easily extensible. for new features and new protocols.

I'm all for anything that increases utility with PS, and this seems like a much-needed addition for PS Core. The ability to handle multiple different downloading protocols would definitely help PS have a much more useful presence.

I find myself wondering if PS would ever be capable of handling torrent downloads via magnet: links, but that is a whole other project right there!

To be clear, you're planning on bringing over the -Resume parameter from the Invoke-WebRequest cmdlet as well? Because that would be one heck of an oversight otherwise!

Also, regarding filename conflicts. I can definitely see the utility here in providing files with a datetime stamp where conflicts arise. However, I'm not sure that really has a massive amount of merit, as the file itself would carry that metadata (CreationTime) regardless, would it not?

Perhaps a simpler approach that is more commonly present would also be okay. Most browsers simply append an integer to the filename, incrementing it until they get a unique filename. I think that's likely to be perfectly sound for most purposes.

I think this is a great idea. You mentioned "Automatic Filename Resolution Feature" and "AsJob" in Download-Cmdlet.md and this really hit home because I actually wrote some C# not too long ago that accomplished exactly those features. It's mostly cobbled together from a few stackoverflow threads, so I don't really feel comfortable sharing (i.e. not a lot of original thought went into it), but I definitely had a need for those features.

Along the same lines, I also wrote something that checks if there is enough space on the target drive before downloading the target. Not sure if that's something you could fit into your plans. In case you're interested, here's the basic idea:

$HTTP_Request = [System.Net.WebRequest]::Create($TargetDownloadUrl)
$HTTP_Response = $HTTP_Request.GetResponse()

$bytes = $HTTP_Response.GetResponseHeader("Content-Length")
$TargetSizeInMB = [Math]::Round($bytes / 1MB)

# Determine if we have enough space on the $DownloadDirectory's Drive before downloading
# NOTE: Use 'LinkType' property just in case it's a symlink
if ([bool]$(Get-Item $DownloadDirectory).LinkType) {
    $DownloadDirLogicalDriveLetter = $(Get-Item $DownloadDirectory).Target[0].Substring(0,1)
}
else {
    $DownloadDirLogicalDriveLetter = $DownloadDirectory.Substring(0,1)
}
$DownloadDirDriveInfo = Get-CimInstance Win32_LogicalDisk -ComputerName $env:ComputerName -Filter "DeviceID='$DownloadDirLogicalDriveLetter`:'"

# Make sure we always have at least 2GB of free space left after download
if ($([Math]::Round($DownloadDirDriveInfo.FreeSpace / 1MB)-2000) -lt $TargetSizeInMB) {
    Write-Error "Not enough space on $DownloadDirLogicalDriveLetter`:\ Drive to download the target file! Halting!"
    $global:FunctionResult = "1"
    return
}

Also, I'm pretty sure the above won't always work depending on the website hosting the file. But I think you know a lot more than me about that stuff.

Can we not use Invoke? In fact, can we not name this generically at all?

How about Get-RemoteFile? It doesn't download non-file things, right?

@Jaykul I'm not married to the name. I have even considered Get-RemoteFile. I'll use what ever makes the most sense.... i'm bad at naming things :)

@vexx32 I have updated the PR to include the missing resume documentation. I had it listed in the syntax, but didn't call it out as a feature explicitly.Originally, that feature I had planed to only implement in the new cmdlet and not make it available in the web cmdlets, but there would be some pain points for advanced download scenarios that it still needed to address. In any case, the Resume feature is a first class feature and requires the protocol handler to implement IInvokeDownloadResumable

Re: auto name resolution using date instead of an iterated integer. It's not about meta data. The reason it was chosen is that it is less likely to already exist, that makes it easier to implement. if you have use wget enough you know that multiple files get downloaded maybe you delete the .1 but the next one is still .50... it's a pain point I have always had with that implementation. the time stamp makes it quick and sortable.

@pldmgg re: space check. I think that is something we can iterate with later and maybe not focus on for the initial inclusion. if enough transfer protocols support the ability to query the size before transfer, then it can be extended with another interface and some more internal logic.

After doing some mocking of this, I think I should revisit the interface definitions. I personally hate methods so I don't even know why I originally chose them. I think properties work best for most of those.

Also, I realized that the Dictionary would be awkward if the handler is directly queried from there. So it would probably be best to include a facorty interface and use that as the value type for the dictionary:

    internal interface IInvokeDownloadProtocolFactory
    {
        IInvokeDownloadProtocol NewInvokeDownloadProtocol();
    }
    public class InvokeDownloadCmdlet : PSCmdlet
    {
        static Dictionary<string,IInvokeDownloadProtocolFactory> ProtocolHandlers;
    }

Anyway, I have done a mockup of the interfaces with properties and the new factory and how they would work in the cmdlet here:

https://gist.github.com/markekraus/61016e81bde2198cdc4faf330c1e80d9

It's obviously not functional code. It's more just to demonstrate how I envision the interfaces working in the cmdlet to enable features on a protocol.

I'll wait for some more feed back before updating the RFC with those changes.

Also, as far as determining the file name, what about the Content-Disposition header -- you know, this thing:

Content-Disposition: attachment; filename="filename.jpg"`

I think there are a couple of other tricks too -- otherwise, you're going to end up with a lot of downloads named download.aspx and the like 😉

I think the alternative of using incrementing numbers to differentiate is probably desirable, because the full date-time stamp is going to make file names extremely large (although perhaps this could be an option?)

Finally, I wonder if you shouldn't passthru by default. Specifically, if you're generating names whenever you hit a collision, that will mean people will almost always need to -Passthru in order to reliably write scripts using this cmdlet.

@Jaykul

Re: Content-Disposition. I've gone back and forth with this in my head for months. Assuming the cmdlet is protocol agnostic, and not all protocols provide the file name, coupled with resolving the name when a preexisting file has that name... I could not come up with a decent implementation that would both allow the protocol handler to suggest a name and for the cmdlet to handle the incrementing. Different protocol handlers would implement this differently and documenting it would be a pain. I'm open to design suggestions there, but I settled on this because it was predictable.

I think the alternative of using incrementing numbers to differentiate is probably desirable

I will add two comments here, one for using the time stamp and another for using the incremental number and let people vote on them. I personally hate the .1 shenanigans (it is my least favorite thing about modern browsers, wget, and curl), but I also agree that the timestamp makes for long file names.

Finally, I wonder if you shouldn't passthru by default.

That's a good point. and if this is changed to a Get- verb that would be the preferable mode.

If you would prefer the automatic name resolution to use .1 if the file exists and to increment until a filename is found that doesn't exist. Give this comment thumbs up. Thumbs down will not be counted, instead vote for the other comment.

Downloading file.exe: if file.exe already exists, file.1.exe will be used. if file.exe and file.1.exe exist, file.2.exe will be used.

If you would prefer the automatic name resolution to use the timestamp if the file exists and to try again until a filename is found that doesn't exist. Give this comment thumbs up. Thumbs down will not be counted, instead vote for the other comment.

Downloading file.exe: if file.exe already exists, file.20180331124709627.exe will be used. if file.exe and file.20180331124709627.exe exist, file.20180331124709945.exe will be used.

@Jaykul , @markekraus

I think the Invoke- verb is best here. Get- typically refers to things in the local ecosystem. Also, I can't think of any Get- cmdlets that have the -AsJob parameter like seems to be proposed here. Invoke- carries the implication that it could take a good amount of time to complete the task, as it well could.

Vote for Invoke-Download. Thumbs down will not count, instead vote on the other comment.

Vote for Get-RemoteFile. Thumbs down will not count, instead vote on the other comment.

I think Get-RemoteFile is not the best name for the cmdlet, since you could use it to retrieve files from localhost services as well

We got into an intense discussion today with @PowerShell/powershell-committee on the naming today.

We brainstormed a little without thinking about approved verbs. When I think of downloading a file, I think "download, fetch, get, pull, obtain, acquire".

@JamesWTruher felt pretty strongly about using Receive-File, but my take was that Receive was associated with a listener in my mind.

Where we landed--and we'd like feedback because we haven't had time time to think through this all the way--was that we should have a greater discussion on whether it might make sense to significantly expand Copy-Item to support more parameter sets for more protocols. There's already precedent there with -From/ToSession, and it would be cool for us to add new providers like FTP that could insert themselves into Copy-Item.

What are folks' thoughts on this approach?

That being said, it's worth mentioning we have PowerShell/PowerShell#4714 which might be worth revisiting as well

@joeyaiello I'm not a fan of adding this functionality to copy-item. While that initially seems like a good idea, one motivation behind this is multiple requests across issues, at conventions, on twitter, and in person for functionality similar to wget

wget $url

Where it downloads the file and saves it to the CWD without a tone of parameters/arguments.

Though, from the perspective of having something lightweight added into the base language (allowing for micro-installations of PS without all the additional modules) baking some kind of base functionality for HTTP/FTP get into Copy-Item does seem attractive. However, Copy-Item has a complex overloaded API and I feel adding web functionality to is is just going to open flood gates on extending that API further.

On the naming..... *sigh... can someone PLEASE look at the API and see if it makes sense? I don't care what color the bikeshed is.... I want to know if it will stand on its own and keep the rain out. ;-P

@markekraus Yes, it may be more complicated, but the desired experience for download is:

copy-item https://foo.com/file.zip
copy-item ftp://foo.com/file.zip
copy-item smb://foo.com/file.zip
copy-item sftp://foo.com/file.zip

Uploads would be similar.

@SteveL-MSFT Does that mean this would be supported too?

Copy-Item c:\some\other\file\path\file.txt

@markekraus well, if you follow Explorer, it would create file (1).txt...

I don't think remote and local operations have to be symmetric

They certainly don't have to be, but that would mean further causing the behaviour of the cmdlet to vary in terms of what parameters are expected depending on which type of provider path you give it. I'm not sure diverging the behaviour there makes sense. I'm sure the dynamic parameter code for that cmdlet is already far too complicated. 😕

Well, Copy-Item c:\some\other\file\path\file.txt already works (although probably known to few, including me up to 5 minutes ago) and implicitly uses the current dir. as the -Destination - so, no variation needed.
(If the source and destination are the same, you get Cannot overwrite the item <path> with itself.)

Those used to both a source and a destination argument being required can simply use . as the
-Destination:

copy-item https://foo.com/file.zip .

@mklement0 oh wow... I should not have just assumed that didn't work... O_O

Yeah, @markekraus, it is is surprising; as I just found out, cmd.exe's internal copy command and xcopy.exe support omitting the destination argument as well, but not the Unix cp utility.

I personally would like a new cmdlet instead of overriding Copy-Item. I just think it's more discoverable that way, and also, the resulted code is probably easier to maintain.

@daxian-dbw Can you provide some feedback on what I propose for the internals? There has been a ton of discussion about the name and public PowerShell command API, but I've had very little feedback on the internal design.

@daxian-dbw:

I personally would like a new cmdlet instead of overriding Copy-Item

I understand the desire to avoid monolithic cmdlets that are overwhelming in terms of both implementation complexity and UX (a bewildering array of parameter sets).

How about CopyFrom-Uri?
This gives use the commonality in terms of what action is being performed (to effectively copy something) and specific discoverability (via the term URI).

Of course, that implies that we may need a separate CopyTo-Uri cmdlet, but I'd expect that substantial parts of the implementation could be shared.

Tabling the discussion on the name right now, as we wasted lots of time last week on it without no meaningful progress.

Right now, we in the @PowerShell/powershell-committee think that this implementation proposal looks mostly right, but we won't be sure about the interface definitions until someone (ideally @markekraus) starts implementing a couple different protocols against it. Ideally, we would like to have someone else (on @SteveL-MSFT's team, if no one else signs up) to implement one of these protocols to validate that the interface is general enough.

For now, we say rock on the implementation behind an experimental feature flag, and we can revise this RFC as we learn more.

Can you make any planned updates, update your filename and front-matter to RFC0034, and re-target the 1-Draft folder (we're going to hold off on a full acceptance until we review the partial implementation).

I love how PowerShell can do magical things. From this point of view, it is much more promising to implement Copy-Item and inter-provider copying. It realizes fantastic flexibility.
Of course make the cmdlet easier. We could start with a separate experimental nuget package. Then implement full provider.

If this path is correct then the name of the cmdlet is not essential.

Tabling the discussion on the name right now, as we wasted lots of time last week on it without no meaningful progress.

@joeyaiello Since there is no clear preference, you could ask the project’s founders to decide

Given how we're moving forward on experimenting with modules/cmdlets in the Gallery before delivering them into PowerShell, we'd love to see someone ship a prototype or v1 of this to validate that the current thinking makes sense (and to exhaust the naming issue/discussion in the wild).

Given the gap since Waiting on Author, we're also marking as Abandoned. Anyone else who is willing to take this one should also feel free to publish a prototype to the Gallery.

@joeyaiello, is this path to potential inclusion in PowerShell itself via gallery-published prototypes documented / formalized somewhere, in a detailed, procedural manner? It would help to have a document to point people to that explains the process.