You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
EDIT - Docs announcement - I will be doing a presentation to RTPUG on WEDNESDAY evening 8pm EDT. I will be talking about PowerShell-Docs and how to contribute.
This is happening tonight 9/16, so this will be too late for the community call.
Reacted by Rain Sallow (/u/ta11ow), TylerLeonhardt and Justine
Discussion topic: separation of native cmdlets into modules that are compatible with Windows PowerShell 5.1 as well as PowerShell 7.x.
Specifically, I'm wondering about the ConvertTo-Json and ConvertFrom-Json cmdlets, the need to move those forward to .NET Core JSON APIs, the need to provide SDK-level access to JSON conversion APIs so that .NET code that uses S.M.A. can do conversions without having to invoke PowerShell, and the desire to address the issues with the JSON cmdlets that would otherwise be breaking if those changes were to be made in the native, built-in cmdlets we have today.
TylerLeonhardt and Rob Holt (@rjmholt) That's great for .NET Core, but that doesn't help Windows PowerShell, right? Although I suppose I could have a separate DLL for .NET Core that wraps those APIs and exposes them to my Windows PowerShell .NET Framework executable.
Even with that in place though, it doesn't speak to the desire to move forward to .NET Core JSON APIs, how to deal with case-insensitive JSON that contains multiple entries with the same value but different cases (as can happen from a simple Get-ADUser -filter * -properties * | ConvertTo-Json), right? Or maybe it does address some of my issues if it exposes the -AsHashtable functionality via the SDK. I'll have to take a closer look.
Would be good to document this usage somewhere for cmdlet authors to reference, I think. Had no idea we could do that. 👀
Needs to go in the XML comments so it can appear in the dev documentation.
That's great for .NET Core, but that doesn't help Windows PowerShell, right?
Yeah, this API is only available in PS 7+. Windows PowerShell has half the API (not such which) but is missing the other half, and in fact I think uses a totally different JSON engine.
To pick apart your question a bit better so I understand it:
You'd like to convert JSON in SDK code, away from the pipeline (so no PowerShell invocation required)
You want to customise some behaviour, like case-insensitive key handling
Also, what happens when the underlying JSON API changes
So to answer as best I can given my understanding:
PowerShell's own JSON conversion APIs are provided not to convert arbitrary JSON, but to convert JSON in a PowerShell-specific way. There are other, lower-overhead ways to convert JSON, but they won't handle PSObjects or special keys in objects the way PowerShell does.
The behaviour of those conversion APIs is tuned to PowerShell's conversion cmdlets. They don't seek to offer configuration, but instead align with the Convert*-Json cmdlets.
For your own custom JSON conversion needs, you should prefer a configurable API that isn't coupled to PowerShell, like the Newtonsoft library or the new .NET JSON conversion library.
As for moving to .NET's new JSON library in our own implementation, our intent is very much for users not to know or care. Newtonsoft and System.Text.Json are both already shipped in PS 7. Which one is used by PowerShell's own JSON conversion logic should be transparent to consumers.
Let me know if that makes sense or doesn't address what you're looking for
PowerShell's own JSON conversion APIs are provided not to convert arbitrary JSON, but to convert JSON in a PowerShell-specific way. There are other, lower-overhead ways to convert JSON, but they won't handle PSObjects or special keys in objects the way PowerShell does.
That's fine, I'm only doing conversion on PowerShell object data that is returned from a script. I have a runtime that runs PowerShell, and then I send the results off via a REST API call. Those results need to be converted into JSON first, and compressed. Then we decompress and store them. So this is exactly what I need, not some general purpose JSON conversion that isn't about PSObjects, special keys, etc.
As for moving to .NET's new JSON library in our own implementation, our intent is very much for users not to know or care. Newtonsoft and System.Text.Json are both already shipped in PS 7. Which one is used by PowerShell's own JSON conversion logic should be transparent to consumers.
Right, end users shouldn't know or care, but if the backend library is more performant, and possibly more forgiving/inline with PowerShell that prefers case insensitive over case sensitive information, as someone who works with a runtime that executes PowerShell, I do have some preferences over what is used on the back end, especially when it comes to performance.
Thanks Rob Holt (@rjmholt) for those answers. My interest in a cross platform standalone JSON library may become significantly less important if I can plug in these APIs and cherry pick the newer functionality that is in the SDK that powers those cmdlets in PowerShell 7.x.
So this is exactly what I need, not some general purpose JSON conversion that isn't about PSObjects, special keys, etc.
Understood.
Right, end users shouldn't know or care, but if the backend library is more performant, and possibly more forgiving/inline with PowerShell that prefers case insensitive over case sensitive information, as someone who works with a runtime that executes PowerShell, I do have some preferences over what is used on the back end, especially when it comes to performance.
Absolutely. And it's something we've been discussing on and off. Certainly if we were making the move today we'd be using System.Text.Json. The concern is that making the move without breaking small edge cases is hard. On that note, the case sensitivity isn't an implementation detail of the JSON engine, it's a deliberate choice for PowerShell, since PowerShell tends to think case-insensitively -- so moving over to a new engine, we'd keep that behaviour. If you needed to change that behaviour, you'd need to move away from PowerShell's own JSON APIs to get the specific behaviour you're looking for.
a cross platform standalone JSON library
For this goal, I would say you've got a couple of possibilities:
Wrap calls to Convert*Json with the PowerShell.AddCommand().Invoke() API
Wrap a runtime-multiplexed call to the appropriate underlying PowerShell API
Implement your own conversion logic based on a cross-platform library
(2) tends to be what we do in PowerShellEditorServices. For example, the debugging APIs introduced (with your help Kirk Munro (@KirkMunro)) in PS 7 allow us to do new things in the debugger (for example set a breakpoint while script is running), but don't exist in older versions. So at load-time we do some reflection to discover those APIs and if they exist we call them, but if not we fall back to calling the appropriate cmdlet (in this case Set-PSBreakpoint). For JSON cmdlets, I would look for the corresponding API and if it doesn't exist, fall back to calling Convert*-Json on a pipeline.
The discussion with Kirk Munro (@KirkMunro) was started in #11198 and I hope MSFT team accept my suggestion PowerShell/PowerShell#11198 (comment) (in short - to use .Net Runtime dev model) - It's incredible but it's easier to invest in .Net today than in PowerShell.
We're establishing Working Groups to decentralize management of Area-* labeled PRs and issues
What is a progress in the intention?
I would like to invest more in file system provider, web and json cmdlets (for which we have many issues), engine optimizations but now it just doesn't make sense...
.NET MAUI (cross-platform GUI) could allow for PowerShell's GUI cmdlets to support Mac/Linux.
PowerShell's updatable help is a powerful but underutilized feature. One of the blockers is hosting the HelpInfo.xml and cab files on a web server. Could PowerShell Gallery be updated to allow module authors to host these files?
Reacted by Michael Klement and Rain Sallow (/u/ta11ow)
Discuss the path toward resolution of PowerShell/PowerShell#1995 (broken passing of empty arguments and arguments with embedded double quotes to native executables), specifically:
The need to widely publicize this experimental feature once implemented, to gather important feedback whether fixing the problem by default is acceptable, despite breaking existing workarounds, so as to put an end to 15 years of quoting hell.
Also, I feel like PSStandard still includes a ton of DLLs that have to be manually removed for Windows Powershell 5.1 for those modules to import correctly with Add-Type, it seems like an additional privateassets dependency list may be required to cut down on the assemblies generated for a binary/hybrid Powershell Module. Any potential for optimizing some of that?
Q: Any updates on Assembly Load Context, etc. to work with side-by-side issues between modules? I imagine system.text.json is going to become the new common problem here. Maybe add something to import-module that if multiple modules are added simultaenously, discover from the manifests the highest version assembly and load that first so the others will binding redirect up in .NET core?
Reacted by Rain Sallow (/u/ta11ow) and Thomas Nieto
Activity
sdwheeler commentedon Sep 15, 2020
EDIT - Docs announcement - I will be doing a presentation to RTPUG on WEDNESDAY evening 8pm EDT. I will be talking about PowerShell-Docs and how to contribute.
This is happening tonight 9/16, so this will be too late for the community call.
KirkMunro commentedon Sep 15, 2020
Discussion topic: separation of native cmdlets into modules that are compatible with Windows PowerShell 5.1 as well as PowerShell 7.x.
Specifically, I'm wondering about the
ConvertTo-JsonandConvertFrom-Jsoncmdlets, the need to move those forward to .NET Core JSON APIs, the need to provide SDK-level access to JSON conversion APIs so that .NET code that uses S.M.A. can do conversions without having to invoke PowerShell, and the desire to address the issues with the JSON cmdlets that would otherwise be breaking if those changes were to be made in the native, built-in cmdlets we have today.Relevant discussion: tail end of the conversation on this open PR: https://ucclearly.myjetbrains.com/youtrack/dashboard?id=e5b14068-e378-437b-baa4-7199fc8f9a17.
TylerLeonhardt commentedon Sep 15, 2020
Kirk Munro (@KirkMunro) I believe we added this in 6.2 because of Azure Functions
rjmholt commentedon Sep 15, 2020
In https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/JsonObject.cs.
Example of usage: https://github.com/PowerShell/PowerShell/blob/3effa204103460c996a8612aa70718fdf924047d/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommand.cs#L115-L125
vexx32 commentedon Sep 15, 2020
Would be good to document this usage somewhere for cmdlet authors to reference, I think. Had no idea we could do that. 👀
KirkMunro commentedon Sep 15, 2020
TylerLeonhardt and Rob Holt (@rjmholt) That's great for .NET Core, but that doesn't help Windows PowerShell, right? Although I suppose I could have a separate DLL for .NET Core that wraps those APIs and exposes them to my Windows PowerShell .NET Framework executable.
Even with that in place though, it doesn't speak to the desire to move forward to .NET Core JSON APIs, how to deal with case-insensitive JSON that contains multiple entries with the same value but different cases (as can happen from a simple
Get-ADUser -filter * -properties * | ConvertTo-Json), right? Or maybe it does address some of my issues if it exposes the-AsHashtablefunctionality via the SDK. I'll have to take a closer look.rjmholt commentedon Sep 15, 2020
Needs to go in the XML comments so it can appear in the dev documentation.
Yeah, this API is only available in PS 7+. Windows PowerShell has half the API (not such which) but is missing the other half, and in fact I think uses a totally different JSON engine.
To pick apart your question a bit better so I understand it:
So to answer as best I can given my understanding:
Convert*-Jsoncmdlets.Let me know if that makes sense or doesn't address what you're looking for
KirkMunro commentedon Sep 15, 2020
That's fine, I'm only doing conversion on PowerShell object data that is returned from a script. I have a runtime that runs PowerShell, and then I send the results off via a REST API call. Those results need to be converted into JSON first, and compressed. Then we decompress and store them. So this is exactly what I need, not some general purpose JSON conversion that isn't about PSObjects, special keys, etc.
Right, end users shouldn't know or care, but if the backend library is more performant, and possibly more forgiving/inline with PowerShell that prefers case insensitive over case sensitive information, as someone who works with a runtime that executes PowerShell, I do have some preferences over what is used on the back end, especially when it comes to performance.
Thanks Rob Holt (@rjmholt) for those answers. My interest in a cross platform standalone JSON library may become significantly less important if I can plug in these APIs and cherry pick the newer functionality that is in the SDK that powers those cmdlets in PowerShell 7.x.
rjmholt commentedon Sep 15, 2020
Understood.
Absolutely. And it's something we've been discussing on and off. Certainly if we were making the move today we'd be using System.Text.Json. The concern is that making the move without breaking small edge cases is hard. On that note, the case sensitivity isn't an implementation detail of the JSON engine, it's a deliberate choice for PowerShell, since PowerShell tends to think case-insensitively -- so moving over to a new engine, we'd keep that behaviour. If you needed to change that behaviour, you'd need to move away from PowerShell's own JSON APIs to get the specific behaviour you're looking for.
For this goal, I would say you've got a couple of possibilities:
Convert*Jsonwith thePowerShell.AddCommand().Invoke()API(2) tends to be what we do in PowerShellEditorServices. For example, the debugging APIs introduced (with your help Kirk Munro (@KirkMunro)) in PS 7 allow us to do new things in the debugger (for example set a breakpoint while script is running), but don't exist in older versions. So at load-time we do some reflection to discover those APIs and if they exist we call them, but if not we fall back to calling the appropriate cmdlet (in this case
Set-PSBreakpoint). For JSON cmdlets, I would look for the corresponding API and if it doesn't exist, fall back to callingConvert*-Jsonon a pipeline.iSazonov commentedon Sep 16, 2020
The discussion with Kirk Munro (@KirkMunro) was started in #11198 and I hope MSFT team accept my suggestion PowerShell/PowerShell#11198 (comment) (in short - to use .Net Runtime dev model) - It's incredible but it's easier to invest in .Net today than in PowerShell.
What is a progress in the intention?
I would like to invest more in file system provider, web and json cmdlets (for which we have many issues), engine optimizations but now it just doesn't make sense...
ThomasNieto commentedon Sep 17, 2020
Michael Klement (@mklement0)'s issue/comment on removing the requirement for curly braces when using null-conditional operator. Do not require { ... } around variable names for null-conditional member access PowerShell#11379 (comment)
.NET MAUI (cross-platform GUI) could allow for PowerShell's GUI cmdlets to support Mac/Linux.
PowerShell's updatable help is a powerful but underutilized feature. One of the blockers is hosting the HelpInfo.xml and cab files on a web server. Could PowerShell Gallery be updated to allow module authors to host these files?
mklement0 commentedon Sep 17, 2020
Discuss the path toward resolution of PowerShell/PowerShell#1995 (broken passing of empty arguments and arguments with embedded double quotes to native executables), specifically:
The state of the
PSEscapeDoubleQuotedStringForNativeExecutablesexperimental-feature PR WIP: Try to manage quotes in strings when calling native executables PowerShell#13482, with respect to:msiexec-style CLIs on Windows: Arguments for external executables aren't correctly escaped PowerShell#1995 (comment)The need to widely publicize this experimental feature once implemented, to gather important feedback whether fixing the problem by default is acceptable, despite breaking existing workarounds, so as to put an end to 15 years of quoting hell.
JustinGrote commentedon Sep 17, 2020
Q: What is the plan for Powershell Standard with .NET 5 since the .NET Standard approach is changing?
https://devblogs.microsoft.com/dotnet/the-future-of-net-standard/
Also, I feel like PSStandard still includes a ton of DLLs that have to be manually removed for Windows Powershell 5.1 for those modules to import correctly with Add-Type, it seems like an additional privateassets dependency list may be required to cut down on the assemblies generated for a binary/hybrid Powershell Module. Any potential for optimizing some of that?
JustinGrote commentedon Sep 17, 2020
Q: Any updates on Assembly Load Context, etc. to work with side-by-side issues between modules? I imagine system.text.json is going to become the new common problem here. Maybe add something to
import-modulethat if multiple modules are added simultaenously, discover from the manifests the highest version assembly and load that first so the others will binding redirect up in .NET core?SteveL-MSFT commentedon Sep 17, 2020
Thanks everyone. Listen to the recording for answers/discussion to these questions.
ili101 commentedon Sep 23, 2020
Steve Lee (@SteveL-MSFT) Can you share the recording?
Thank you
mklement0 commentedon Sep 23, 2020
In case there are others (like me until now) who are unfamiliar with the process: The recordings and transcripts are usually posted at https://aka.ms/PSCommunityCall, which, however, hasn't happened yet for the September call.
vexx32 commentedon Sep 23, 2020
ili101 Prior to those going up you can also view the most recent recording via Teams itself from the https://aka.ms/JoinPSCall link, which will have the prior meeting's recording available until they update that shortlink for the next month's meeting.
mklement0 commentedon Sep 24, 2020
Thanks, Rain Sallow (/u/ta11ow) (@vexx32), good to know.
However, it would still be beneficial to have the YouTube link posted soon after and within a predictable time frame:
It is a permalink.
It allows linking to specific portions of the recording, which can be helpful in discussions on issues.