Skip to content

Conversation

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Nov 26, 2019

PR Summary

Address #8393

  1. Move to new .Net Core Json API. Detects and throws on cycles and depth.
  2. Add PSObject, DBNull, NullString, Enum64 Json converters
  3. Add JsonIgnoreAttribute support. Address ConvertTo-Json Honors JsonPropertyAttribute #6847
  4. Set Depth default to 64 (.Net Core default). It is good compromise: still no hangs and exceptions for common scenarios but noticeable delays are if problems exist.
  5. Set MaxDepthAllowed = 1000. It is .Net Core limitation.
  6. Fix Class hidden properties still serialized to JSON #9847 as side effect.
  7. Escaping based on new .Net Core Json API. 'Default' is nearly like previous. 'EscapeHtml' - more differences, 'EscapeNonAscii' now like 'Default' but could be improved but I did not found noticeable using EscapeHtml and EscapeNonAscii on GitHub.
  8. Cancellation doesn't work (I guess only hosted scenarios are affected). Underlying sync API don't support this. I did not found reasons to switch to async Json API. See Enhance Json Serialize() with CancellationToken dotnet/runtime#611
  9. Depth parameter is renamed to MaxDepth to more accurately reflect the new semantics.

PR Context

https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-migrate-from-newtonsoft-how-to

PR Checklist

@iSazonov
Copy link
Collaborator Author

@mklement0 @adamdriscoll @rkeithhill @CollinChaffin @msftrncs @markekraus @vexx32 @SteveL-MSFT @daxian-dbw @iRon7 @mdalepiane
Welcome to download compiled artifact and discuss. Please also look failed tests that can be useful for the discussion.

@iSazonov iSazonov force-pushed the convertto-json-core branch from 00d9900 to 9974612 Compare November 26, 2019 06:11
@SteveL-MSFT SteveL-MSFT added this to the 7.1.0-preview.1 milestone Dec 2, 2019
@iSazonov iSazonov force-pushed the convertto-json-core branch from a26f08e to 77a4667 Compare December 3, 2019 15:12
@iSazonov iSazonov force-pushed the convertto-json-core branch from 3711b76 to 0167cf4 Compare December 4, 2019 11:33
@KirkMunro
Copy link
Contributor

KirkMunro commented Sep 14, 2020

@iSazonov Can we please move this forward as a new, standalone library that is compatible with Windows PowerShell 5.1 as well as PowerShell 7.x? The issues identified here are also issues in 5.x. There have been enhancements in the cross platform PowerShell such as -EnumsAsStrings and -AsHashtable and -AsArray and -EscapeHandling that are not available to Windows PowerShell 5.1. Yet there are scenarios where simply pointing users to PowerShell 7 as the solution forward is not viable. Plus there are discussions here about breaking changes. A standalone module that carries JSON conversion forward using the .NET Core JSON APIs while addressing the bulk of the issues with JSON in PowerShell today (including fully bi-directional conversion) would be a much better way to bring the Microsoft JSON.NET libraries into PowerShell than simply updating what we have as a built-in module that isn't available elsewhere.

Also this hasn't moved forward in some time. Is that partly due to concerns about these breaking changes? If so, again, a standalone optional module that supersedes what Windows PowerShell 5.1 and PowerShell 7.x have baked in today seems like a better solution.

@KirkMunro
Copy link
Contributor

Looking at the code, I see you've implemented an entirely new set of classes, so hopefully it should be straightforward to move that into a standalone module that could work for Windows PowerShell 5.1 as well as PowerShell 7.x.

Also, while it seems like this is covered, a use case that I want to specifically call out is the need to access the conversion to/from JSON via an API, so that the same logic that is used within PowerShell can be used in products that leverage PowerShell without having to invoke PowerShell directly. Please make sure this use case is covered.

If the conversion logic was localized within a library that PowerShell 7.x used in cmdlets, and if that library was packaged up as a Nuget package, that would cover the needs of developers building solutions on top of PowerShell as well -- at least in my case it would.

@KirkMunro
Copy link
Contributor

One more thing @iSazonov: I have immediate need for this in Windows PowerShell 5.1, or as a library, and I'm happy to assist with the development and testing. I don't want to simply create duplicate work in parallel though, so please let me know what the plans are to move this forward, whether or not you think creating this as a standalone module is feasible and something you agree with, and we'll go from there. If you're too busy and you'd like me to move this forward, I'm happy to do so.

@SteveL-MSFT: Curious to hear your thoughts on the idea of this being broken out into a separate module and NuGet library (library necessary to consume/convert from JSON produced from PowerShell scripts in an environment that does not run PowerShell).

@KirkMunro
Copy link
Contributor

KirkMunro commented Sep 14, 2020

Does this PR resolve #3705 such that the -AsHashtable workaround is not required? i.e. With a case insensitive JSON conversion logic, what happens when it is fed some json that contains case-sensitive property names? Hopefully it has an option to just write each value once, because in scenarios where you can get duplicate property names with the same values but a different case, it would be nice if you could just ignore those scenarios.

An example where such a scenario can occur is this:
Get-ADUser -Filter * -Properties * | ConvertTo-Json | ConvertFrom-Json

That will generate JSON containing both ObjectGuid and ObjectGUID, both of which contain the same value.

@iSazonov
Copy link
Collaborator Author

I am fan .Net Runtime development model. I strongly sure we could benefit from the dev model too. They put all code dev process in one repository and greatly simplify it. As result they get huge community contributions.
With the model all you would need to do you would add publishing of a new nuget package.

Unfortunately, PowerShell MSFT team cannot find a middle ground - many community PRs are frozen here and PowerShell/modules repository does not work at all.
Of course, you can always develop your own project to meet your needs... I suspect that if you need quick results today, then you are destined for it (own project). :-(

Also this hasn't moved forward in some time. Is that partly due to concerns about these breaking changes?

My intentions were:

  • to create early implementation so that we can feedback to .Net team while they actively worked on .Net 5.0.
  • to avoid breaking changes as possible and pass all our current tests without changes
  • later to add new fixes in separate commits so that we explicitly see new breaking changes.

As for breaking changes, there are two places:

  • in .Net
    They implemented new API with compatibility with NewtonSoft Json.Net and fix some bugs and inconsistences. As result the new API has many breaking changes compared with Json.Net ( and PowerShell too because PowerShell uses Json.Net)
  • in PowertShell
    PowerShell has bugs and inconsistences too. Fixing its will add breaking changes.

In the end, we come to the conclusion that in any case, when we switch to the new API, we will receive many breaking changes. I hoped that we get this in 7.1 as experimental feature for easy community adoption. But :-(

a use case that I want to specifically call out is the need to access the conversion to/from JSON via an API, so that the same logic that is used within PowerShell can be used in products that leverage PowerShell without having to invoke PowerShell directly.

Only specific convertor is created for PSObject. Do you ask to put it in nuget library and reference SMA (PowerShell SDK)?

I have immediate need for this in Windows PowerShell 5.1, or as a library, and I'm happy to assist with the development and testing.

Press MSFT team :-) to follow .Net Runtime dev model. This is my strict belief. A good development environment is 80 percent of the effort. If we do this in one place, we will significantly speed up development.

Does this PR resolve #3705 such that the -AsHashtable workaround is not required?

This was not my goal. As I said, I wanted to avoid any breaking changes and bug fixes and make them explicitly later.

@ghost ghost removed the Review - Needed The PR is being reviewed label Sep 15, 2020
@KirkMunro
Copy link
Contributor

@iSazonov Thank you for the detailed reply.

Of course, you can always develop your own project to meet your needs... I suspect that if you need quick results today, then you are destined for it (own project). :-(

This is what we have now -- some proxy functions to override the ConvertTo-Json/ConvertFrom-Json cmdlets where needed. But ultimately I don't want to spend time on something that should come from PowerShell itself as a separate module, and I don't want to own the development of something that is so fundamental that it really should be managed by Microsoft. I'm doing it because I have to, but with bugs like #3705 plus performance benefits from switching to .NET Core JSON APIs, I'd rather converge on a PowerShell team-managed solution than roll my own, because I have much bigger things to focus on that are far more important than JSON conversion (which should be something people can simply take for granted as being there with a solid, performant implementation, in both Windows PowerShell and PowerShell).

In the end, we come to the conclusion that in any case, when we switch to the new API, we will receive many breaking changes. I hoped that we get this in 7.1 as experimental feature for easy community adoption. But :-(

I think a standalone module and Nuget package would be far superior to an experimental feature in this case. But I suppose I already said that.

Only specific convertor is created for PSObject. Do you ask to put it in nuget library and reference SMA (PowerShell SDK)?

That would be fine. I have no problem needing the PowerShell SDK as a dependency. I'm already using it. I'm also only looking for conversion to/from PS entities such as PSObject, Hashtable, etc. I just want to avoid needing to call into PowerShell when I already have my object data that needs conversion in my C# code.

@ghost ghost added the Review - Needed The PR is being reviewed label Sep 23, 2020
@ghost
Copy link

ghost commented Sep 23, 2020

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

@strawgate
Copy link
Contributor

Any idea what is needed to get this PR ready to go? I see a couple of things but if you've got a list that would be super helpful

@iSazonov
Copy link
Collaborator Author

iSazonov commented Jul 2, 2021

Any idea what is needed to get this PR ready to go? I see a couple of things but if you've got a list that would be super helpful

I can not do all work - investigate, develop, review and merge :-) If you have an interest you could grab the PR. I guess we could get this working in one-two weeks and then switch to ConvertFrom-Json.

@ghost ghost removed the Review - Needed The PR is being reviewed label Jul 2, 2021
@ghost ghost added the Review - Needed The PR is being reviewed label Jul 9, 2021
@ghost
Copy link

ghost commented Jul 9, 2021

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Breaking-Change breaking change that may affect users CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log Committee-Reviewed PS-Committee has reviewed this and made a decision Documentation Needed in this repo Documentation is needed in this repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Class hidden properties still serialized to JSON