Make Select-String more intelligent for non-primitive types

Summary of the new feature/enhancement

As a user,
I want to be able to easily use Select-String to find string data in formatted output,
so that I can find and process data in PowerShell more quickly.

By default, when you pass non-string data into the Select-String cmdlet, it will find matches based on the object's ToString() method results. This is great when the ToString output actually represents the data you want to match, but very often ToString results do not represent the data that you see in PowerShell, and this can result in incorrect or failed matches.

For example, consider this script:

# First, capture the date:
$date = Get-Date

# Now, get the current month in string format:
$month = $date.ToString("MMMM")

# Now, look at how the date renders in PowerShell, showing the month as a string
$date

# Now, try to match the month using Select-String. This returns nothing.
Get-Date | Select-String -Pattern $month

That script shows the current date, including the month in string format, but if you try to select that string based on the actual string month, there are no matches. Why does this happen? Because ToString() on DateTime objects returns the date time with a numeric month, not the string month.

Now let's look at a more realistic example:

# Get some services, including the Windows Update service, and filter output on the
# string "Update"
Get-Service wuauserv,bits | Select-String Update

That returns nothing. Why? Because the ToString method on service objects returns the name of service, so you can't filter output based on a partial match of a display name string this way.

Here is one more example:

Get-Process -Id $PID | Format-List * | Select-String Memory

This also returns nothing, because of the ToString method on Format cmdlet output objects returns their type name, none of which match Memory.

It is reasonable for a user to expect to be able to easily and consistently parse/filter output that is rendered in the PowerShell console, but this is not possible unless they pipe to Out-String -Stream before they then pipe that streamed result to Select-String.

Proposed technical implementation details (optional)

[EDIT]: Updated to reflect the parameter name we decided to move forward in the discussion below.

I want to make Select-String better by adding a new ~~-ConsoleOutput switch parameter (or some better parameter name: suggestions welcome)~~ -FromFormattedOutput switch parameter that indicates that you want to select a string based on the console output of the data that is piped into Select-String, which would automatically take care of the formatting and output of non-value and non-string types (and non-MatchInfo, but that's a special case internally for Select-String), and select string matches based on that output rather than based on the ToString method output of individual objects.

Additional details

Personally I would prefer if Select-String worked this way by default for non-value and non-string types, but that would be a breaking change at this point, so that's not an option; however, users who want it to work this way by default can use $PSDefaultParameterValues['Select-String:FromFormattedOutput'] = $true, and that will have the same result.

From examples above it is not clear that is desired output. Can we get it with a workaround like Out-String or anything like?

Ilya (@iSazonov)

I both love the idea and agree that it should have been the default behavior.

Ilya (@iSazonov), I think of it as the equivalent of piping to Out-String -Stream first, and then performing the usual line-by-line search.

In fact, in my profile I have the following simple function:

Set-Alias slsd Select-DisplayString
function Select-DisplayString
{
  # Note: Since we want to use $Input for simplicity, we canNOT
  #       make this an advanced function.
  param([string[]] $Pattern)

  if ($args.Count) { Throw "Unexpected arguments specified: $args" }

  if ($Pattern) {
    $input | Out-String -Stream | Select-String -Pattern $Pattern
  }
  else {
    $input | Out-String -Stream
  }

}

Example use, in a directory that has files / directories that are symlinks, such as / on macOS:

PS> Get-ChildItem | Select-DisplayString ^l
l-r--            7/2/2012  3:31 PM                User Guides And Information -> /Library/Documentation/User Guides And Information.localized
l-r--          12/20/2012  5:55 PM                User Information -> /Library/Documentation/User Information.localized

That is, the for-display output of Get-ChildItem was filtered by the first character on each line, which is the Mode column, whose first character being l indicates a symlink.

I wonder if we could do a simple check... Something like:

string objectString = LanguagePrimitives.ConvertTo<string>(InputObject);
if (objectString == InputObject.BaseObject.GetType().Fullname)
{
    // Use out-string on InputObject
}
else
{
    // Use objectString
}

That way we only have the object being resolved this way if there are no other conversion paths available. This would also be a good candidate for an experimental feature if we have concerns about the possibly breaking change.

To get straight to bike-shedding regarding what to name the new switch:

We already have switches that modify output behavior with prefix As, such as -AsHashtable and -AsString for Group-Object.

As for input treatment, the only instance I could find that somewhat fits is Copy-Item's -FromSession

From as the prefix strikes me as reusable, so perhaps -FromDisplay [representation] or -FromDisplayOutput.

In any event, incorporating the word display strikes me as preferable to console.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32): That test isn't sufficient, I'm afraid, because there are cases with different .ToString() behavior that is still distinct from PowerShell's output formatting; e.g., on Windows:

PSonWin> (Get-Process)[0].psobject.BaseObject.ToString()
System.Diagnostics.Process (ApplicationFrameHost)

Edit: Note how the output is more than just the full type name - the process image name is appended.

Michael Klement (@mklement0)

Michael Klement (@mklement0): I like the -From prefix for the parameter name. Maybe we should also borrow the fact that this is related to the Out-String -Stream parameter, and call it -FromInputStream (or maybe -StreamInput, since that's stating what the switch would do?). I like those better than -FromDisplay, because this is more about streaming object input into the command. It could be described as follows:

Indicates that the cmdlet searches for text and text patterns in streamed input. Objects piped into this command will first be rendered in their default format, and that output will be streamed into this command for text matching. By default, objects piped into this command are converted into their string equivalent using their ToString method, and that output is used in this command for text matching.

That's a little wordy, but something along those lines should do.

mklement

mklement LanguagePrimitives.ConvertTo<string>() is not the same as .ToString(); it will follow the same behaviour as a [string] cast in PowerShell.

Besides, I thought that's exactly what we wanted -- if the object does not have a direct string conversion, we go a different method and run it through Out-String first before having Select-String parse it. So that would hit the test correctly -- Select-String would grab the value, see it's the same as the object's type name, and then pass the PSObject through Out-String before attempting to process the value.

EDIT: Oh, I see what you mean, there's an extra piece attached to that ToString() huh... hmm. maybe if the conversion to string .StartsWith() the type name?

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) That's still not sufficient, because you're dependent on how the creator of the object wanted it to render as a single string in that case. For example:

This is different. It's about being able to explicitly indicate you want the object streamed into Select-String with its default output, so that you can select string matches from that.

Kirk Munro (@KirkMunro)

Kirk Munro (@KirkMunro) aye, I think the explicit switch is a good idea. However, I also think that (perhaps as an experimental feature), Select-String could (should?) attempt to detect when its InputObjects are being converted into useless data and attempt to automatically compensate, using the behaviour that can also be forced with a switch.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) Here's another example showing how it is different:

If I want to select "October", I can only do that from the rendered output.

However, I also think that (perhaps as an experimental feature), Select-String could (should?) attempt to detect when its InputObjects are being converted into useless data and attempt to automatically compensate, using the behaviour that can also be forced with a switch.

Maybe. That might just be another source for confusion though since the way an object renders as string is 100% dependent on how the author decided to implement it, and there's really no model for consistency there.

I'd just like to see it work this way by default. Select-String is often associated with grep, but users of grep expect to select information that they would see in a console if they were to output it there, not information that comes from a method and varies from object to object. If you're getting into object methods or properties, there are better commands for that.

That's a good point... Yeah, I can't really see any reason I'd pipe complex objects into Select-String without Out-String at present. 🤔

Note to self: when implementing this, consider #8963 (i.e. What happens if you're selecting text that is formatted and possibly colored/highlighted already -- will the highlighting still work properly and provide a good experience for users)?

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32): Re .ToString(): I should have worded by comment more clearly (since amended), but it sounds like we're all on the same page now.

Kirk Munro (@KirkMunro):

Maybe we should also borrow the fact that this is related to the Out-String -Stream parameter, and call it -FromInputStream (or maybe -StreamInput, since that's stating what the switch would do?)

I think streaming is too general a term, and the connection with Out-String may not be obvious; after all, the term is also applied to how the pipeline functions fundamentally: objects stream through the pipeline.

What's specific to the scenario at hand is that it is text-line-by-text-line streaming derived from the for-display output (as rendered by PowerShell's formatting system).

I think the word display offers an important hint, and if we want to also incorporate the line-by-line aspect, we could use -FromDisplayOutputLines (the use of the plural is nonstandard, but the singular sounds awkward), though that is quite verbose.

My thinking was that mentioning lines wouldn't be necessary, as my sense is that line-by-line search is what users would naturally expect, in line with grep and findstr.

Either way we should also think about a short alias name, such as -dl.

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello) What is the perceived value in adding another cmdlet to do essentially the same job? 😕

The way I see it at the moment is:

We have a Select-String cmdlet,
for which a lot of potential piped input is effectively useless, because
most objects' ToString() values are significantly less useful than their Out-String values.

Adding another cmdlet effectively only duplicates Select-String (and likely quite a lot of code if we're not very careful and do a lot of refactoring of the original cmdlet as well), and would effectively make Select-String a second-class citizen -- we introduce a new cmdlet that does everything Select-String does, and more, and then Select-String becomes effectively useless, since the new cmdlet would already do what Select-String does.

I don't see a particular need for this to be a whole extra cmdlet, and nor do I think adding one additional parameter would be a huge change for this cmdlet. If we're talking adding parameters to Invoke-WebRequest, sure, cause for caution. Here? Not so much, in my opinion.

That said, looking at the current syntax diagram for Select-String, we clearly need to do something about excessive listing of possible values for -Culture, but on the whole it doesn't have that many parameters, and that issue is something we need to fix in the help system, not the cmdlet itself.

Kirk Munro (@KirkMunro)

Some small comments:

Kirk Munro (@KirkMunro)

Now, try to match the month using Select-String. This returns nothing.

Get-Date | Select-String -Pattern $month

This seems rather artificial to me given that PowerShell is an object-based shell. Why are you not doing:

Get-Date | where Month -match $month

And if you really want to grep against formatted output, it's simple:

Get-Date | Out-String -Stream | Select-String -Pattern $month

Michael Klement (@mklement0)

On the input side, there are probably not too many people who rely on the current behavior,

On what basis are you making this claim? It would certainly break my daily workflow. Oh and BTW, Select-String works against files too so that has to be taken into account.

Finally

Going through the formatter is an order of magnitude slower than where or sls (as it is today).
Formatted output has no guarantee of compatibility. Changing how objects are rendered is one of the things that is explicitly called out as being completely open to change. So writing scripts against formatted output is discouraged.

Bruce Payette (@bpayette)

Bruce Payette (@bpayette) There's scripting in scripts, where you want performance and you want to use .NET objects, and there's ad-hoc use where you are pulling data and you want to get chunks of it quickly and easily. That's why grep or findstr are used even by folks on the PS Team or PMs not on the PS team in place of Select-String, and in place of using Out-String -Stream, Where-Object, etc. Being able to get information quickly and easily is highly useful, which is the motivation here. The motivation here, at list for me, isn't to use Select-String to filter output in scripts.

My only argument for updating Select-String vs a new command is to have a single command to do easy filtering with highlighting of what it found in-box so that I don't have to worry about what system I'm on when I use it. Given PowerShell is a shell, this seems like solid value to offer across the board. I can use grep or findstr, but I tend to stick to native, in-box PS that just works everywhere. That's where I think augmenting Select-String vs using Steve Lee (@SteveL-MSFT)'s Select-Text in his upcoming module adds value.

Given the pushback on augmentation of Select-String and the concern around breaking changes, I'll just hope that a successor gets enough traction to replace Select-String with a suitably short alias in-box, and then just switch to that.

Rain Sallow (/u/ta11ow) (@vexx32)

In addition to the points made by Rain Sallow (/u/ta11ow) (@vexx32) and Kirk Munro (@KirkMunro):

And if you really want to grep against formatted output, it's simple:

Select-String's sole purpose is to search through strings (and being able to do that "quick and dirty", as Kirk Munro (@KirkMunro) explains, is an invaluable interactive tool).
Now, if the input isn't composed of strings, searching what string representation of the input makes more sense?
- What you see in the console (host), i.e. the formatted representations?
- Or the result from a .ToString() call which produces a near-useless and hard-to-predict stringification you don't typically get to see elsewhere (which answers the on what basis question).

Piping to | Out-String -Stream isn't simple: it's an obscure, cumbersome workaround for something the cmdlet should have done automatically to begin with.

Oh and BTW, Select-String works against files too so that has to be taken into account.

Yes, the -LiteralPath binding via Get-Item / Get-ChildItem output for file-content searching would have to be retained, which makes for a (preexisting) inconsistency - but an easily explained one.

That is, if you really wanted to search a directory listing as printed to the screen, then - and only then - would you need | Out-String -Stream.

Apparently we have an oss function wrapper for that now - that such a function was created speaks to how often you currently have to resort to that workaround.

It would certainly break my daily workflow.

Assuming the -LiteralPath binding is retained, what would break?

Other than someone inappropriately using Select-String with non-string input in a script (the scenario that's Joey Aiello (@joeyaiello)'s concern) - which is where you should definitely check object properties instead - nothing should break, and much is gained.

To me, that makes it a bucket 3 change, which spares us the confusion of introducing another cmdlet.

Revisiting some of the earlier points:

Primarily, we question the overall usefulness of this given the existing findstr and grep on platforms.

The different syntax and varying capabilities between these utilities , let alone having to use different utilities on different platforms to begin with is reason alone to make Select-String support the intuitively expected behavior of searching the formatted representations.

Going through the formatter is an order of magnitude slower than where or sls (as it is today).

If the intent is to search the formatted representations, that cost is inevitable - and using findstr or grep already incurs it.

Again, the primary use case is quick-and-dirty interactive search.
In programmatic use, Where-Object should be used instead.
Providing such guidance - along with conversely saying: in scripts, use Select-String only with string input - as part of the documentation should be sufficient.

throw it up on the Gallery, and then we can actually play with an implementation

It's not in the Gallery, but I've created a Select-StringFormatted wrapper function (which will be slow compared to a proper implementation as part of SelectString) in this Gist (also part of this SO answer), which has Out-String -Stream built in - this is what Select-String itself should do.

I recommend defining scs as its alias, which is what Select-String's alias should always have been, had its name followed the naming conventions.

Assuming you have looked at the linked code to ensure that it is safe (which I can personally assure you of, but it's always good to check), you can install it as follows:

irm https://gist.github.com/mklement0/46fea9e6e5ef1a3ceaf681c976cb68e3/raw/Select-StringFormatted.ps1 | iex

Sample call (assumes Set-Alias scs Select-StringFormatted):

PS> Get-Process | scs service

     16     3.74      14.09       0.00    5664   0 SecurityHealthService
     10     3.61       6.84       0.00     632   0 services
     12     3.21       7.15       0.00    2388   0 VGAuthService

While this is already very convenient, there's room for improvement, given that simply string-filtering the lines doesn't include the header line with the column names; for (implicitly to be) Format-Tabled input, the header line could be included for display, analogous to the existing for-display enhancement of coloring the matching part of the line.

I think that having Select-String search against the formatted output of objects would be a great idea. As it is, I only use it to search files or plain text as it otherwise acts odd. Same thing with Set-Clipboard. If a cmdlet exists to do things with strings, then it should handle it in a way that makes the most sense. If I manually format something, I don't want Select-String to be searching the FormatEntryData string that it gets from ToString. I would expect it to search the formatted output.

FireInWinter

Hear, hear, FireInWinter, but note that - as with Set-Clipboard - it is the - often useless - .ToString() return value applied directly to the input object that is used:

PS> @{ foo=1 } | sls hash   # @{ foo=1 }.ToString() -> 'System.Collections.Hashtable'

System.Collections.Hashtable

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

Make Select-String more intelligent for non-primitive types #10726

Summary of the new feature/enhancement

Proposed technical implementation details (optional)

Additional details

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make Select-String more intelligent for non-primitive types #10726

Description

Summary of the new feature/enhancement

Proposed technical implementation details (optional)

Additional details

Activity

iSazonov commented on Oct 8, 2019

mklement0 commented on Oct 8, 2019

vexx32 commented on Oct 8, 2019

mklement0 commented on Oct 8, 2019

mklement0 commented on Oct 8, 2019

KirkMunro commented on Oct 8, 2019

vexx32 commented on Oct 8, 2019

KirkMunro commented on Oct 8, 2019

vexx32 commented on Oct 8, 2019

KirkMunro commented on Oct 8, 2019

KirkMunro commented on Oct 8, 2019

vexx32 commented on Oct 8, 2019

KirkMunro commented on Oct 9, 2019

mklement0 commented on Oct 9, 2019

16 remaining items

vexx32 commented on Dec 19, 2019

bpayette commented on Dec 19, 2019

KirkMunro commented on Dec 19, 2019

mklement0 commented on Dec 19, 2019

mklement0 commented on Nov 1, 2020

FireInWinter commented on Nov 10, 2021

mklement0 commented on Nov 10, 2021

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions