Do not require { ... } around variable names for null-conditional member access

Rob Holt (@rjmholt)

Follow-up from #3240.

Null-conditional member access (?.) was implemented and will be available as an experimental feature in 7.0

However, in the interest of backward compatibility the implementation currently requires that the variable name be enclosed in {...}, because ? is technically a legal character in a variable name whose use surprisingly doesn't require {...}.

That is, if you want to ignore an attempt to call$obj.Method() if $obj is $null (or undefined), you would expect the following, analogous to C#:

# Does NOT work as intended.
# 'obj?' as a whole is interpreted as the variable name.
$obj?.Method()

Instead, you must currently use:

# !! Must enclose 'obj' in {...} to signal that '?' is a syntactic element as part of '?.'
${obj}?.Method()

Update: Null-coalescing - $var??='default and $a??$b - and the ternary operator - $var?1:0 - are similarly affected (though there the use of whitespace before the ? resolves the issue).

This requirement is (a) unexpected and (b) cumbersome:

New users will not expect the need for {...} and won't necessarily know that it is what is required when the following fails (possibly undetected, unless Set-StrictMode -Version 2 or higher is in effect): $o = [pscustomobject] @{ one = 1 }; $o?.one
Even once users do know about the need for {...}:
- They will forget to use it on occasion, because of the counter-intuitive need for it.
- When they do remember, this seemingly artificial requirement will be an ongoing source of frustration, especially since {...} is hard to type.

Use of {...} shouldn't be necessary, and while not requiring it amounts to a breaking change, it arguably falls into bucket 3: Unlikely Grey Area.

Why this change should be considered acceptable:

Note: Rob Holt (@rjmholt) discusses why changing PowerShell's syntax is highly problematic in general in this comment, but for the reasons presented below I think it is worth making an exception here.

?. is a new syntactic feature; by definition scripts that use it cannot (meaningfully) run on older versions - unless you specifically add conditionals that provide legacy-version-compatible code paths, but that seems hardly worth it - you would then just stick with the legacy features.

Yes, the interpretation of $var?.foo in old scripts that used $var? as a variable name would break, but:

? should never have been allowed as part of an identifier without enclosing it in {...}.
Since being able to do so is unexpected and probably even unknown to many, such variable names are exceedingly rare in the wild, speaking from personal experience, but @mburszley's analysis provides more tangible evidence.
- Even Ruby only allows ? (and !) at the end of identifiers, and there only of method identifiers, and I suspect that most Ruby users are aware that they should not assume that other languages support the same thing.

So, pragmatically speaking:

The vast majority of existing code will not be affected - a token such as $var?.foo will simply not be encountered.
If you write $var?.foo with the new semantics, then yes, running that on older versions could result in different behavior (rather than breaking in an obvious manner), depending on what strict mode is in effect - but you should always enforce the minimum version required to run your code as intended anyway (#requires -Version, module-manifest keys).

All in all, to me this a clear case of a bucket 3 change: a technically breaking change that breaks very little existing code while offering real benefits (or, conversely, avoiding perennial headaches due to unexpected behavior).

To clarify, the milestone just means it's something the team should discuss based on the feedback, not that we are committed to making this change.

Michael Klement (@mklement0)

Michael Klement (@mklement0)

? should never have been allowed as part of an identifier without enclosing it in {...}.

Ummm - the initial base language for PowerShell was the Posix Shell which uses $? for the exit code q.e.d '?' is allowed in variable names unquoted.

it cannot (meaningfully) run on older versions

Sure it can:

PS>  $abc?=@{a=1; b=2; c=3}
PS>  $abc?.b                                                                                        1
PS>  2

Sure it can:

It cannot run meaningfully with the new semantics (null-conditional access) - that's all I meant to say.

Yes, your example currently works, but with different semantics (abc? including the ? being considered the variable name) - but for the reasons stated this corner case is worth abandoning for the sake of making ?. work as expected.

The initial base language for PowerShell was the Posix Shell which uses $? for the exit code

$? is an exception in POSIX-like shells; you can not do the following:

# bash, ksh, zsh, dash (all common /bin/sh implementations)
foo?='bar' # BOOM! E.g. "bash: foo?=bar: command not found"

(And, obviously, no one's asking that $? be abolished in PowerShell.)

And, as a matter of course, it should be mentioned, that all other automatic variables in the same space as $? (i.e., $^ and $$) are explicitly special-cased in the tokenizer. And, as a matter of fact, so is $?, even though it (currently) doesn't need to be.

It seems clear to me that it was expected at the time that code was written that ? would not be a valid identifier character.

James Brundage (@StartAutomating)

I only know of one person who has used ? at the end of variable names in their scripts, and that's James Brundage (@StartAutomating). Since he's actually used that syntax, I thought it worth drawing his attention to this discussion.

Also Rain Sallow (/u/ta11ow) (@vexx32), the reason that $? is special cased in the tokenizer is because you cannot create a variable with a name that starts with ?. e.g. $??? = 'foo' will not parse. Variable names must start with alphanumeric characters or an underscore, and currently can contain alphanumeric characters, underscores, or, questionmarks. I'm in agreement that ?. should be parsed as the null-conditional member access operator, but I wanted to clarify why ? is special-cased in the tokenizer.

Thanks for drawing my attention to this chat.

For what it's worth, I have a few thoughts:

While these variables are rare, no one want's to play whack-a-mole when back-compat breaks their scripts
Deeply geeky folks aside, I'd expect that ${obj}?.Method() or $obj?.Method() will get little uptick. There's a decade of guidance out on the web of how to manage nulls out there, and I don't see that many people switching from if ($obj) { $obj.Method() } just to get on the v7 bandwagon
Additionally, most people who have problems with unexpected variable names learn the ${} syntax (you'd already hit it naturally with an underscore). I'd expect the venn diagram overlap of "people who want to use this feature" to "people who know ${} syntax already" to be near 100%
Since said syntax isn't as powerful as a full subexpression, and full subexpressions work both in core and not, I'd expect that most people will continue to use this going forward.

So, running the math, there are several concrete reasons not to do this, and only one vague reason to do this (it might be easier, for some users). Given the low prevalence of ? in variable names, you might be helping more people than you harm, but in my opinion it's a coin-toss.

Given such coin-toss potential of benefit, I will paraphrase Robert McNamara "If we're darned if we do and darned if we don't, I'll pick darned if we don't".

I suggest doing it so that $obj?.Method() works as $obj?.Method() by default, and only fall back to $obj?.Method() when $obj doesn’t exist in scope, but $obj? does and #requires ‑Version isn’t v7+.

ExE Boss (@ExE-Boss)

ExE Boss (@ExE-Boss), that seems like a lot of exceptions to make for this particular rule.

I repeat my questioning of the value of this particular rule.

Past that, the question is "what else do we not know about variable syntax that we might break?"

James Brundage (@StartAutomating)

If the consensus is to force ${...} syntax for this, support for OptionalFeatures would be appreciated here, so that folks like myself who will use this syntax (I'll be using it a lot) can opt out of the noise.

Since we're paraphrasing here:
"Logic clearly dictates that the needs of the many outweigh the needs of the few." - Spock

So, running the math, there are several concrete reasons not to do this, and only one vague reason to do this (it might be easier, for some users).

James Brundage (@StartAutomating): There are multiple reasons, and they are concrete, not vague. Also, one of the goals of this is code simplicity. Having to wrap variable names with {} works directly against that goal.

Michael Klement (@mklement0)

I'm with Michael Klement (@mklement0) and Kirk Munro (@KirkMunro) on this. Personally I find the ${...}? syntax unnatural and unnecessary complex. We have had breaking changes already and in this case it is small price to pay in exchange for clarity.

James Brundage (@StartAutomating)

James Brundage (@StartAutomating)

Additionally, most people who have problems with unexpected variable names learn the ${} syntax (you'd already hit it naturally with an underscore). I'd expect the venn diagram overlap of "people who want to use this feature" to "people who know ${} syntax already" to be near 100%

This doesn't hold up. Underscores are valid characters in variable names.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32)

It seems clear to me that it was expected at the time that code was written

Nope.

Michael Klement (@mklement0)

Michael Klement (@mklement0) I rounded up so if you want to feel just a bit better its actually 0.00058% 😄

In PowerShell/PowerShell-RFC#223 (comment) Aditya Patwardhan (@adityapatwardhan) did an analysis using the same public corpus I used and came up with 62% with variables ending with a question mark. I didn't see a response on how he came up with that percentage. Did the PS committee use that analysis to determine this request?

Aditya Patwardhan (@adityapatwardhan)

Looking at the linked comment, I think what Aditya Patwardhan (@adityapatwardhan) is saying that among all variables that have ? in their name 62% have a ? as the name's last character.

By contrast, your analysis of the actual prevalence of (curly-brace-less) ending-in-? variables (as a percentage of variable usage overall) seems much more relevant.

Thomas Nieto (@ThomasNieto)

At this point, it sounds like the committee is just saying "we don't want to" and if that's the case, why even have this feature if it's going to be clunky and un-used as a result?

Thomas Nieto (@ThomasNieto) likewise I did an analysis somewhere around these issues and came up to similar results to yours.

@mburszley Thanks for doing your analysis. I wanted to use another method (ast vs regex) to confirm the results you got last year.

I hear you, @mburszley, but re:

the committee is just saying "we don't want to"

If this truly was a fiat decision - one not grounded in analysis of real-world usage - it wouldn't bode well for the community at large, especially given that the stated reason was a grounding in such analysis.

Thus, we should give the committee a chance to show the real-world usage analysis the decision was based on - or to recognize that the analysis fell short and reconsider the decision.

What about my suggestion in #11379 (comment)?

I suggest doing it so that $obj?.Method() works as $obj?.Method() by default, and only fall back to $obj?.Method() when $obj doesn’t exist in scope, but $obj? does and #requires ‑Version isn’t v7+.

I would expect that it would address most of the backwards compatibility concerns.

Thomas Nieto (@ThomasNieto)

Thomas Nieto (@ThomasNieto)

Michael Klement (@mklement0) is correct, I did mean that among the variables using ?, 62% use it at the end.

From the results shown in comment here seems all of the variables which use ? use it at the end. Hence the proposal to make a non-breaking change.

If the requirement for {..} is removed then the script will still parse but the semantics will be different, which in my opinion is very dangerous.

Sure... but analysis shows literally almost nobody is using them.

Same as any other breaking change, it can be announced, introduced as an experimental feature for a time, and then documented and brought into stable.

I see no reason to recommend an esoteric syntax when ? as an identifier character is not actually being used in a remotely significant number of cases.

If the requirement for {..} is removed then the script will still parse but the semantics will be different, which in my opinion is very dangerous.

Aren't you also worried about folks using ?. the way they do in C/C++/C# and getting the wrong result with no indication they did anything wrong? For example:

PS> $res = "My length is 15"
PS> $res?.Length
0

Sure, strict-mode will find this but I don't know that a lot of folks use strict mode. So, to prevent a very small percentage of scripts from breaking, it seems we're setting up a lot of folks for failure using ?.. I'm not crazy about the trade-off. That said, I definitely understand the desire to avoid breaking folks even if it's a very small number of folks.

Maybe PowerShell needs an equivalent of Enable-ExperimentalFeature call it Enable-PSFeature, for features that are no longer in the experimental state but are not ready to be turned on by default. For scripts that execute with code like this $?.GetType() you could start emitting warnings that in the future this will break and that you should use ${?}.GetType() instead. Individual scripts could enable the feature with a #requires -version 7.1 -feature PSNullConditionalOperators. Then maybe whenever PS 8 hits, the feature could be enabled by default.

Keith Hill (@rkeithhill)

Keith Hill (@rkeithhill) 's example goes to straight to the point.
Forget for a moment what you know about how PowerShell works today. When you see

PS> $res?.Length

It looks like name, operator, name - which of course it is.
Which would make more sense: "?" is part of the operator , or "?" is part of the first name-element ?
If these operators are needed (and I think they are "not PowerShelly") its a choice between something which will be used incorrectly and a small amount of breakage.

I would start putting rules into psScriptAnalyzer now to say it's bad style to use certain [legal] characters in variable names.

I don't like Enable-feature. Someone puts it in script to make something they want to use work, and an old script breaks if run after the new script, (but works the rest of the time). I've seen third party script X fails after running script Y because Y turns on strict mode and X relies on it being off; the authors of both scripts are in my bad books.

I would start putting rules into psScriptAnalyzer now to say it's bad style to use certain [legal] characters in variable names.

Here's the problem: if you change the way PowerShell tokenises variables, PSScriptAnalyzer will see them differently too. If you have a variable like $res?, then looking at something like this:

{
$res?.Length
}.Ast.EndBlock.Statements[0].PipelineElements[0].Expression.Expression.VariablePath.UserPath

gives you

res?

BUT, if you change the way PowerShell is parsed in, say, 7.2, then you will get

res

Now PSScriptAnalyzer can't see the ?, because it's no longer part of the variable name (by definition, since it's the change we're trying to test for). This is because PSScriptAnalyzer is a PowerShell module and reuses the same parser that ships in PowerShell. (And it would be totally unmaintainable and a recipe for disaster if it tried to reimplement it).

There are ways around this, like conditionally compiling PSScriptAnalyzer (and shipping a whole new assembly) to test the new 7.2 logic to supply the same diagnostic. But then you've increased complexity everywhere (or alternatively decreased your test surface) to service a somewhat cosmetic scenario.

Worse, all the other tools out there that also rely on the AST won't have the capacity to sustain this kind of complexity even if PSScriptAnalyzer does. Most people building AST-targeting tools out there aren't going to compile against individual minor versions of PowerShell for perfect correctness. They probably won't even know about such a small breaking change, especially since it's not an error in any version; it's the most insidious form of breaking change, where one behaviour quietly becomes another.

That's basically what makes parser/tokeniser-level changes like this one more dangerous. Even PSScriptAnalyzer must bend over backwards to service them correctly, meaning they must be weighed against such onerous changes.

Rob Holt (@rjmholt)

Rob Holt (@rjmholt), while I think we can all appreciate how tricky a change like this is in principle, in this particular case it doesn't matter in practice, and I don't think a PSSA rule is even needed:

@ThomasNieto's analysis has shown that - if we believe the corpus to be representative of all PowerShell code out there - that virtually no one uses variables with names that end in ?.
If I were to guess, most people wouldn't even think to use such variable names, because they wouldn't expect them to work (me included), and probably don't even notice the exception that the automatic $? variable constitutes (whose name is equally an exception in POSIX-compatible shells such as bash).
- In fact, looking closer at Thomas Nieto (@ThomasNieto)'s analysis reveals that, among the 8 files listed:
  - 5 contain only false positives, namely variables uses inside strings such as "Are you sure you want to kill $vParam?" - in fact, these uses are broken, and reveal that the script authors did not expect ? to be part of the variable name.
  - 2 of those files are no longer publicly available.
  - Only 1 of them is a bona fide use of ?-ending variables - as Boolean variables (e.g., $key1?), which, as an aside, are therefore not combined with the member-access operator, . /cc Steven Ayers (@stevenayers).

Based on the above, this strikes me as a textbook Bucket 3: Unlikely Grey Area change, which is therefore acceptable - and I think the benefits of this change have been argued persuasively.

Documenting the change in behavior (with a rationale) should suffice.

Of course:

this is predicated on (a) the analysis being correct and (b) the publicly available corpus being representative.
it is starkly at odds with the committee's claim that "there was quite a bit of usage of variable names ending with the question mark" (which, of course, would only be a problem if such variable names aren't enclosed in {...}).

Proceeding on the assumption that 1. is true (we haven't heard anything to support 2.), we have two options:

Rip off the band-aid and simply disallow ? in variable names going forward, unless enclosed in {...} (with the obvious exception of $?) - this will break the vanishingly small proportion of scripts that currently rely on that.
- That is, something like $key1? that is neither followed by . nor another ? ($key1??'else') nor a ternary expression ($key1?1:0) would cause a parser error.
  - As the null-coalescing and ternary operator examples show, they too would benefit from this change - currently, you either need a space - $key1 ?1:0 / $key1 ??'else' - or {...} - ${key1}?1:0 / ${key1}??'else'
- Inside expandable strings, ? would then no longer be considered part of a (non-{...}-enclosed) variable name, so we would actually fix existing scripts that have mistakenly used strings such as "Are you sure you want to kill $vParam?". 😁
If we really feel we need to avoid breaking the vanishingly small proportion of existing scripts, we could consider the logic proposed by @ExE-Boss, but I don't think we want to introduce this conceptual complexity - let alone implementation complexity (which I can't really speak to).

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT)

Thanks for discussing this issue in the September community call (https://aka.ms/PSCommunityCall; as of this writing, the September call's recording has not been posted there, but is accessible at https://aka.ms/JoinPSCall, until the next call), starting at 46:00, based on Thomas Nieto (@ThomasNieto)'s suggestion in the prior solicitation for discussion topics at PowerShell/PowerShell-RFC#260:

(I was hoping to provide a direct link to the relevant portion of the recording, but that will have to wait until it is posted to YouTube and linked to at https://aka.ms/PSCommunityCall.)

Based on your remarks in the recording:

Given that you indicated willingness to revisit this, please reopen this issue and re-tag it as Review - Committee.
As for your remarks that enclosing variable names in curly braces is a preexisting feature and that people object to having to use them in this case on aesthetic grounds:
- I don't think anyone here isn't aware that {...} can and sometimes must be used to disambiguate variable names.
- It is also not about aesthetics, it is primarily about not getting the expected behavior in manner that fails quietly, due to not expecting to have to use {...} in this case; a secondary concern is typing inconvenience.

GitHub
PowerShell/PowerShell-RFC
RFC (Request for Comments) documents for community feedback on design changes and improvements to PowerShell ecosystem - PowerShell/PowerShell-RFC

Microsoft Teams
Join conversation

I'm concerned that this issue - now seemingly indefinitely in a closed state - will fall through the cracks, so I've opened #14025 as a reminder to revisit it, and I invite everyone who's interested to show their support there.

Navigation Menu

Repository navigation

Do not require { ... } around variable names for null-conditional member access #11379

Why this change should be considered acceptable:

35 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Do not require { ... } around variable names for null-conditional member access #11379

Description

Why this change should be considered acceptable:

Activity

SteveL-MSFT commented on Dec 17, 2019

bpayette commented on Dec 17, 2019

mklement0 commented on Dec 17, 2019

vexx32 commented on Dec 17, 2019

KirkMunro commented on Dec 17, 2019

StartAutomating commented on Dec 17, 2019

ExE-Boss commented on Dec 17, 2019

StartAutomating commented on Dec 17, 2019

KirkMunro commented on Dec 17, 2019

kborowinski commented on Dec 17, 2019

morsley commented on Dec 18, 2019

BrucePay commented on Dec 18, 2019

35 remaining items

ThomasNieto commented on Aug 27, 2020

mklement0 commented on Aug 27, 2020

morsley commented on Aug 27, 2020

ThomasNieto commented on Aug 27, 2020

mklement0 commented on Aug 27, 2020

ExE-Boss commented on Aug 28, 2020

adityapatwardhan commented on Sep 1, 2020

vexx32 commented on Sep 1, 2020

rkeithhill commented on Sep 7, 2020

jhoneill commented on Sep 8, 2020

rjmholt commented on Sep 8, 2020

mklement0 commented on Sep 9, 2020

mklement0 commented on Sep 24, 2020

mklement0 commented on Nov 12, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions