Five constructs bash accepts that the grammar rejects #349

Five constructs that bash accepts and the grammar rejects, found in one sweep over a family of shell repositories. They are independent of each other, and I am glad to split them into five issues if that suits the tracker better — one report seemed the smaller imposition.

Each was reduced to the smallest failing case. Every left-hand example below is valid bash: bash -n is quiet and the code runs. All five were seen on tree-sitter-bash v0.25.1, built from source, with tree-sitter 0.25.2, and again on the same grammar inside tree_sitter_language_pack 1.4.1.

Two related rejections are already filed and are not repeated here: ${x:+word} holding an operator character is #267, and a herestring after a redirect is #232.

Why this matters past the line: the recovery can put one ERROR node over the whole file. A tool that reads comments out of the tree then gets a # inside a string as a comment and misses real ones. In one 319-line script here that is exactly what happened.

1. while with a compound condition and no ; before do

while case $x in a) true ;; esac do
  echo hi
done

One ERROR node covers the loop. Bash accepts it, because esac has already ended the compound command:

$ bash -c 'x=a; n=0; while case $x in a) ((n < 2)) ;; esac do n=$((n + 1)); done; echo "$n"'
2

while if ... fi do and while { ...; } do fail the same way. A ; before do parses.

2. A ) inside a bracket expression, in every spelling bash accepts

case "a)" in *[*_\)]) echo hit ;; esac

$ bash -c 'case "a)" in *[*_\)]) echo hit ;; *) echo miss ;; esac'
hit

All six of these are rejected, and bash 5.3 and bash 3.2 accept all six:

case $x in *[*_\)])   ;; esac
case $x in *[*_")"])  ;; esac
case $x in *[*_')'])  ;; esac
[[ $x == *[*_\)]  ]]
[[ $x == *[*_")"] ]]
[[ $x == *[*_')'] ]]

A bare ) is not an alternative: bash ends the case branch at it, and inside [[ ]] answers syntax error in conditional expression: unexpected token ). So there is no written spelling that both bash and the grammar accept, and only a pattern held in a variable gets through:

CLASS='*[*_)]'
case "a)" in $CLASS) echo hit ;; esac

A class with no ) parses, so the bracket expression itself is understood: case $x in *[ab]) ;; esac is fine. So is the regular-expression side, [[ "$x" =~ ^[0-9]+[.\)] ]].

3. A case pattern of a leading * and two double-quoted strings

case "$t" in *"a"*"b"*) echo hit ;; esac

$ bash -c 't=xaybz; case "$t" in *"a"*"b"*) echo hit ;; *) echo miss ;; esac'
hit

I could not reduce this to one rule, so here is the measurement. Every line is valid bash:

*"a"*                 parses
*"a"*b                parses
*"a"*$x               parses
*"a"*?                parses
b*"a"*                parses
"a"*"b"*              parses
*'a'*'b'*             parses
*'a'*"b"*             parses
*$v*"b"*              parses

*"a"b                 ERROR
*"a"*"b"*             ERROR
*"a"*'b'*             ERROR
?*"a"*"b"*            ERROR
*"a b"*"c d"*         ERROR

A leading * followed by a double-quoted string is where it starts, and what comes after decides. Single quotes in that leading position are fine either way.

4. A bare - inside [ ] is read as a unary operator

if [ "$f" = - ]; then echo hit; fi

(program (if_statement condition: (test_command (binary_expression left: (string (simple_expansion (variable_name))) right: (unary_expression (word))) (MISSING "]")) ...))

The - became (unary_expression (word)) and swallowed the ]. Bash accepts it:

$ bash -c 'f=-; [ "$f" = - ] && echo hit'
hit

[ - ] and [ -n - ] fail the same way, so the position does not matter. [ "$f" = -x ] parses, and so does [[ "$f" = - ]]. Quoting the - parses.

This is not #316, which is about a [ merging across a redirected statement. Here one test command stands alone.

5. A heredoc ends at a body line that begins with the delimiter

cat <<'MODE'
MODE=safe
MODE

(program (redirected_statement body: (command ...) redirect: (heredoc_redirect (heredoc_start) (heredoc_body) (heredoc_end))) (MISSING ";") (command ...) (command ...))

The heredoc closed at MODE=safe, and what follows is read as two commands. Bash reads the body whole:

$ bash -c "cat <<'MODE'
MODE=safe
MODE"
MODE=safe

Any line beginning with the delimiter does it, not only an assignment: a body line MODEL closes a <<'MODE' body too. The same for <<MODE unquoted and for <<-MODE. A longer delimiter does not help, because it is a prefix match rather than a length — MODE_BODY=safe closes a <<'MODE_BODY' body. A leading space saves the line, and xMODE is fine.

POSIX ends the body at a line containing only the delimiter, so a prefix match ends it too early.

This is not #346, which is about a heredoc whose start line continues with a ;. Here the start line is ordinary and the body closes itself.