Indices of array variables are sometimes considered unset (or just display an error).

[Great Big Dot]

uname output: Linux ArchBox0 4.18.16-arch1-1-ARCH #1 SMP PREEMPT Sat Oct 20 22:06:45 UTC 2018 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu
Bash Version: 4.4
Patch Level: 23
Release Status: release
--text follows this line--
Description:
The parameter expansion "${!var[@]}" expands to the indices of an array
(whether linear or associative). The expansion "${var-string}"
returns "${var}" iff var is set and 'string' otherwise. These two
features do not play well together:

$ declare -a -- array=([0]=hello [1]=world)
$ printf -- '%s\n\n' "${!array[@]-Warning: unset}"
bash: hello world: bad substitution

$ declare -a -- array=([0]='helloworld')
$ printf -- '%s\n\n' "${!array[@]-Warning: unset}"
Warning: unset

$ declare -a -- array=([0]='hello world')
$ printf -- '%s\n\n' "${!array[@]-Warning: unset}"
bash: hello world: bad substitution

$ declare -a -- array=()
$ printf -- '%s\n\n' "${!array[@]-Warning: unset}"
Warning: unset

As you can see, accessing the index list of multiple-element arrays
fails when you append the unset expansion. With single-element
arrays, it fails iff the element in question contains any special
characters or whitespace, and thinks the array is unset otherwise.
(Further testing shows that a value of the empty string also throws
an error.) Finally, empty arrays are also considered unset. (This is
the one thing that is consistent with the rest of bash, since empty
arrays themselves are also considered unset by this expansion; that
is, "${array[@]-unset}" yields 'unset' when array isn't set.)

This pattern of behavior is apparently unaffected by changes to IFS,
using a normal variable as a one-element array, using an unset
variable as a zero-element array, or using an associative instead of
linear array. That last one has an interesting wrinkle, however:

$ declare -A -- assoc=(['k e y']='element')
$ printf -- '%s\n\n' "${!assoc[@]-Warning: unset}"
Warning: unset

$ declare -A -- assoc=(['key']='e l e m e n t')
$ printf -- '%s\n\n' "${!assoc[@]-Warning: unset}"
bash: e l e m e n t: bad substitution

Strangely, whether a single-element array errors (as opposed to
giving the wrong result) is only dependent on the the characters in
the *element*, not the *key*---despite the fact that only the key's
value is being requested!


Repeat-By:
$ declare -a arr_2_=(zero one); printf '%s\n' "${!arr_2_[@]-unset}"
bash: zero one: bad substitution
$ declare -a arr_1a=('z e r o'); printf '%s\n' "${!arr_1a[@]-unset}"
bash: z e r o: bad substitution
$ declare -a arr_1b=('zero'); printf '%s\n; "${!arr_1b[@]-unset}"
unset

Fix:
To avoid this problem, you just need to spend another line or two
writing out the relevant conditional explicitly; for example:

# <command> ... "${!array[@]-<default>}"
if [ -v 'array[@]' ]; then
<command> ... "${!array[@]}" ...
else
<command> ... <default> ...
fi

Note that `test -v 'array[@]'` has the same "feature" that
"${array[@]-default}" does: it treats empty arrays as unset.

[Great Big Dot]

(whether linear or associative).

Hold up... when I view this email on the public archives, all of my
"${array[@]}"'s (that is, "${array[<at sign>]}"'a) got turned to
"***@hidden"'s. Was I supposed to use some escape sequence or
something? Is everyone who's subscribed to the mailing list able to see the
actual text? Or should I resend this bug report with all \@-signs escaped
somehow?

Testing...
***@example.com
[e-mail]
testing﹫example.com
testing\@example.com
testing @ example.com

[Greg Wooledge]

Post by Great Big Dot
Hold up... when I view this email on the public archives, all of my

Sadly, there's nothing we can do about that. The maintainers of the
list archive would have to make their anti-spam measure a bit smarter
than it currently is.

[Great Big Dot]

[... A]ccessing the index list of multiple-element arrays
fails when you append the unset expansion. With single-element
arrays, it fails iff the element in question contains any special
characters or whitespace, and thinks the array is unset otherwise.
(Further testing shows that a value of the empty string also throws
an error.) Finally, empty arrays are also considered unset[...]

Oops, just realized what's causing this. I guess it isn't necessarily a
bug? Debatable, I guess.

What's actually happening here is that the *indirection* expansion
"${!foo}", and not the *indices* expansion "${!foo[@]}", is what is being
preformed on something like "${!array[@]-}". Both expansions, while
unrelated, happen to use the same syntax, with the exception that
indirections apply to normal variables and index expansions apply to array
variables. For some reason, adding on the "${foo-default}" expansion causes
the former to be used instead of the latter. This can be seen here:

$ array=(foo)
$ printf -- '%s\n' "${!foo[@]-unset}"
unset
$ foo='hello world'
$ printf -- '%s\n' "${!foo[@]-unset}"
hello world

So first the array is expanded, and then it's treated as a redirection, and
then the unset part kicks in if the array's value isn't an extant variable
name. This explains all the observations I made.

I still think it makes more sense if the "!" in "${!array[@]}" triggered
index expansion instead. At the very least, surely it should be one of
those expansion combinations that just isn't allowed, like
"${#foo[@]-default}" (actually, why is that disallowed?). Anyways, I don't
really see the point of the current behavior.

This pattern of behavior is apparently unaffected by changes to IFS[...]

Upon further examination, and in light of the above realization, this
actually isn't true. In particular, iff the first character of IFS is
alphanumeric or an underscore (or if IFS is the empty string), and if you
use the "${array[*]}" form instead, then the expansion doesn't throw an
error when the array contains more than one element. E.g.:

$ array=(foo bar)
$ printf -- '%s\n' "${!array[*]-Warning: unset}"
bash: foo bar: bad substitution
$ IFS='_'
$ printf -- '%s\n' "${!array[*]-Warning: unset}"
Warning: unset
$ foo_bar='Beto2018'
$ printf -- '%s\n' "${!array[*]-Warning: unset}"
Beto2018
$ IFS=''
$ printf -- '%s\n' "${!array[*]-Warning: unset}"
Warning: unset
$ foobar='Hello, world'
$ printf -- '%s\n' "${!array[*]-Warning: unset}"
Hello, world

Though I understand it now, the above behavior doesn't seem especially
motivated to me. I mean, the variables that end up getting expanded don't
actually have their names stored anywhere, yet the indirection points to
them.

Is there a good reason for treating "${!array[@]-}" and "${!array[*]-}"
like indirections instead of index expansions (or just throwing an error)?

[Eduardo Bustamante]

On Mon, Nov 5, 2018 at 6:01 PM Great Big Dot <***@gmail.com> wrote:
(...)

Post by Great Big Dot

[... A]ccessing the index list of multiple-element arrays
fails when you append the unset expansion. With single-element
arrays, it fails iff the element in question contains any special
characters or whitespace, and thinks the array is unset otherwise.
(Further testing shows that a value of the empty string also throws
an error.) Finally, empty arrays are also considered unset[...]

Oops, just realized what's causing this. I guess it isn't necessarily a
bug? Debatable, I guess.
What's actually happening here is that the *indirection* expansion
unrelated, happen to use the same syntax, with the exception that
indirections apply to normal variables and index expansions apply to array
variables. For some reason, adding on the "${foo-default}" expansion causes

Sorry, I'm having a hard time following this email thread.

What is your ultimate goal or the actual problem you're trying to solve?

(BTW, I would recommend against trying to do three expansions in one.
It might be more terse, but it's hard to read and as you found out,
leads to weird behavior)

[Grisha Levit]

Post by Eduardo Bustamante
Sorry, I'm having a hard time following this email thread.

I *think* the point is that OP expected that:

(a) ${!var[@]-foo} expands to the indexes of var if ${var[@]} if set, else
to `foo'

whereas the behavior they observed is:

(b) ${!var[@]-foo} expands to the value of the variable whose name is
stored
in ${var[@]} or to `foo' if that variable is unset

Their expectation seems reasonable since "the variable whose name is stored
in ${var[@]}" is kind of a weird thing.

[Great Big Dot]

Crap, looks like I accidentally just replied to a single person instead of
the whole list. Here it is (and sorry if I'm uselessly cluttering up
everyone's inboxes!):

---------- Forwarded message ---------
From: Great Big Dot <***@gmail.com>
Date: Tue, Nov 6, 2018 at 5:45 PM
Subject: Re: Indices of array variables are sometimes considered unset (or
just display an error).

Post by Eduardo Bustamante
Sorry, I'm having a hard time following this email thread.

My bad, sorry if I wasn't clear enough. I expected this:

$ array=(foo)
$ printf '%s\n' "${!array[@]-unset}"
0

That is, I expected the keys of 'array' to be printed, unless the array was
unset. Instead, I either got errors or the "unset" string, and never the
actual keys. I didn't even realize that the indirection had something to do
with it, though once it hit me it made the behavior much clearer. I
implicitly assumed that the indirection stuff only applied to non-array
variables, and index-expansion only to array variables.

Post by Eduardo Bustamante
What is your ultimate goal or the actual problem you're trying to solve?

Honestly, I was just messing around at the command line to get the hang of
some of the expansions. IIRC, I was looking at the "parameter
transformation" stuff ("${***@Q}", "${***@A}", etc.). I was looking for any
edge cases that might trip me up in the future, and eventually I wondered
what would happen if you used an array. Somehow that led to me trying other
expansions with arrays, and I ended up getting errors I couldn't explain.

Post by Eduardo Bustamante
(BTW, I would recommend against trying to do three expansions in one.
It might be more terse, but it's hard to read and as you found out,
leads to weird behavior)

Yeah, I think I'm starting to get that now...

[Chet Ramey]

Post by Great Big Dot
What's actually happening here is that the *indirection* expansion
unrelated, happen to use the same syntax, with the exception that
indirections apply to normal variables and index expansions apply to array
variables. For some reason, adding on the "${foo-default}" expansion causes
the former to be used instead of the latter.

The `some reason' is that the current behavior is documented and has been
the way bash has worked since I added indirect expansion (bash-2.0) and the
array keys expansion (bash-3.0).
I'm going to use bash-4.4 for my explanation.

Post by Great Big Dot
$ array=(foo)
unset

Of course. There is no variable "foo"; the indirect expansion results in a
null string.

Post by Great Big Dot
$ foo='hello world'
hello world

(I don't get that result with any version of bash. I went back to bash-3.0
before I quit trying.)

This is a case where bash is trying to be helpful, maybe more so than is
desired. `foo' isn't an array variable, but the !foo[@] is first identified
as a candidate for indirect expansion, then checked to see whether or not
it is the entire expansion between ${ and }. Since it's not, it's not a
candidate for array keys expansion (this is as documented).

Since the array keys expansion isn't valid, the attempt to perform variable
indirection holds. This is where the helpful part comes in. Bash variables
can be referenced as arrays, even if they are not, using `0', `@', or `*'
as subscripts. That means that foo[@] gets expanded into "hello world",
which the shell tries to use as a variable name, resulting in:

$ ../bash-4.4-patched/bash ./x16
./x16: line 2: hello world: bad substitution

The error message in bash-5.0 is a little better:

$ ../bash-5.0-beta/bash ./x16
./x16: line 2: hello world: invalid variable name

Post by Great Big Dot

This pattern of behavior is apparently unaffected by changes to IFS[...]

Upon further examination, and in light of the above realization, this
actually isn't true. In particular, iff the first character of IFS is
alphanumeric or an underscore (or if IFS is the empty string), and if you
use the "${array[*]}" form instead, then the expansion doesn't throw an
error when the array contains more than one element.

Sure, since a double-quoted expansion using `*' separates words using the
first character of IFS.

Post by Great Big Dot
$ foo_bar='Beto2018'
$ printf -- '%s\n' "${!array[*]-Warning: unset}"
Beto2018

Nice. Let's hope he pulls it off today.

Post by Great Big Dot
like indirections instead of index expansions (or just throwing an error)?

When I added array variables and the array keys expansion, I used the ksh93
syntax (${!var[sub]}) and tried to avoid conflict with the existing
indirect expansion as much as possible (that was back when we still thought
there was a chance that POSIX would standardize arrays and it was useful to
have consistent implementations). The ksh93 expansion syntax made it
invalid to use the array keys expansion as part of the ${param:-word}
expansion, so that's how I made it work. Since it wasn't a candidate for
that family of expansions, the existing variable indirection syntax
controls.

[Chet Ramey]

Post by Great Big Dot
(whether linear or associative). The expansion "${var-string}"
returns "${var}" iff var is set and 'string' otherwise. These two

You seem to have neglected a significant section of the documentation:

"If the first character of parameter is an exclamation point (!), and
parameter is not a nameref, it introduces a level of indirection. Bash
uses the value formed by expanding the rest of parameter as the new
parameter; this is then expanded and that value is used in the rest of
the expansion, rather than the expansion of the original parameter.
This is known as indirect expansion. The value is subject to tilde
expansion, parameter expansion, command substitution, and arithmetic
expansion. If parameter is a nameref, this expands to the name of the
parameter referenced by parameter instead of performing the complete
indirect expansion. The exceptions to this are the expansions of
${!prefix*} and ${!name[@]} described below. The exclamation point
must immediately follow the left brace in order to introduce indirec-
tion."

Post by Great Big Dot
$ declare -a -- array=([0]='helloworld')
Warning: unset

This happens to "work" because there is a single array element set, and it
expands to a single word.

[Great Big Dot]

Post by Chet Ramey

Post by Great Big Dot
What's actually happening here is that the *indirection* expansion
unrelated, happen to use the same syntax, with the exception that
indirections apply to normal variables and index expansions apply to array
variables. For some reason, adding on the "${foo-default}" expansion causes
the former to be used instead of the latter.

The `some reason' is that the current behavior is documented and has been
the way bash has worked since I added indirect expansion (bash-2.0) and the
array keys expansion (bash-3.0).

Fair enough. In retrospect, the dual usage of "!" should have been the
first thing I should have blamed when it didn't behave as I expected.

Post by Chet Ramey

Post by Great Big Dot
$ foo='hello world'
hello world

(I don't get that result with any version of bash. I went back to bash-3.0
before I quit trying.)

Oops, must have made a copy-paste error. Sorry for wasting your time
with that! What I meant was something like:

$ foo='hello world'
$ array=(foo)
$ echo "${!array[@]-unset}"
hello world

Post by Chet Ramey

Post by Great Big Dot
Upon further examination, and in light of the above realization, this
actually isn't true. In particular, iff the first character of IFS is
alphanumeric or an underscore (or if IFS is the empty string), and if you
use the "${array[*]}" form instead, then the expansion doesn't throw an
error when the array contains more than one element.

Sure, since a double-quoted expansion using `*' separates words using the first character of IFS.

Yeah, that was what led me to make that observation. Before it hit me
that bash was interpreting it as an indirection, on a hunch I messed
around a bit with IFS and nothing seemed to change, and said so. After
I realized what was happening, I just wanted to correct my false claim
that IFS was irrelevant.

Post by Chet Ramey

Post by Great Big Dot
like indirections instead of index expansions (or just throwing an error)?

When I added array variables and the array keys expansion, I used the ksh93
syntax (${!var[sub]}) and tried to avoid conflict with the existing
indirect expansion as much as possible (that was back when we still thought
there was a chance that POSIX would standardize arrays and it was useful to
have consistent implementations). The ksh93 expansion syntax made it
invalid to use the array keys expansion as part of the ${param:-word}
expansion, so that's how I made it work. Since it wasn't a candidate for
that family of expansions, the existing variable indirection syntax
controls.

Makes sense. IMO, it still might be worth mentioning explicitly in the
manual what happens when you try to mix and match various expansions,
at least in cases where there's more than one reasonable
interpretation. (Granted, a closer reading of the relevant sections
would have made it clear that *only* array[@] and array[*] were privy
to the secondary meaning of "!", as opposed to variations.)