Assignment of $* to a var removes spaces on unset IFS.

[Bize Ma]

Executing this code:

set -- " foo " " bar baz " " quux "
unset IFS
a=$*
b="$*"
printf '[%s]' "$a" "$b"; echo

Leads to this results in several shells:

ash : [ foo bar baz quux ][ foo bar baz
quux ]
dash : [ foo bar baz quux ][ foo bar baz
quux ]
b205sh : [ foo bar baz quux ][ foo bar baz
quux ]
b30sh : [ foo bar baz quux ][ foo bar baz
quux ]
b32sh : [ foo bar baz quux ][ foo bar baz
quux ]
b41sh : [ foo bar baz quux ][ foo bar baz
quux ]
b42sh : [ foo bar baz quux ][ foo bar baz
quux ]
b43sh : [foo bar baz quux][ foo bar baz quux ]
b44sh : [foo bar baz quux][ foo bar baz quux ]
bash : [foo bar baz quux][ foo bar baz quux ]
posixbash : [foo bar baz quux][ foo bar baz quux ]
lksh : [ foo bar baz quux ][ foo bar baz
quux ]
mksh : [ foo bar baz quux ][ foo bar baz
quux ]
ksh93 : [ foo bar baz quux ][ foo bar baz
quux ]
attsh : [ foo bar baz quux ][ foo bar baz
quux ]
zsh : [ foo bar baz quux ][ foo bar baz
quux ]


Bash since 4.3 fails to follow what the documentation describes as that on
an assignment
values do not undergo splitting or globing.

[Robert Elz]

Date: Mon, 13 Aug 2018 11:51:08 -0400
From: Bize Ma <***@gmail.com>
Message-ID: <CAFra36g+***@mail.gmail.com>

| Bash since 4.3 fails to follow what the documentation describes as that on
| an assignment values do not undergo splitting or globing.

This is already fixed.

What's more, reporting bugs in bash 4.3 is just a waste of everyone's time.
Upgrade to (at least) the current released version, and use that instead.

kre

[Eduardo Bustamante]

On Mon, Aug 13, 2018 at 5:25 PM Robert Elz <***@munnari.oz.au> wrote:
[...]

Post by Robert Elz
What's more, reporting bugs in bash 4.3 is just a waste of everyone's time.
Upgrade to (at least) the current released version, and use that instead.

Adding to this point:

* bash 4.3 was released 4 years ago (2014-02-26)
* bash 4.4 is the current release of bash (2016-09-15)
* bash 5.0 is in alpha state (2018-05-22)

I know that some stable Linux distributions might still be
distributing 4.3, but that just means that you have to report the bug
against these distros, not upstream.

[Bize Ma]

Post by Eduardo Bustamante
[...]

Post by Robert Elz
What's more, reporting bugs in bash 4.3 is just a waste of everyone's

time.

Post by Robert Elz
Upgrade to (at least) the current released version, and use that instead.

* bash 4.3 was released 4 years ago (2014-02-26)
* bash 4.4 is the current release of bash (2016-09-15)
* bash 5.0 is in alpha state (2018-05-22)

You both are confused. The original table included

Post by Eduardo Bustamante
b43sh : [foo bar baz quux][ foo bar baz quux ]
b44sh : [foo bar baz quux][ foo bar baz quux ]

That's 4.3.48(2)-release and 4.4.19(2)-release


Both are failing.

Note that 4.4.19 is newer than what is available in
https://www.gnu.org/software/bash/

The report is valid for present releases.
The bug is still present (since 4.3).

Please, do not waste our time in incorrect claims.

Do your homework and test !

Post by Eduardo Bustamante
I know that some stable Linux distributions might still be
distributing 4.3, but that just means that you have to report the bug
against these distros, not upstream.

That is not the problem, don't jump to conclusions.

[Eduardo A. Bustamante López]

On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote:
(...)

Post by Bize Ma
Please, do not waste our time in incorrect claims.
Do your homework and test !

(...)

Did you test Bash 5.0? Because that's where the current bug fixes are going to.
I don't see Chet releasing a new version of 4.4 to fix something that's already
fixed there.

***@ubuntu:~/src/gnu/bash$ cat /tmp/script
echo $BASH_VERSION
set -- " foo " " bar baz " " quux "
unset IFS
a=$*
b="$*"
printf '[%s]' "$a" "$b"; echo

***@ubuntu:~/src/gnu/bash$ bash /tmp/script
4.4.19(1)-release
[foo bar baz quux][ foo bar baz quux ]

***@ubuntu:~/src/gnu/bash$ ./bash /tmp/script
5.0.0(1)-alpha
[ foo bar baz quux ][ foo bar baz quux ]

Please read this thread:

* http://lists.nongnu.org/archive/html/bug-bash/2017-09/msg00058.html

[Bize Ma]

Post by Eduardo A. Bustamante López
(...)

Post by Bize Ma
Please, do not waste our time in incorrect claims.
Do your homework and test !

(...)
4.4.19(1)-release
[foo bar baz quux][ foo bar baz quux ]

So, it is confirmed that the bug exists in the present release, Is it not?

Post by Eduardo A. Bustamante López
Did you test Bash 5.0? Because that's where the current bug fixes are going to.

So, is it a "wont fix" for 4.4 (present release) ?

Post by Eduardo A. Bustamante López
I don't see Chet releasing a new version of 4.4 to fix something that's already
fixed there.

That version is not even beta, it is still alpha, are you asking that
everyone should use
non-released (and not yet tested as beta) alpha release ?

Post by Eduardo A. Bustamante López
echo $BASH_VERSION
set -- " foo " " bar baz " " quux "
unset IFS
a=$*
b="$*"
printf '[%s]' "$a" "$b"; echo

Thanks for copying the supplied script.....

Post by Eduardo A. Bustamante López
5.0.0(1)-alpha
[ foo bar baz quux ][ foo bar baz quux ]

Thanks for testing. !!

Post by Eduardo A. Bustamante López
* http://lists.nongnu.org/archive/html/bug-bash/2017-09/msg00058.html
This is a bug in bash and it should be fixed, not excused.

To which I agree. After a year, nothing else have been said about it.

It seems about time to get it solved. Or say that it won't be.

[Eduardo A. Bustamante López]

On Mon, Aug 13, 2018 at 10:52:20PM -0400, Bize Ma wrote:
(...)

Post by Bize Ma
That version is not even beta, it is still alpha, are you asking that
everyone should use
non-released (and not yet tested as beta) alpha release ?

(...)

I didn't say that. My point is that, in the context of bug reports, it's
important that you always test against the *unreleased* version of the code,
since that's where most bug fixes are queued up (or at least the ones that break
backwards compatibility).

Post by Bize Ma

Post by Eduardo A. Bustamante López
This is a bug in bash and it should be fixed, not excused.

To which I agree. After a year, nothing else have been said about it.

For some reason the threading broke in the archive browser. Here's the response
from Chet:

http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html

The fix for this issue is already in the Git repository (devel branch) and in
Bash 5.0.

I cannot answer for Chet, but please consider this:

The fix for this bug breaks backwards compatibility, so that means it cannot be
a patch-level fix for 4.4. It should either be a minor version increase (4.5) or
go into the next major release, which is 5.0 (which was already declared alpha
state, and includes the fix).

Post by Bize Ma
It seems about time to get it solved. Or say that it won't be.

Again, it's fixed already and the fix is scheduled to go out in the next
release (5.0).

Given that, do you think this bug is severe enough to have to issue a new minor
version just for it? (4.5),

or can we just wait for 5.0 to come out which fixes this and a bunch of other
bugs? (see the CHANGES file, in particular, entry `oo',
http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b)

[Pierre Gaston]

On Tue, Aug 14, 2018 at 6:18 AM Eduardo A. Bustamante López <

Post by Eduardo A. Bustamante López
(...)

Post by Bize Ma
That version is not even beta, it is still alpha, are you asking that
everyone should use
non-released (and not yet tested as beta) alpha release ?

(...)
I didn't say that. My point is that, in the context of bug reports, it's
important that you always test against the *unreleased* version of the code,
since that's where most bug fixes are queued up (or at least the ones that break
backwards compatibility).

Post by Bize Ma

Post by Eduardo A. Bustamante López
This is a bug in bash and it should be fixed, not excused.

To which I agree. After a year, nothing else have been said about it.

For some reason the threading broke in the archive browser. Here's the response
http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html
The fix for this issue is already in the Git repository (devel branch) and in
Bash 5.0.
The fix for this bug breaks backwards compatibility, so that means it cannot be
a patch-level fix for 4.4. It should either be a minor version increase (4.5) or
go into the next major release, which is 5.0 (which was already declared alpha
state, and includes the fix).

Post by Bize Ma
It seems about time to get it solved. Or say that it won't be.

Again, it's fixed already and the fix is scheduled to go out in the next
release (5.0).
Given that, do you think this bug is severe enough to have to issue a new minor
version just for it? (4.5),
or can we just wait for 5.0 to come out which fixes this and a bunch of other
bugs? (see the CHANGES file, in particular, entry `oo',
http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b
)

[Bize Ma]

(...) My point is that, in the context of bug reports, it's
important that you always test against the *unreleased* version of the code,

Not if the version to test is the released one.

http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html
The fix for this issue is already in the Git repository (devel branch) and in
Bash 5.0.

Excellent, that is good to know. There is a solution.

Chet explains it clearly.

Just that it is not applied to the present release version 4.4.
Maybe it should be put into a "stack of pending patches" that all will be
applied when a new number release (4.5) has to be done for whatever
other reason.

When that happens, we can say that it "is solved for current release".

The fix for this bug breaks backwards compatibility

In what sense? Other shells don't have this issue and the change got
introduced in 4.3, so, this change is backward compatible with several
versions up to 4.3 (backward compatible).

so that means it cannot be
a patch-level fix for 4.4. It should either be a minor version increase (4.5) or
go into the next major release, which is 5.0 (which was already declared alpha
state, and includes the fix).

I am not who must make that call. Someone else must.

If this is not going to be solved in 4.4, then it is a "wontfix" for that
release.

It seems about time to get it solved. Or say that it won't be.
Again, it's fixed already

No, it is not "fixed already" on 4.4 release.
Let's call things as they are, not as anyone "hope" they can be.

and the fix is scheduled to go out in the next

release (5.0).

Excellent, great, version 5.0 has one issue less to solve. Yay!!

Given that, do you think this bug is severe enough to have to issue a new minor
version just for it? (4.5),

Don't ask me, it is not a call that I can make, I am just a random user.

or can we just wait for 5.0 to come out which fixes this and a bunch of other
bugs? (see the CHANGES file, in particular, entry `oo',
http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=
bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b)

Yes, there it says that some changes were made to match POSIX
interpretation of $* $@.
I believe that the issue presented in the OP is one of those changes for
5.0.

[Chet Ramey]

 > This is a bug in bash and it should be fixed, not excused.
To which I agree. After a year, nothing else have been said about it.
It seems about time to get it solved. Or say that it won't be.

It's fixed in the current development versions, and that fix will be in
the next release. It looks like it's been fixed since April, 2017 as part
of a code cleanup to implement Posix interp 888:

http://lists.nongnu.org/archive/html/bug-bash/2017-04/msg00001.html

[Greg Wooledge]

Post by Bize Ma
Note that 4.4.19 is newer than what is available in
https://www.gnu.org/software/bash/

The current patch level of bash can be obtained by downloading the most
recent tarball from <http://ftp.gnu.org/gnu/bash/> and then applying the
subsequent patches from <http://ftp.gnu.org/gnu/bash/bash-4.4-patches/>,
or whichever directory is appropriate for the current version.

The patches for 4.4 currently go up to bash44-023, which would correspond
to bash version 4.4.23 once compiled.

I will also repeat, once more, my advice that one should NEVER write
a script containing an unquoted $* or $@ expansion. It breaks in all
kinds of ways in more than one shell. Just don't do it, and these
problems go away.

[Chet Ramey]

Post by Greg Wooledge

Post by Bize Ma
Note that 4.4.19 is newer than what is available in
https://www.gnu.org/software/bash/

The current patch level of bash can be obtained by downloading the most
recent tarball from <http://ftp.gnu.org/gnu/bash/> and then applying the
subsequent patches from <http://ftp.gnu.org/gnu/bash/bash-4.4-patches/>,
or whichever directory is appropriate for the current version.

It's always easier to download

http://git.savannah.gnu.org/cgit/bash.git/snapshot/bash-master.tar.gz

That is the head of the master branch, which is always the current
released version plus all released patches.

[Robert Elz]

Date: Mon, 13 Aug 2018 21:36:23 -0400
From: Bize Ma <***@gmail.com>
Message-ID: <CAFra36gUtG8x8SoNP9fGNSp-***@mail.gmail.com>

| You both are confused. The original table included

You misinterpreted my message.

I said two independant things ... first that the bug is already fixed.
I didn't say that the fixed version was released yet.

Second, that you should stop reporting bugs in 4.3 (which is
relatively ancient). If you had reported it as a bug in 4.4
(which means that you had indicated that you had verified
it there) the second sentence would not have been in my
reply.

| The report is valid for present releases.

That's true. This is an insignificant issue really and not
even really worthy of a patch. But it will be fixed in the next
release.


Eduardo A. Bustamante López <***@gmail.com> said:
| My point is that, in the context of bug reports, it's important that you
| always test against the *unreleased* version of the code,

(first, sorry if your name gets messed up ... it looks correct in my
mail composition window, but the message is likely to be marked
as "charset=us-ascii" or something, and the accented character
is unlikely to survive correctly...)

I think that is going too far - if a bug is in the latest released
version, it is perfectly reasonable for people to report it.

Asking people to even know where the development version is,
let alone how to fetch it, build it, etc, is a bit too much to expect.
It is nice if that can be done, but certainly should not be a
requirement.

Sometimes (for any bug report) the submitter will just be told
"already fixed" which means that a later release will have the fix
- which is the best result that could be achieved if it was not
already fixed.

| The fix for this bug breaks backwards compatibility,

All bug fixes break backwards compat, but only for scripts
that incorrectly assumed that the broken behaviour was
correct - such scripts are also broken.

I mean, what would you say to someone who said "xxx always
dumped core when I did ..., and I was relying on that to detect
the error by the presence of the core file, in the latest revision
it no longer dumps core, but instead gives an error message,
please put the core dump back."

There is no reason the fix for this could not be in a 4.4 patch.
On the other hand this is such an obscure issue that getting
the fix distributed in a hurry is also not all that important.

kre

[Greg Wooledge]

This reply was sent to me without Cc-ing the list. I have added the Cc.

Post by Greg Wooledge
I will also repeat, once more, my advice that one should NEVER write

That is plainly INCORRECT, Greg.

You are incorrect.

Post by Greg Wooledge
It breaks in all kinds of ways in more than one shell.

That several shells do different things is a bug on those shells, not bash.

Agreed. And the fact that IN REAL LIFE, THOSE SHELLS EXIST AND HAVE
THOSE BUGS, which are triggered by incorrect code, is a reason to write
code correctly so as not to trigger those bugs.

Or maybe you're one of those people who doesn't care about reality.

Post by Greg Wooledge
Just don't do it, and these problems go away.

echo $*
There is nothing wrong with that (don't claim that it change in different
shells, that is a different issue than using split+glob in bash, go back to
the point above about other shells if you wish).

There is absolutely definitely positively 100% certainly something wrong
with that.

Let's break your script, shall we?

Here's your script, except I'm going to represent it as a function. Doing
it as a script would have the same effect.

glob() {
# "Return" (write to stdout, one per line) the expansions of all
# arguments as globs against the current working directory.
printf %s\\n $*
}

Will you at least agree that this is your intent, and a fair
representation of your proposed solution? I'll take that as a "yes".

So now, you can pass SOME globs to it (properly quoted), and it will
appear to work for those globs:

wooledg:~$ glob '*.pdf'
0400_0001.pdf
11412687.pdf
11412859.pdf
[...]
Epic Web Service - PatientLookup.pdf
[...]

At this point, the naive script writer will say "Yay, it worked!"

The experienced script writer knows that it did not, in fact, "work".
It only "worked" for this one degenerate kindergarten-level example.

Let's test it again.

wooledg:~$ glob '*Web Service*'
*Web
Service*

But... but... but... the PREVIOUS glob worked! Why didn't this one
work?

Because the script (function) does not ACTUALLY work. It is broken.
You can't solve this problem using an unquoted $* expansion. Not in
reality.

Some of us care about reality.

Other languages have no problem with this. Tcl, for example, has a [glob]
command that takes a list of glob-patterns and returns a list of files
that match them.

wooledg:~$ tclsh
% puts [join [glob {*Web Service*}] \n]
Epic Web Service - PatientLookup.pdf
% puts [join [glob *.pdf] \n]
Infoblox_DNS_mgmt.pdf
prtest.pdf
PMCNoteCoverPage.pdf
[...]

Of course, it doesn't sort them, because I didn't call [lsort]. Adding
that is trivial.

Now, you tried to implement Tcl's [glob] in bash, but you did it naively
and the naive version does not work. You took a shortcut. That shortcut
falls off a cliff.

I'll leave the non-broken implementation as an exercise for the reader.

Now, back to the original points:

1) Certain shells have bugs in them. These shells are in widespread use
on real systems in real life.

2) One of those shells is bash, which makes it relevant to bug-bash.

3) Some of those bugs involve the unquoted expansions of $* and $@.

4) Even in the ABSENCE of such bugs, the use of an unquoted $* or $@
expansion does not actually solve the problems you claim it solves.

5) Therefore, for TWO reasons, you should not use unquoted $* or $@
in your shell scripts.

Reason 1: it doesn't solve the problem.
Reason 2: it sometimes breaks even worse due to shell bugs.

[Ilkka Virta]

Post by Greg Wooledge
glob() {
# "Return" (write to stdout, one per line) the expansions of all
# arguments as globs against the current working directory.
printf %s\\n $*
}
But... but... but... the PREVIOUS glob worked! Why didn't this one
work?

I'm sure you know what word splitting is.

Post by Greg Wooledge
I'll leave the non-broken implementation as an exercise for the reader.

$ glob() { local IFS=; printf '%s\n' $*; }
$ touch "foo bar.txt" "foo bar.pdf"
$ glob "foo bar*"
foo bar.pdf
foo bar.txt

(Well, you'd probably want 'nullglob' too, and there's the minor issue
that printf '%s\n' prints at least one line even if there are no
arguments but I'll ignore that for now.)


Of course, in most cases, and unquoted expansion is not what one wants,
but if there's need to glob in the shell, then an unquoted expansion is
what has to be used. How IFS affects word splitting isn't just about
$* , the issue is the same even if you only have one glob in a regular
variable.

[Bize Ma]

Post by Greg Wooledge
This reply was sent to me without Cc-ing the list. I have added the Cc.

Post by Greg Wooledge
I will also repeat, once more, my advice that one should NEVER write

That is plainly INCORRECT, Greg.

You are incorrect.

I am always incorrect, I like to be so.
But we are discussing an issue, not my personal problems.

Post by Greg Wooledge

Post by Greg Wooledge
It breaks in all kinds of ways in more than one shell.

That several shells do different things is a bug on those shells, not

bash.
Agreed.

Excellent, we agree.

Post by Greg Wooledge
And the fact that IN REAL LIFE, THOSE SHELLS EXIST AND HAVE
THOSE BUGS, which are triggered by incorrect code

No, sorry, but bugs are triggered by perfectly good code, that's why they
are called bugs.

Post by Greg Wooledge
, is a reason to write
code correctly so as not to trigger those bugs.

Writing code to *work around* bugs is *not* the correct solution.
It is only a way to *perpetuate* those bugs.
The correct solution is to resolve the bugs.

But this is just a smoke curtain. There are no bugs about $@ and $* in Bash.

Or maybe you're one of those people who doesn't care about reality.
Sometimes I don't , most of the time when I sleep, sometimes when I day
dream.

Post by Greg Wooledge

Post by Greg Wooledge
Just don't do it, and these problems go away.

echo $*
There is nothing wrong with that (don't claim that it change in different
shells, that is a different issue than using split+glob in bash, go back

to

the point above about other shells if you wish).

There is absolutely definitely positively 100% certainly something wrong
with that.

Ah, yes, that's correct, thanks, I should have used printf, not echo.
But you corrected it, thanks again.

Let's break your script, shall we?

Post by Greg Wooledge
Here's your script, except I'm going to represent it as a function. Doing
it as a script would have the same effect.
glob() {
# "Return" (write to stdout, one per line) the expansions of all
# arguments as globs against the current working directory.
printf %s\\n $*
}
Will you at least agree that this is your intent,

No, that is *not* my intent.
You can not implement a glob function when *both: split and glob* are in
effect.

If you want a glob function, stop the split, set IFS to null:

$ cat script
#!/bin/bash
glob() ( IFS=; printf %s\\n $* )
glob '*Web Service*'

$ ./script
Epic Web Service - PatientLookup.pdf


and a fair

Post by Greg Wooledge
representation of your proposed solution? I'll take that as a "yes".

No it is not a fair representation of anything I said.
Only of a twist that you want to present here.

So now, you can pass SOME globs to it (properly quoted), and it will
But it will fail to work on the next example as you want to demonstrate,
therefore you have not coded correctly to workaround the shells bugs
(your own words).

Some of us care about reality.
Yes, when I am awake.

Other languages have no problem with this. Tcl,

(...)

But we are in bash, are we not?


Now, you tried to implement Tcl's [glob] in bash, but you did it naively

Post by Greg Wooledge
and the naive version does not work. You took a shortcut. That shortcut
falls off a cliff.

No, *you* did. You implemented a flawed glob function.

Post by Greg Wooledge
I'll leave the non-broken implementation as an exercise for the reader.

You are not able to do it?

Post by Greg Wooledge
1) Certain shells have bugs in them. These shells are in widespread use
on real systems in real life.
2) One of those shells is bash, which makes it relevant to bug-bash.

Not in bash.

Post by Greg Wooledge
expansion does not actually solve the problems you claim it solves.

I claimed nothing. I presented a valid use. Don't put words in my mouth.

Post by Greg Wooledge
in your shell scripts.
Reason 1: it doesn't solve the problem.
Reason 2: it sometimes breaks even worse due to shell bugs.

There is no therefore if the premises are wrong.