exiting noninteractive shells on 'shift 2'

[Eric Blake]

If I'm reading POSIX correctly, shift is a special built-in utility, and
if '$#' is 0 or 1, then 'shift 2' counts as a utility error that shall
exit the shell, per the table in 2.8.1 Consequences of Shell Errors:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01


dash gets this right:

$ dash -c 'set 1

shift 2
echo "oops"'

dash: 2: shift: can't shift that many

but bash happily lets 'shift 2' fail with $? set to 1 but continues on
with execution of the rest of the script, even when in POSIX mode:

$ bash -c 'set 1

shift 2
echo "oops"'

oops
$ bash -c 'set 1; set -o posix

shift 2
echo "oops"'

oops

I spent more than an hour today figuring out why my shell script was
inf-looping, and traced it back to failure to check for non-zero status
from 'shift 2', where I had been expecting immediate hard failure of the
entire script. I don't care if bash doesn't hard-exit when not in POSIX
mode, and it must not hard-exit when in interactive use; but the
non-interactive use not doing a hard exit seems like a bug.

On the other hand, the POSIX wording for shift, under EXIT STATUS,
weakens the "shall fail" from the earlier table into a weaker "may fail":

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#shift

If the n operand is invalid or is greater than "$#", this may be
considered a syntax error and a non-interactive shell may exit; if the
shell does not exit in this case, a non-zero exit status shall be
returned. Otherwise, zero shall be returned.

Tested on Fedora 28 with bash-4.4.23-1.fc28.x86_64

[Chet Ramey]

If I'm reading POSIX correctly, shift is a special built-in utility, and if
'$#' is 0 or 1, then 'shift 2' counts as a utility error that shall exit
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01
$ dash -c 'set 1

Post by Eric Blake
shift 2
echo "oops"'

dash: 2: shift: can't shift that many
but bash happily lets 'shift 2' fail with $? set to 1 but continues on with

As you later note:

"If the n operand is invalid or is greater than "$#", this may be
considered a syntax error and a non-interactive shell may exit; if the
shell does not exit in this case, a non-zero exit status shall be returned.
Otherwise, zero shall be returned."

So the bash behavior is not a conformance issue, and allowed by the
standard.

[Eric Blake]

Post by Chet Ramey

If I'm reading POSIX correctly, shift is a special built-in utility, and if
'$#' is 0 or 1, then 'shift 2' counts as a utility error that shall exit
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01
$ dash -c 'set 1

Post by Eric Blake
shift 2
echo "oops"'

dash: 2: shift: can't shift that many
but bash happily lets 'shift 2' fail with $? set to 1 but continues on with

"If the n operand is invalid or is greater than "$#", this may be
considered a syntax error and a non-interactive shell may exit; if the
shell does not exit in this case, a non-zero exit status shall be returned.
Otherwise, zero shall be returned."
So the bash behavior is not a conformance issue, and allowed by the
standard.

Well, there's STILL a conformance issue - the standard requires that
unless documented otherwise, any time a command line tool exits with
non-zero status, that it outputs a message to stderr explaining the
error. The page for 'shift' does not document an exception for an exit
status of 1 being silent, yet bash is completely silent when 'shift 2'
returns non-zero status. Producing an error message to stderr would
call the developer's attention to the bug in their script and the fact
that bash did not shift out 1 argument, whether or not you also change
bash to exit the script or continue with execution.

[Chet Ramey]

Well, there's STILL a conformance issue - the standard requires that unless
documented otherwise, any time a command line tool exits with non-zero
status, that it outputs a message to stderr explaining the error. 

Where?

[Eric Blake]

Post by Chet Ramey

Well, there's STILL a conformance issue - the standard requires that unless
documented otherwise, any time a command line tool exits with non-zero
status, that it outputs a message to stderr explaining the error.

Where?

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#shift

"STDERR

The standard error shall be used only for diagnostic messages."

"CONSEQUENCES OF ERRORS

Default."

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html

1.4 Utility Description Defaults:

STDERR

"Default Behavior: When this section is listed as "The standard error
shall be used only for diagnostic messages.", it means that, unless
otherwise stated, the diagnostic messages shall be sent to the standard
error only when the exit status indicates that an error occurred and the
utility is used as described by this volume of POSIX.1-2017."

CONSEQUENCES OF ERRORS

"The following shall apply to each utility, unless otherwise stated:

If the requested action cannot be performed on an operand
representing a file, directory, user, process, and so on, the utility
shall issue a diagnostic message to standard error and continue
processing the next operand in sequence, but the final exit status shall
be returned as non-zero.

For a utility that recursively traverses a file hierarchy (such as
find or chown -R), if the requested action cannot be performed on a file
or directory encountered in the hierarchy, the utility shall issue a
diagnostic message to standard error and continue processing the
remaining files in the hierarchy, but the final exit status shall be
returned as non-zero.

If the requested action characterized by an option or
option-argument cannot be performed, the utility shall issue a
diagnostic message to standard error and the exit status returned shall
be non-zero.

When an unrecoverable error condition is encountered, the utility
shall exit with a non-zero exit status.

A diagnostic message shall be written to standard error whenever an
error condition occurs."

Since 'shift 2' when $# is 1 is an error, and results in non-zero
status, it should print a diagnostic to stderr.

[Chet Ramey]

Post by Chet Ramey

Well, there's STILL a conformance issue - the standard requires that unless
documented otherwise, any time a command line tool exits with non-zero
status, that it outputs a message to stderr explaining the error.

Where?

    A diagnostic message shall be written to standard error whenever an
error condition occurs."
Since 'shift 2' when $# is 1 is an error, and results in non-zero status,
it should print a diagnostic to stderr.

So maybe the thing to do is to turn on the shift_verbose option in posix
mode.

[Eric Blake]

Post by Chet Ramey

    A diagnostic message shall be written to standard error whenever an
error condition occurs."
Since 'shift 2' when $# is 1 is an error, and results in non-zero status,
it should print a diagnostic to stderr.

So maybe the thing to do is to turn on the shift_verbose option in posix
mode.

Cool - I didn't realize we already had a shopt for that. Yes, it sounds
like in POSIX mode, shift_verbose should be turned on.

On a related note, should turning on POSIX mode also auto-enable xpg_echo?

[Chet Ramey]

Post by Eric Blake

Post by Chet Ramey

     A diagnostic message shall be written to standard error whenever an
error condition occurs."
Since 'shift 2' when $# is 1 is an error, and results in non-zero status,
it should print a diagnostic to stderr.

So maybe the thing to do is to turn on the shift_verbose option in posix
mode.

Cool - I didn't realize we already had a shopt for that. Yes, it sounds
like in POSIX mode, shift_verbose should be turned on.
On a related note, should turning on POSIX mode also auto-enable xpg_echo?

No. That's just more of a can of worms than I want to deal with. Bash
claims strict conformance with both posix and xpg_echo enabled (that's what
the `strict-posix' configure option does, among other things).

[Grisha Levit]

Post by Eric Blake
If the n operand is invalid or is greater than "$#", this may be
considered a syntax error and a non-interactive shell may exit; if the
shell does not exit in this case, a non-zero exit status shall be
returned.

This seems to say that "shift N" *may* be considered an error *in which
case* the rules in 2.8.1 must apply. But since bash does not consider it to
be an error, it behaves as the second half of the sentence proscribes and
returns a non-zero status.