Bash removes unrequested characters in bracket expressions (not a range).

[Bize Ma]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: Linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib
-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2
-fdebug-prefix-map=/build/bash-7fckc0/bash-4.4=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall
-no-pie -Wno-parentheses -Wno-format-security
uname output: Linux io 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2
(2018-10-27) x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 4.4
Patch Level: 12
Release Status: release



Description:

Bash is removing characters not explicitly listed in a bracket
expression (character range).
In this example, it is removing digits from other languages.

Also tested (and it fails) in bash 3.{0,1,3} 4.{1,2,3} and 5.0
Not a problem in bash 2.{0,1}



Repeat-By:

If the characters are a problem: please visit:
https://unix.stackexchange.com/q/483743/265604

$ a='0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९'
$ echo "${a//[0123456789]}"
۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९

[Chet Ramey]

Post by Bize Ma
Bash Version: 4.4
Patch Level: 12
Release Status: release
Bash is removing characters not explicitly listed in a bracket
expression (character range).
In this example, it is removing digits from other languages.

What is your locale?

[Bize Ma]

Post by Chet Ramey

Post by Bize Ma
Bash Version: 4.4
Patch Level: 12
Release Status: release
Bash is removing characters not explicitly listed in a bracket
expression (character range).
In this example, it is removing digits from other languages.

What is your locale?

The locale used was en_US.utf-8 but also happens with 459
locales out of 868 available under Debian (not in C, for example).

Also in all locales affected (except one), setting either
LC_ALL=$loc or LC_COLLATE=$loc did the same.
Except in zh_CN.gb18030

But IMO locale collation should not be used for an explicit list.

I have been made aware that there is a
cstart = cend = FOLD (cstart);
inside the `sm_loop.c` file that will convert into a range many
individual character. If that understanding is correct that is the
source of the difference with other shells.

I have the perception that a collation table *must have a "total order"*,
in fact, an strict total order. If two characters `a` and `b` could sort as
equal the order will fail to provide a confirmation that a character is
absent from the list. Consider characters `a`, `b` and `c`, if a and b
sort as equal, a sorted list in which we find `a` followed by `c` doesn't
confirm that `b` is absent as the order could well be `b a c`.

In this case, there must not be any other character than `a` in the
range `a-a` and using a range `a-a` is equivalent (just slower and
more complex) to the single character `a`.

If this is not the case, the error is in the collation table, not in using
single (faster) characters. And what should be updated is such
collation table IMO.

[Bize Ma]

I am going to forward the rest of our exchange to bug-bash;

Great!

you left the mailing list off your replies for some reason.
Chet

Since it was *you* who directed the answer only to me, I assumed
(now seemingly incorrectly) that that was what you wanted to do.

[Chet Ramey]

Post by Bize Ma
But IMO locale collation should not be used for an explicit list.

Collation order is used for each individual character in a bracket
expression when compared against the string, as posix specifies.

Post by Bize Ma
I have been made aware that there is a
      cstart = cend = FOLD (cstart);
inside the `sm_loop.c` file that will convert into a range many
individual character. If that understanding is correct that is the
source of the difference with other shells.

I'm not sure what you mean by "convert into a range." If cstart and cend
were treated as a range, the start end and end characters would be the
same. If cstart == cend, a character that collates >= cstart and <= cend
would have to collate equal to cstart and cend.

If I were you, I would file a bug report with Debian against wcscoll.

It returns 0 (equal) for L"٠" and L"0" without setting errno. That's
clearly a problem with wcscoll (if the character isn't valid in the current
locale) or the locale definition.

[Bize Ma]

[...]

Post by Bize Ma
I have been made aware that there is a
cstart = cend = FOLD (cstart);
inside the `sm_loop.c` file that will convert into a range many
individual character. If that understanding is correct that is the
source of the difference with other shells.

I'm not sure what you mean by "convert into a range." If cstart and cend
were treated as a range, the start end and end characters would be the
same. If cstart == cend, a character that collates >= cstart and <= cend
would have to collate equal to cstart and cend.

Yes, exactly, a range where the start and the end are the same.

Try:

$ touch 0 1 ٠ ١ ۰ ۱ ߀ ߁ ० १
$ echo [1]
1 ١

It is converted to the same range as this

$ echo [1-1]
1 ١

That happens because up to glibc 2.27 this has been the collation order of
those characters (search in /usr/share/i18n/locales/iso14651_t1_common) :

<U0030> <0>;<BAS>;<MIN>;IGNORE
<U0660> <0>;<BAS>;<MIN>;IGNORE

Collate to exactly the same values. This breaks the capacity to detect that
a character is absent in a list ordered by the collation order.

[Bize Ma]

Post by Chet Ramey

Post by Bize Ma
But IMO locale collation should not be used for an explicit list.

Collation order is used for each individual character in a bracket
expression when compared against the string, as posix specifies.

Yes, values resulting from a glob expansion should be compared with strcoll.

How many characters should there be in a range like [0-0] ?
Or to be more precise: in a [0] bracket expression? one?

If I were you, I would file a bug report with Debian against wcscoll.
And I would be told that wcscoll is doing what the collation file 14651 is
telling it to do.

And, that in any case, that file has been updated in glib2.8 anyway.

Post by Chet Ramey
It returns 0 (equal) for L"٠" and L"0" without setting errno. That's
clearly a problem with wcscoll (if the character isn't valid in the current
locale) or the locale definition.

Both characters collate to the same position as I have already explained.

I don't follow you about what you mean with:
*(if the character isn't valid in the current locale).*

[Chet Ramey]

 [...]

Post by Bize Ma
I have been made aware that there is a
      cstart = cend = FOLD (cstart);
inside the `sm_loop.c` file that will convert into a range many
individual character. If that understanding is correct that is the
source of the difference with other shells.

I'm not sure what you mean by "convert into a range." If cstart and cend
were treated as a range, the start end and end characters would be the
same. If cstart == cend, a character that collates >= cstart and <= cend
would have to collate equal to cstart and cend.
Yes, exactly, a range where the start and the end are the same.

A range like that is exactly equivalent to a single ordinary character.

POSIX: "An ordinary character in the list should only match that character,
but may match any single character that collates equally with that
character"

[Chet Ramey]

Post by Bize Ma

Post by Bize Ma
But IMO locale collation should not be used for an explicit list.

Collation order is used for each individual character in a bracket
expression when compared against the string, as posix specifies.

Yes, values resulting from a glob expansion should be compared with strcoll.
How many characters should there be in a range like [0-0] ?
Or to be more precise: in a [0] bracket expression? one?

There should be one character ("0") that matches as many characters as
collate equal to the character "0", as per the POSIX quote in my previous
message.

Post by Bize Ma
If I were you, I would file a bug report with Debian against wcscoll.
And I would be told that wcscoll is doing what the collation file 14651 is
telling it to do.

Sure.

Post by Bize Ma
And, that in any case, that file has been updated in glib2.8 anyway.

That should fix the problem without forcing applications to attempt to
impose a total ordering even when strcoll/wcscoll returns 0.

Post by Bize Ma
It returns 0 (equal) for L"٠" and L"0" without setting errno. That's
clearly a problem with wcscoll (if the character isn't valid in the current
locale) or the locale definition.
Both characters collate to the same position as I have already explained.

Yes, so the locale definition files imposing a total ordering will be a
clear improvement.

Post by Bize Ma
I don't follow you about what you mean with: /(if the character isn't valid
in the current
locale)./

There are codepoints that correspond to characters in one locale but don't
map to a valid character in another.

Chet

[Chet Ramey]

Post by Chet Ramey

Post by Bize Ma
Bash is removing characters not explicitly listed in a bracket
expression (character range).
In this example, it is removing digits from other languages.

What is your locale?
 
The locale used was en_US.utf-8 but also happens with  459
locales out of 868 available under Debian (not in C, for example).
Also in all locales affected (except one), setting either
LC_ALL=$loc or LC_COLLATE=$loc did the same.
Except in zh_CN.gb18030
But IMO locale collation should not be used for an explicit list.

Collation order is used for each individual character in a bracket
expression when compared against the string, as posix specifies.

Post by Chet Ramey
I have been made aware that there is a
      cstart = cend = FOLD (cstart);
inside the `sm_loop.c` file that will convert into a range many
individual character. If that understanding is correct that is the
source of the difference with other shells.

I'm not sure what you mean by "convert into a range." If cstart and cend
were treated as a range, the start end and end characters would be the
same. If cstart == cend, a character that collates >= cstart and <= cend
would have to collate equal to cstart and cend.

[Chet Ramey]

Post by Bize Ma
Bash Version: 4.4
Patch Level: 12
Release Status: release
Bash is removing characters not explicitly listed in a bracket
expression (character range).
In this example, it is removing digits from other languages.
Also tested (and it fails) in bash 3.{0,1,3} 4.{1,2,3} and 5.0
Not a problem in bash 2.{0,1}

I can't reproduce this:

$ cat ./x4

a='0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९'

recho "${a}"
recho "${a//[0123456789]}"
$ ../bash-5.0-beta2/bash ./x4
argv[1] = <0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९>
argv[1] = < ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९>
$ ../bash-4.4-patched/bash ./x4
argv[1] = <0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९>
argv[1] = < ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९>

[Bize Ma]

If you could take a look at https://unix.stackexchange.com/a/483835/265604
you will see that it has been confirmed on "Ubuntu 17.10 (glibc 2.26) and on
Ubuntu 18.04 (glibc 2.27), but it seems to be fixed on Ubuntu 18.10 (glibc
2.28)"

It is interesting that (finally) glibc 2.28 has added a fourth sort key
equal to the
Unicode code point. That forces the order of all characters to be unique.

[Chet Ramey]

Post by Bize Ma
 
If you could take a look at https://unix.stackexchange.com/a/483835/265604
you will see that it has been confirmed on "Ubuntu 17.10 (glibc 2.26) and on
Ubuntu 18.04 (glibc 2.27), but it seems to be fixed on Ubuntu 18.10 (glibc
2.28)"

I must have used systems without this problem.

Post by Bize Ma
It is interesting that (finally) glibc 2.28 has added a fourth sort key
equal to the
Unicode code point. That forces the order of all characters to be unique.

One of the POSIX future directions.