[mapfile]: problem reentrance with normal file

[Didou Serge]

Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i686'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib -Wdate-time
-D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/bash-1ixwpb/bash-4.4=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall -no-pie
-Wno-parentheses -Wno-format-security
uname output: Linux MX36 4.10.0-42-generic #46-Ubuntu SMP Mon Dec 4
14:36:05 UTC 2017 i686 i686 i686 GNU/Linux
Machine Type: i686-pc-linux-gnu

Bash Version: 4.4
Patch Level: 7
Release Status: release

Description:
When use mapfile from normal file that call function that use mapfile, the
first mapfile lost data.

Repeat-By:
I create 3 functions:
ww(){ echo "$@" | mapfile -C yy -c 1 -t TT ; }
xx(){ mapfile -C yy -c 1 -t TT < <(echo "$@") ; }
yy(){ echo "$@" ;}

I create 1 normal file:
echo -e "Premiere ligne" A{1..45} "\nDeuxieme ligne" B{1..45} "\nTroisieme
ligne" C{1..45} >fic.txt

This work but I can't use array TT:
mapfile -C ww -c 1 UU <fic.txt
0 0 Premiere ligne A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15 A16
A17 A18 A19 A20 A21 A22 A23 A24 A25 A26 A27 A28 A29 A30 A31 A32 A33 A34 A35
A36 A37 A38 A39 A40 A41 A42 A43 A44 A45
0 1 Deuxieme ligne B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11 B12 B13 B14 B15 B16
B17 B18 B19 B20 B21 B22 B23 B24 B25 B26 B27 B28 B29 B30 B31 B32 B33 B34 B35
B36 B37 B38 B39 B40 B41 B42 B43 B44 B45
0 2 Troisieme ligne C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16
C17 C18 C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C29 C30 C31 C32 C33 C34 C35
C36 C37 C38 C39 C40 C41 C42 C43 C44 C45

This not work (lost data in second line and after) :
mapfile -C xx -c 1 UU <fic.txt
0 0 Premiere ligne A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15 A16
A17 A18 A19 A20 A21 A22 A23 A24 A25 A26 A27 A28 A29 A30 A31 A32 A33 A34 A35
A36 A37 A38 A39 A40 A41 A42 A43 A44 A45
0 1 B17 B18 B19 B20 B21 B22 B23 B24 B25 B26 B27 B28 B29 B30 B31 B32 B33
B34 B35 B36 B37 B38 B39 B40 B41 B42 B43 B44 B45
0 2 ligne C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18
C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C29 C30 C31 C32 C33 C34 C35 C36 C37
C38 C39 C40 C41 C42 C43 C44 C45

Regards.

[Greg Wooledge]

Post by Didou Serge
When use mapfile from normal file that call function that use mapfile, the
first mapfile lost data.

In this function, mapfile is being executed in a subshell. It will
read the input, and then when the subshell terminates, the array will
be discarded.

[Chet Ramey]

Post by Didou Serge
Bash Version: 4.4
Patch Level: 7
Release Status: release
When use mapfile from normal file that call function that use mapfile, the
first mapfile lost data.
echo -e "Premiere ligne" A{1..45} "\nDeuxieme ligne" B{1..45} "\nTroisieme
ligne" C{1..45} >fic.txt

[...]

Post by Didou Serge
mapfile -C xx -c 1 UU <fic.txt
0 0 Premiere ligne A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15 A16
A17 A18 A19 A20 A21 A22 A23 A24 A25 A26 A27 A28 A29 A30 A31 A32 A33 A34 A35
A36 A37 A38 A39 A40 A41 A42 A43 A44 A45
0 1 B17 B18 B19 B20 B21 B22 B23 B24 B25 B26 B27 B28 B29 B30 B31 B32 B33
B34 B35 B36 B37 B38 B39 B40 B41 B42 B43 B44 B45
0 2 ligne C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18
C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C29 C30 C31 C32 C33 C34 C35 C36 C37
C38 C39 C40 C41 C42 C43 C44 C45

This is a buffering issue caused by nested calls to mapfile in the same
process. The first call to mapfile reads a bufferful of data from the
input file, and the second mapfile (the one executed as part of the
callback) reads data into that same buffer. When the second call finishes,
and the first mapfile is called again, the remaining characters in the
buffer have already been discarded. It's not safe to call mapfile as part
of a callback to a mapfile invocation.

Chet

[Didou Serge]

Ok, in this case, this should not work either:

mapfile -C xx -c 1 UU < <( echo -e "Premiere ligne" A{1..45} "\nDeuxieme
ligne" B{1..45} "\nTroisieme ligne" C{1..45})

But, this form work fine.

Post by Didou Serge

Post by Didou Serge
Bash Version: 4.4
Patch Level: 7
Release Status: release
When use mapfile from normal file that call function that use mapfile,

the

Post by Didou Serge
first mapfile lost data.
echo -e "Premiere ligne" A{1..45} "\nDeuxieme ligne" B{1..45}

"\nTroisieme

Post by Didou Serge
ligne" C{1..45} >fic.txt

[...]

Post by Didou Serge
mapfile -C xx -c 1 UU <fic.txt
0 0 Premiere ligne A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15 A16
A17 A18 A19 A20 A21 A22 A23 A24 A25 A26 A27 A28 A29 A30 A31 A32 A33 A34

A35

Post by Didou Serge
A36 A37 A38 A39 A40 A41 A42 A43 A44 A45
0 1 B17 B18 B19 B20 B21 B22 B23 B24 B25 B26 B27 B28 B29 B30 B31 B32 B33
B34 B35 B36 B37 B38 B39 B40 B41 B42 B43 B44 B45
0 2 ligne C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18
C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C29 C30 C31 C32 C33 C34 C35 C36

C37

Post by Didou Serge
C38 C39 C40 C41 C42 C43 C44 C45

This is a buffering issue caused by nested calls to mapfile in the same
process. The first call to mapfile reads a bufferful of data from the
input file, and the second mapfile (the one executed as part of the
callback) reads data into that same buffer. When the second call finishes,
and the first mapfile is called again, the remaining characters in the
buffer have already been discarded. It's not safe to call mapfile as part
of a callback to a mapfile invocation.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates

[Didou Serge]

I tested with another configuration and ALL work fine:
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: cygwin
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash.exe' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='cygwin' -DCONF_MACHTYPE='x86_64-unknown-cygwin'
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -DRECYCLES_PIDS -I.
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4/include
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4/lib -DWORDEXP_OPTION -ggdb
-O2 -pipe -Wimplicit-function-declaration
-fdebug-prefix-map=/usr/src/bash-4.4.12-3.x86_64/build=/usr/src/debug/bash-4.4.12-3
-fdebug-prefix-map=/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4=/usr/src/debug/bash-4.4.12-3
-Wno-parentheses -Wno-format-security
uname output: CYGWIN_NT-6.1 EB-5CG63201SJ 2.8.0(0.309/5/3) 2017-04-01 20:47
x86_64 Cygwin
Machine Type: x86_64-unknown-cygwin

Bash Version: 4.4
Patch Level: 12
Release Status: release

Post by Didou Serge
mapfile -C xx -c 1 UU < <( echo -e "Premiere ligne" A{1..45} "\nDeuxieme
ligne" B{1..45} "\nTroisieme ligne" C{1..45})
But, this form work fine.

Post by Didou Serge

Post by Didou Serge
Bash Version: 4.4
Patch Level: 7
Release Status: release
When use mapfile from normal file that call function that use mapfile,

the

Post by Didou Serge
first mapfile lost data.
echo -e "Premiere ligne" A{1..45} "\nDeuxieme ligne" B{1..45}

"\nTroisieme

Post by Didou Serge
ligne" C{1..45} >fic.txt

[...]

Post by Didou Serge
mapfile -C xx -c 1 UU <fic.txt
0 0 Premiere ligne A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15

A16

Post by Didou Serge
A17 A18 A19 A20 A21 A22 A23 A24 A25 A26 A27 A28 A29 A30 A31 A32 A33 A34

A35

Post by Didou Serge
A36 A37 A38 A39 A40 A41 A42 A43 A44 A45
0 1 B17 B18 B19 B20 B21 B22 B23 B24 B25 B26 B27 B28 B29 B30 B31 B32 B33
B34 B35 B36 B37 B38 B39 B40 B41 B42 B43 B44 B45
0 2 ligne C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18
C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C29 C30 C31 C32 C33 C34 C35 C36

C37

Post by Didou Serge
C38 C39 C40 C41 C42 C43 C44 C45

This is a buffering issue caused by nested calls to mapfile in the same
process. The first call to mapfile reads a bufferful of data from the
input file, and the second mapfile (the one executed as part of the
callback) reads data into that same buffer. When the second call finishes,
and the first mapfile is called again, the remaining characters in the
buffer have already been discarded. It's not safe to call mapfile as part
of a callback to a mapfile invocation.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates

[Didou Serge]

But not work under linux...

Post by Didou Serge
Machine: x86_64
OS: cygwin
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash.exe' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='cygwin' -DCONF_MACHTYPE='x86_64-unknown-cygwin'
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -DRECYCLES_PIDS -I.
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4/include
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4/lib -DWORDEXP_OPTION -ggdb
-O2 -pipe -Wimplicit-function-declaration
-fdebug-prefix-map=/usr/src/bash-4.4.12-3.x86_64/build=/usr/src/debug/bash-4.4.12-3
-fdebug-prefix-map=/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4=/usr/src/debug/bash-4.4.12-3
-Wno-parentheses -Wno-format-security
uname output: CYGWIN_NT-6.1 EB-5CG63201SJ 2.8.0(0.309/5/3) 2017-04-01
20:47 x86_64 Cygwin
Machine Type: x86_64-unknown-cygwin
Bash Version: 4.4
Patch Level: 12
Release Status: release

Post by Didou Serge
mapfile -C xx -c 1 UU < <( echo -e "Premiere ligne" A{1..45}
"\nDeuxieme ligne" B{1..45} "\nTroisieme ligne" C{1..45})
But, this form work fine.

Post by Didou Serge

Post by Didou Serge
Bash Version: 4.4
Patch Level: 7
Release Status: release
When use mapfile from normal file that call function that use mapfile,

the

Post by Didou Serge
first mapfile lost data.
echo -e "Premiere ligne" A{1..45} "\nDeuxieme ligne" B{1..45}

"\nTroisieme

Post by Didou Serge
ligne" C{1..45} >fic.txt

[...]

Post by Didou Serge
mapfile -C xx -c 1 UU <fic.txt
0 0 Premiere ligne A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15

A16

Post by Didou Serge
A17 A18 A19 A20 A21 A22 A23 A24 A25 A26 A27 A28 A29 A30 A31 A32 A33

A34 A35

Post by Didou Serge
A36 A37 A38 A39 A40 A41 A42 A43 A44 A45
0 1 B17 B18 B19 B20 B21 B22 B23 B24 B25 B26 B27 B28 B29 B30 B31 B32

B33

Post by Didou Serge
B34 B35 B36 B37 B38 B39 B40 B41 B42 B43 B44 B45
0 2 ligne C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17

C18

Post by Didou Serge
C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C29 C30 C31 C32 C33 C34 C35

C36 C37

Post by Didou Serge
C38 C39 C40 C41 C42 C43 C44 C45

This is a buffering issue caused by nested calls to mapfile in the same
process. The first call to mapfile reads a bufferful of data from the
input file, and the second mapfile (the one executed as part of the
callback) reads data into that same buffer. When the second call finishes,
and the first mapfile is called again, the remaining characters in the
buffer have already been discarded. It's not safe to call mapfile as part
of a callback to a mapfile invocation.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
http://tiswww.cwru.edu/~chet/

[Chet Ramey]

Post by Didou Serge
But not work under linux...

It's probably the difference between using pipes and FIFOs for process
substitution and whether or not the OS makes the underlying file object
seekable.

[Didou Serge]

Ok, maybe one day we can do it ...

Thanks for your clarifications

Regards,
Serge.

Post by Chet Ramey

Post by Didou Serge
But not work under linux...

It's probably the difference between using pipes and FIFOs for process
substitution and whether or not the OS makes the underlying file object
seekable.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates

[Chet Ramey]

I don't use cygwin, so I can't verify what's happening. It's probably
that everything is line-buffered, but I can't say for sure.

[Chet Ramey]

mapfile -C xx -c 1 UU  < <( echo -e "Premiere ligne" A{1..45} "\nDeuxieme
ligne" B{1..45} "\nTroisieme ligne" C{1..45})
But, this form work fine.

Because the read function underlying mapfile notes that the file descriptor
is not seekable and reads only a single line at a time rather than a full
buffer of data.

Chet