Strange behaviour from jobs -p in a subshell

[Christopher Jefferson]

Consider the following script. While the 3 sleeps are running, both jobs
-p and $(jobs -p) will print 3 PIDs. Once the 3 children are finished,
jobs -p will continue to print the 3 PIDs of the done Children, but
$(jobs -p) will only print 1 PID. $(jobs -p) always seems to print at
most 1 PID of a done child.


#!/usr/bin/bash

(sleep 2 ) &
(sleep 2 ) &
(sleep 2 ) &

while /bin/true
do
    echo A
    echo $(jobs -p)
    echo B
    jobs -p
    echo C
    sleep 1

[Chet Ramey]

Post by Christopher Jefferson
Consider the following script. While the 3 sleeps are running, both jobs
-p and $(jobs -p) will print 3 PIDs. Once the 3 children are finished,
jobs -p will continue to print the 3 PIDs of the done Children, but
$(jobs -p) will only print 1 PID. $(jobs -p) always seems to print at
most 1 PID of a done child.

Since the $(jobs -p) is run in a subshell, its knowledge of its parent's
jobs is transient. In this case, the subshell deletes knowledge of the
jobs it inherits from its parent, but hangs onto the last asynchronous job
in case the subshell references $!.

Chet

[Greg Wooledge]

Post by Chet Ramey

Post by Christopher Jefferson
Consider the following script. While the 3 sleeps are running, both jobs
-p and $(jobs -p) will print 3 PIDs. Once the 3 children are finished,
jobs -p will continue to print the 3 PIDs of the done Children, but
$(jobs -p) will only print 1 PID. $(jobs -p) always seems to print at
most 1 PID of a done child.

Since the $(jobs -p) is run in a subshell, its knowledge of its parent's
jobs is transient. In this case, the subshell deletes knowledge of the
jobs it inherits from its parent, but hangs onto the last asynchronous job
in case the subshell references $!.
Chet

If the goal is to obtain the result of "jobs -p" and use it in a script,
I would suggest redirecting the output of jobs -p to a temp file, then
reading it. That skips the subshell.

[Christopher Jefferson]

Post by Chet Ramey

Post by Christopher Jefferson
Consider the following script. While the 3 sleeps are running, both jobs
-p and $(jobs -p) will print 3 PIDs. Once the 3 children are finished,
jobs -p will continue to print the 3 PIDs of the done Children, but
$(jobs -p) will only print 1 PID. $(jobs -p) always seems to print at
most 1 PID of a done child.

Since the $(jobs -p) is run in a subshell, its knowledge of its parent's
jobs is transient. In this case, the subshell deletes knowledge of the
jobs it inherits from its parent, but hangs onto the last asynchronous job
in case the subshell references $!.

Is this a case of "works as intended" then? I find the current behaviour
very strange -- I could understand if 'jobs -p' showed no information
about processes spawned from the parent shell, or all of it, but the
current position seems quite inconsistent.

This originally came up as I was implementing a poor way of limiting how
many background processes I spawn, by doing:

    while (( $(jobs -p | wc -l) >= $JOBCOUNT ))
    do
            sleep 1
    done

Here, the $(jobs -p | wc -l) decreases to 1 as jobs finish, but never
reaches 0.


I've now changed 'sleep 1' to 'jobs > /dev/null; sleep 1'.


I can't find any documentation that says this, but it seems 'jobs' will
clean up children which are done, which 'jobs -p' does not (this makes
sense of course, as jobs -p doesn't report if a child is done, but I
still can't find it documente

[Chet Ramey]

Post by Christopher Jefferson

Post by Chet Ramey

Post by Christopher Jefferson
Consider the following script. While the 3 sleeps are running, both jobs
-p and $(jobs -p) will print 3 PIDs. Once the 3 children are finished,
jobs -p will continue to print the 3 PIDs of the done Children, but
$(jobs -p) will only print 1 PID. $(jobs -p) always seems to print at
most 1 PID of a done child.

Since the $(jobs -p) is run in a subshell, its knowledge of its parent's
jobs is transient. In this case, the subshell deletes knowledge of the
jobs it inherits from its parent, but hangs onto the last asynchronous job
in case the subshell references $!.

Is this a case of "works as intended" then? I find the current behaviour
very strange -- I could understand if 'jobs -p' showed no information
about processes spawned from the parent shell, or all of it, but the
current position seems quite inconsistent.

Why? The shell has to keep the value and status of $! because POSIX says
you can always request it. A subshell inherits that pid, so it's `known'
to the subshell, and it will stick around until you get notfied of the
status. You don't restrict the `jobs -p' output to running jobs, so it's
going to show up, and you don't request the status of the job (even jobs
or jobs -l would be sufficient) so it's not going to go away.

[Greg Wooledge]

Post by Christopher Jefferson
Consider the following script. While the 3 sleeps are running, both jobs
-p and $(jobs -p) will print 3 PIDs. Once the 3 children are finished,

[...]

... hey, I think I just figured out the GOAL!

You want to run a whole bunch of jobs in parallel, but only 3 at a
time. Right? There are some solutions for that at
<https://mywiki.wooledge.org/ProcessManagement> (Advanced questions #4).

Our bot in IRC also suggests this example:

parallel() {
local workers=$1 handler=$2 w i
shift 2
local elements=("$@")
for (( w = 0; w < workers; ++w )); do
for (( i = w; i < ${#elements[@]}; i += workers )); do
"$handler" "${elements[i]}"
done &
done
wait
}
parallel 5 md5 *.txt

Some of the examples on the wiki page have a stricter guarantee of
maintaining the number of jobs when their durations are unpredictable,
but I include the above example for completeness. Choose whichever
solution you like best.