How to run shell commands in parallel while limiting their concurrency.
I had a use case recently where I wanted to speed up some Bash processing that I was doing, but still keep concurrency constrained. So I wondered: is there a straightforward way to create thread pools in Bash? The answer is: yes, but it takes some work to be semi-resilient and portable.
You may ask: why do this in Bash at all? It seems silly to use a language with poor concurrency controls. You'd be right to ask this question, and my answer is: because I have a fondness for Bash, and I wanted the challenge. But you're still probably better off using another language, such as Python.
Here is code that creates a command queue that accepts shell commands to execute. You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.
1
2
3queue_start() {
4
5 if [[ ! ${1:-} =~ ^[1-9][0-9]*$ ]]; then
6 echo "queue_start: size must be a positive integer, got '${1:-}'" >&2
7 return 1
8 fi
9
10
11
12 if [[ -n ${_queue_fd:-} ]]; then
13 eval "exec ${_queue_fd}>&-"
14 unset _queue_fd _queue_pid
15 fi
16
17
18 local fd=3
19 while { : <&"${fd}"; } 2>/dev/null; do fd=$(( fd + 1 )); done
20 if [[ ${fd} -gt 9 ]]; then
21 echo "queue_start: no free fd from 3 to 9" >&2
22 return 1
23 fi
24
25
26 local dir
27 dir=$(mktemp -d "${TMPDIR:-/tmp}/queue.XXXXXX")
28 mkfifo "${dir}/queue"
29
30
31 local shell
32 shell=$(ps -o comm= -p "$$")
33 shell=${shell#-}
34
35
36
37
38
39 eval "exec ${fd}>&2"
40 local opts=$-
41 set +m
42 {
43
44 xargs -0 -n 1 -P "$1" "${shell}" -c '
45 # Output is captured rather than streamed so that command output stays separate
46 out=$(eval -- "$1" </dev/null 2>&1); rc=$?
47 [ -z "${out}" ] || printf "%s\n" "${out}"
48 # Coerce non-zero exit codes to 1, because a 255 code would stop xargs
49 [ "${rc}" -eq 0 ] || { printf "exit %d: %s\n" "${rc}" "$1" >&2; exit 1;
50 }' _ <"${dir}/queue" 2>&"${fd}" &
51 _queue_pid=$!
52 disown || :
53 } 2>/dev/null
54 if [[ ${opts} == *m* ]]; then set -m; fi
55
56
57
58 eval "exec ${fd}>\"\${dir}/queue\""
59 _queue_fd=${fd}
60 rm -rf "${dir}"
61
62
63 setopt localoptions posixtraps 2>/dev/null || :
64 trap _queue_stop EXIT
65 if [[ $- != *i* ]]; then
66 trap 'exit 129' HUP
67 trap 'exit 130' INT
68 trap 'exit 143' TERM
69 fi
70}
71
72
73
74queue_submit() {
75 if [[ -z ${_queue_fd:-} ]]; then
76 echo "queue_submit: no command queue started in this shell" >&2
77 return 1
78 fi
79
80
81 if [[ $- == *i* ]]; then
82
83
84 env printf '%s\0' "$1" >&"${_queue_fd}"
85 else
86
87 env printf '%s\0' "$1" >&"${_queue_fd}" &
88 wait "$!"
89 fi || {
90 echo "queue_submit: couldn't queue the command; the command queue may have stopped" >&2
91 return 1
92 }
93}
94
95
96queue_wait() {
97 if [[ -z ${_queue_fd:-} ]]; then
98 echo "queue_wait: no command queue started in this shell" >&2
99 return 1
100 fi
101 eval "exec ${_queue_fd}>&-"
102 while kill -0 "${_queue_pid}" 2>/dev/null; do sleep 0.1; done
103 unset _queue_fd _queue_pid
104}
105
106
107
108_queue_stop() {
109
110
111 local parent=${1:-$(exec sh -c 'echo "${PPID}"')}
112 local pid
113 for pid in $(exec pgrep -P "${parent}"); do
114
115 kill -STOP "${pid}" 2>/dev/null || continue
116
117 _queue_stop "${pid}"
118 kill -TERM "${pid}" 2>/dev/null || :
119
120 kill -CONT "${pid}" 2>/dev/null || :
121 done
122}
123
124
125if [[ $- != *i* ]]; then
126 trap _queue_stop EXIT
127 trap 'exit 129' HUP
128 trap 'exit 130' INT
129 trap 'exit 143' TERM
130fi
queue_start initializes the queue (managed by mkfifo) and starts the executor (concurrency managed by xargs -P), queue_submit submits commands to the queue, and queue_wait waits for the queue to drain before cleaning up. Calling queue_wait is optional, command processing will start as soon as you queue it with queue_submit.
You can copy and paste this code into a script, or it's also safe to put in your dotfiles, as it only sets some of the exit traps for interactive shells. It works in Bash and Zsh, and it has as much defensive programming as I know how to add, but I'm confident it has failure modes that are difficult to predict or avoid.
Example usage
My original use case had to do with making a large number of web requests, so here's an example using Postman Echo :
$ queue_start 3
$ for n in 1 2 3 4 5; do
queue_submit "set -o pipefail; \
curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \
-d '{\"request\": ${n}}' \
https://postman-echo.com/post | \
jq -c .json"
done
$ queue_wait
{"request":1}
{"request":2}
{"request":3}
{"request":4}
{"request":5}A more deterministic scenario demonstrating that concurrency is respected, only starting command three after command two has completed:
$ queue_start 2
$ queue_submit 'sleep 1; echo first'
$ queue_submit 'echo second'
$ queue_submit 'echo third'
$ queue_wait
second
third
firstCommand queues are inherited by subshells:
$ queue_start 2
$ ( queue_submit 'echo "submitted from a subshell"' )
$ queue_submit 'echo "submitted from the main shell"'
$ queue_wait
submitted from a subshell
submitted from the main shellDifferent subshells can have their own command queue:
$ (
queue_start 2
for n in 1 2 3 4; do queue_submit "sleep 1; echo first queue, job ${n}"; done
queue_wait
) &
$ first=$!
$ (
queue_start 1
for n in 1 2; do queue_submit "sleep 1; echo second queue, job ${n}"; done
queue_wait
) &
$ second=$!
$ wait "${first}"
$ wait "${second}"
second queue, job 1
first queue, job 1
first queue, job 2
second queue, job 2
first queue, job 3
first queue, job 4Queueing each line of a command's output as a command to execute:
$ queue_start 2
$ while IFS= read -r line; do
queue_submit "${line}"
done < <(printf 'echo "from command output, line %d"\n' 1 2 3)
$ queue_wait
from command output, line 1
from command output, line 2
from command output, line 3Queueing each line of a file as a command:
$ queue_start 2
$ while IFS= read -r line; do
queue_submit "${line}"
done < commands.txt
$ queue_waitKnown issues
Here are some reasons why you might just want a different programming language:
A shell can only have one queue.
The amount of additional code needed to support multiple is complicated and likely isn't needed in most cases. Subshells can still have their own queues.
Command output is intentionally not streamed.
This prevents interleaving the output of most commands. Output larger than
PIPE_BUFmay still get interleaved.queue_waitdoesn't return a value indicating any job failure.**This would require a temporary file or a second file descriptor and some complicated code.
queue_waithas to wait for running processes started afterqueue_start.Processes started after
queue_startwill inherit the queue write file descriptor (which is what allows subshells to work), andxargswon't see the end of the queue until every copy of the write descriptor is closed._queue_stopindiscriminately terminates descendants.Any other backgrounded jobs will be terminated, including ones started before
queue_start.
Why not just xargs -P?
If you know all commands that you want to execute up-front, then the solution gets far simpler:
$ printf 'echo "from command output, line %d"\0' 1 2 3 | \
xargs -0 -n 1 -P 2 bash -c
from command output, line 1
from command output, line 2
from command output, line 3or
$ tr '\n' '\0' < commands.txt | \
xargs -0 -n 1 -P 2 bash -cGNU parallel works similarly, though macOS doesn't ship with it:
$ printf 'echo "from command output, line %d"\n' 1 2 3 | \
parallel -j 2
from command output, line 1
from command output, line 2
from command output, line 3