Running tasks in parallel batches in Bash

I had a requirement to run quite a lot of tasks in parallel with varying parameters. Initial experimentation suggested I might end up with a lot of processes running and potentially cause system issues so I looked into creating a script to run things in parallel, albeit in controlled batches.

In this example, I’ve substituted the actual actions I was taking with a random sleep command so that processes will finish at different times. What would probably be best would be to have the actions in another script that will log its output somewhere either by writing to a file or by using logger so syslog deals with the flurry. Typically unix file writes are atomic up to 4KB so having several processes writing at the same time isn’t a huge issue.

Bash below:

# Loop through items in word list to run actions
# and process in parallel batches to avoid having
# too many processes.
# v1.0 7/5/2019
USAGE="`basename $0` /path/to/wordlist {batch size}"
if [ ! $BATCHSIZE ]; then

if [ ! $1 ] || [ ! $2 ]; then
    echo "$USAGE"
    exit 1

echo "Using wordlist $WORDLIST in batches of $BATCHSIZE"
for word in `cat $WORDLIST`; do
    if [ $(( $i % $BATCHSIZE )) -eq 0 ] && [ $i -ne 0 ]; then
       echo "Batch of $BATCHSIZE done... waiting"

    # Take actions here and run as background processes
    SLEEPRND=`echo $(( $RANDOM % 9 + 1 ))`
    echo "Action: $word - Sleeping for $SLEEPRND"
    sleep $SLEEPRND &

    # Increment counter for tracking
    i=$(( $i + 1 ))
printf "Waiting..."
printf "all jobs run.\n"


[me@server ~]$ ./ wordlist localhost 5
Using wordlist wordlist against host localhost in batches of 5
Action: a - Sleeping for 2
Action: b - Sleeping for 3
Action: c - Sleeping for 2
Action: d - Sleeping for 2
Action: e - Sleeping for 5
Batch of 5 done... waiting
Action: f - Sleeping for 7
Action: g - Sleeping for 1
Action: h - Sleeping for 1
Action: i - Sleeping for 1
Action: j - Sleeping for 8
Batch of 5 done... waiting
Action: k - Sleeping for 2
Action: l - Sleeping for 1
Waiting...All jobs run.
Tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *