Gnu Parallel for Makefile goodness.

With projects I like to have a single make run to kick off the dev project. So when recently building a Vagrant box for a project I wanted my README Getting Started section to just contain:

vagrant up && vagrant ssh -c "make run" 

Now unfortunately, in my situation, make run needed to gracefully start and stop multiple processes. I am running a development app server, Webpack for the JavaScript and Gulp for the SASS (because Webpack sucks with SASS).

Now naively make run could just have done:

python manage.py runserver & gulp watch & webpack-dev-server

But when we Ctrl+C to close, it will only kill the last process as everything else was sent to the background. Not ideal.

Enter GNU parallel. installed with sudo apt-get install parallel on Ubuntu. The following command:

(echo python manage.py runserver; echo gulp watch; echo webpack-dev-server) | parallel

Will run all processes in parallel and then kills them all on a single Ctrl+C.

Perfect. Well almost. My host only had two CPUS, and parallel by default seems to execute at most ${CPU_COUNT} processes at a time. As I had N+1 processes it was waiting for a process to finish before kicking off the third and final process.

parallel -P 0 solved this issue, it tells parallel not to limit on CPU count and to just run everything at once.

Better. But I am still not getting anything in stdout.

By default parallel waits for a process to finish before dumping any output to stdout. For watch processes where you care about the output immediately and it will never finish, this is not ideal.

parallel --ungroup is what we’re missing. Instead of waiting and grouping a commands stdout together, --ungroup tells it to just write straight away and not wait for completion.

Almost there. But I am getting interspersed output on a per line basis.

What we’re missing is parallel --line-buffer, which buffers output from each process on a per line basis before writing to stdout.

So the final make run executes:

(echo python manage.py runserver; echo gulp watch; echo webpack-dev-server) | parallel -P 0 --line-buffer --ungroup

It gracefully starts and stops all 3 processes in a single shell.

My Getting Started section can now just read:

vagrant up && vagrant ssh -c "make run"

Simple.