Sometimes we want parallel processing, but don’t want to pay the cost of ensuring proper multi-threaded handling. Because who wants to spend an extra 30 minutes setting up threads, ensuring race condition safety, et al just to save a few minutes?

If you have access to “xargs,” you have access to a built-in utility that can simplify parallel processing. Say you have a few hundred files you want to convert from json to csv, and it’s enough data to warrant parallel processes, but not enough to warrant the extra time spent building a truly parallel process. Do a simple “ls”, pipe it to “xargs -P8 -n1 python” and now you’ll have 8 separate processes working on 1 file each independently.


ls file.json_* | xargs -P8 -n1 python

xargs explained:
-P8 means we want 8 separate processes (so no Global Interpreter Lock issues)
-n1 means we want xargs to take the arguments and pass only one (file name) at a time as an argument to the python code

Split STDIN to multiple Compressed Files on highly skewed data ELI5 - Jelly Bean Analogy to MapReduce (Hadoop)

Leave a Reply

Your email address will not be published. Required fields are marked *