Okay Okay, I may have oversold it a bit, but here are less than 10 bash lines that resemble (if you squint really<\/strong><\/em> hard) Hadoop\/MapReduce<\/a>.<\/p>\n What will this do? Split the in_file into 100,000 line chunks with an underscore and numbers following (e.g. in_file = “file.tsv”, temp files file.tsv_00000, file.tsv_00001, etc)<\/p>\n Get a list of all temp numbered in files, pass into xargs to run 8 processes of your code_to_run executable passing in the chunked in_file and outputting a chunked out_file.<\/p>\n Then cat chunked out files into single out file as you expect<\/p>\n Cleanup (re: remove) all temporary files; both in and out temporary files will be removed.<\/p>\n For the sake of data safety, we include “&&” following each line to ensure all subsequent commands are not run unless the prior conditions are met.<\/p>\n","protected":false},"excerpt":{"rendered":" Okay Okay, I may have oversold it a bit, but here are less than 10 bash lines that resemble (if you squint really hard) Hadoop\/MapReduce. code_to_run=$1 in_file=$2 out_file=$3 split -d -a 5 -l 100000 $in_file $in_file”_” && \\ ls $in_file”_”* | xargs -P8 -n1 -I file $code_to_run file file.out && \\ cat $in_file”_”*.out > $out_file… Continue readingcode_to_run=$1\r\nin_file=$2\r\nout_file=$3\r\nsplit -d -a 5 -l 100000 $in_file $in_file\"_\" && \\\r\nls $in_file\"_\"* | xargs -P8 -n1 -I file $code_to_run file file.out && \\\r\ncat $in_file\"_\"*.out > $out_file && \\\r\nrm $in_file\"_\"*\r\n<\/pre>\n
\nTakes 3 args<\/p>\n\n
split -d -a 5 -l 100000 $in_file $in_file\"_\"<\/pre>\n
ls $in_file\"_\"* | xargs -P8 -n1 -I file $code_to_run file file.out<\/pre>\n
cat $in_file\"_\"*.out > $out_file<\/pre>\n
rm $in_file\"_\"*\r\n<\/pre>\n