Files sometimes come in (whether via hadoop or other processes) as big globs of data with inter-related parts. Many times I want to process these globs concurrently but see my dilemma unfolding quickly. I could a) write the code to process it serially and be done with it in 1 hour or b) write code to process it concurrently and be done in 1.5 hours because the added overhead of verifying the output, thread safety, etc exceeds the processing time serially. This made me sad, because concurrent processes are awesome. But self-managed thread safe concurrent processes are even more awesome!<\/p>\n

I thought, what if I could split an input file on keys and group those similarly keyed lines into separate files for processing. Aha!<\/p>\n

So I naturally first tried finding existing solutions and to be honest, awk has a pretty killer one liner as noted here<\/a> on Stack Overflow:<\/p>\n

awk '{ print >> $5 }' yourfile<\/pre>\n
This one liner is likely great for many folks (especially when using only small files). But for me, awk threw an error due to having “too many open files” – again, sad face :(.<\/p>\n
So… I wrote my own python command line utility to take an input file and split it into any number of output files by unique keys in the file. So all your input is maintained, just in different files sorted\/segregated on keys you provide.<\/p>\n
While my naming conventions may be lacking\u00a0panache, they are at least clearly intentioned utilities. (But seriously, if you have a better name, I’m all ears)<\/p>\n
Without further ado, I give you
\nsplit_file_by_key<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Files sometimes come in (whether via hadoop or other processes) as big globs of data with inter-related parts. Many times I want to process these globs concurrently but see my dilemma unfolding quickly. I could a) write the code to process it serially and be done with it in 1 hour or b) write code… Continue reading→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/91"}],"collection":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/comments?post=91"}],"version-history":[{"count":3,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/91\/revisions"}],"predecessor-version":[{"id":108,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/91\/revisions\/108"}],"wp:attachment":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/media?parent=91"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/categories?post=91"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/tags?post=91"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}