java.io.IOException: Split metadata size exceeded 10000000.<\/code> was the error I got when trying to process ~20TB of highly compressed logs (~100TB uncompressed) on my 64 node Amazon EMR cluster. Naturally I found some good resources recommending a quick file by modifying the mapred-site.xml file in \/home\/hadoop\/conf\/<\/p>\n

Warning: By setting this configuration to -1, you are effectively removing the limit on the splits’ metadata. Be aware this may cause unintended consequences should your cluster not have the resources to handle the actual job.<\/p>\n

\r\n\r\n\r\nmapreduce.jobtracker.split.metainfo.maxsize<\/name>\r\n    -1<\/value>\r\n<\/property><\/pre>\n
Then since I was modifying an existing running EMR cluster on Hadoop 1.0.4, I needed to restart my jobtracker:
\nsudo service --status-all<\/code>
\nsudo service hadoop-jobtracker restart<\/code><\/p>\n
Restart Yarn ResourceManager for running EMR cluster on Hadoop 2.4.0:
\nsudo service yarn-resourcemanager stop<\/code><\/p>\n
Source post: http:\/\/blog.dongjinleekr.com\/my-hadoop-job-crashes-with-split-metadata-size-exceeded\/<\/p>\n","protected":false},"excerpt":{"rendered":"
java.io.IOException: Split metadata size exceeded 10000000. was the error I got when trying to process ~20TB of highly compressed logs (~100TB uncompressed) on my 64 node Amazon EMR cluster. Naturally I found some good resources recommending a quick file by modifying the mapred-site.xml file in \/home\/hadoop\/conf\/ Warning: By setting this configuration to -1, you are… Continue reading→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/36"}],"collection":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/comments?post=36"}],"version-history":[{"count":4,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/36\/revisions"}],"predecessor-version":[{"id":42,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/36\/revisions\/42"}],"wp:attachment":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/media?parent=36"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/categories?post=36"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/tags?post=36"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}