Deprecated: Creation of dynamic property wpdb::$categories is deprecated in /home/garrens3/public_html/blog/wp-includes/wp-db.php on line 760

Deprecated: Creation of dynamic property wpdb::$post2cat is deprecated in /home/garrens3/public_html/blog/wp-includes/wp-db.php on line 760

Deprecated: Creation of dynamic property wpdb::$link2cat is deprecated in /home/garrens3/public_html/blog/wp-includes/wp-db.php on line 760

Deprecated: Using ${var} in strings is deprecated, use {$var} instead in /home/garrens3/public_html/blog/wp-includes/comment-template.php on line 1747

Deprecated: Optional parameter $term_id declared before required parameter $meta_value is implicitly treated as a required parameter in /home/garrens3/public_html/blog/wp-content/plugins/advanced-code-editor/advanced-code-editor.php on line 1927

Deprecated: Optional parameter $term_id declared before required parameter $meta_value is implicitly treated as a required parameter in /home/garrens3/public_html/blog/wp-content/plugins/advanced-code-editor/advanced-code-editor.php on line 1941

Deprecated: Optional parameter $term_id declared before required parameter $meta_key is implicitly treated as a required parameter in /home/garrens3/public_html/blog/wp-content/plugins/advanced-code-editor/advanced-code-editor.php on line 1956

Deprecated: Optional parameter $term_id declared before required parameter $key is implicitly treated as a required parameter in /home/garrens3/public_html/blog/wp-content/plugins/advanced-code-editor/advanced-code-editor.php on line 1970

Deprecated: Automatic conversion of false to array is deprecated in /home/garrens3/public_html/blog/wp-content/plugins/loginizer/init.php on line 250

Deprecated: Automatic conversion of false to array is deprecated in /home/garrens3/public_html/blog/wp-content/plugins/loginizer/init.php on line 265

Deprecated: Creation of dynamic property WP_Block_Type::$skip_inner_blocks is deprecated in /home/garrens3/public_html/blog/wp-includes/class-wp-block-type.php on line 391

Deprecated: Creation of dynamic property WP_Block_Type::$skip_inner_blocks is deprecated in /home/garrens3/public_html/blog/wp-includes/class-wp-block-type.php on line 391
{"id":114,"date":"2017-04-08T19:21:34","date_gmt":"2017-04-09T03:21:34","guid":{"rendered":"http:\/\/garrens.com\/blog\/?p=114"},"modified":"2018-03-02T20:52:13","modified_gmt":"2018-03-03T04:52:13","slug":"getting-started-and-tips-for-using-apache-parquet-with-apache-spark-2-x","status":"publish","type":"post","link":"https:\/\/garrens.com\/blog\/2017\/04\/08\/getting-started-and-tips-for-using-apache-parquet-with-apache-spark-2-x\/","title":{"rendered":"Tips for using Apache Parquet with Spark 2.x"},"content":{"rendered":"

What is Apache Parquet?<\/strong>
\nIt is a compressable binary columnar data format used in the hadoop ecosystem. We’ll talk about it primarily with relation to the Hadoop Distributed File System (HDFS) and Spark 2.x contexts.<\/p>\n

What role does it fill?<\/strong>
\nIt is a fast and efficient data format great for scalable big data analytics. <\/p>\n

Optimization Tips<\/strong><\/p>\n

  • Aim for around 1GB parquet output files, but experiment with other sizes for your use case and cluster setup (source<\/a>) <\/li>\n
  • Ideally store on HDFS in file sizes of at least the HDFS block size (default 128MB)<\/li>\n
  • Storing Parquet files on S3 is also possible (side note: use amazon athena, which charges based on data read if you want Presto SQL-like queries on demand at low cost)<\/li>\n
  • Use snappy compression if storage space is not a concern due to it being splittable, but for what should be a relatively small performance hit but much better compression, use gzip (source<\/a>)<\/li>\n

    <\/p>\n

    More information: <\/strong>
    \n    https:\/\/parquet.apache.org\/<\/a>
    \n    http:\/\/stackoverflow.com\/questions\/42918663\/is-it-better-to-have-one-large-parquet-file-or-lots-of-smaller-parquet-files\/42919015#42919015<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

    What is Apache Parquet? It is a compressable binary columnar data format used in the hadoop ecosystem. We’ll talk about it primarily with relation to the Hadoop Distributed File System (HDFS) and Spark 2.x contexts. What role does it fill? It is a fast and efficient data format great for scalable big data analytics. Optimization… Continue reading→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[22],"tags":[19,2],"_links":{"self":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/114"}],"collection":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/comments?post=114"}],"version-history":[{"count":11,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/114\/revisions"}],"predecessor-version":[{"id":125,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/114\/revisions\/125"}],"wp:attachment":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/media?parent=114"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/categories?post=114"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/tags?post=114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}