Apache Spark and Parquet (SParquet) are a match made in scalable data analytics and delivery heaven. Spark brings a wide ranging, powerful computing platform to the equation while Parquet offers a data format that is purpose-built for high-speed big data analytics<\/strong>. If this sounds like fluffy marketing talk, resist the temptation to close this tab, because what follows are substantial insights I’ve personally procured and am sharing here to help others get the most out of Parquet and Spark.<\/p>\n What is Parquet? <\/strong><\/p>\n Parquet is a binary compressed columnar file format available to any project in the Hadoop ecosystem (and others outside it even). It’s a mouthful, but let’s break it down.<\/p>\n Binary <\/em>means parquet files cannot be opened by typical text editors natively (sublime text*, vim, etc).<\/p>\n * My former colleague James Yu wrote a Sublime Text plugin you can find here<\/a> to view parquet files.<\/p>\n Columnar<\/em> means the data is stored as columns instead of rows as most traditional databases (MySQL, PostgreSQL, etc) and file formats (CSV, JSON, etc). This is going to be very important.<\/p>\n Compressed<\/em> means the file footprint on disk (HDFS, S3, or local filesystem) is smaller than a typical raw uncompressed file. Parquet handles compression differently than traditional compression of a CSV file for example, but in a similar vein to Avro.<\/p>\n Now that the basic definition is out of the way, let’s get right to it.<\/p>\n How can Parquet help me?<\/strong><\/p>\n Parquet is exceptionally good at high-speed big data analytics<\/strong>. It can store vast quantities of data and operate on it more quickly than many other solutions. Let’s say you have CSV files in S3 (20 TB and growing by 250GiB per day) and a use case that necessitates reporting on those files in a dashboard. A common approach to this problem is aggregating the CSV files down to a MySQL-friendly size, so that reports can be built on this aggregated data. However, this is limited in multiple ways:<\/p>\n If those limitations had you cringing, I’ve made my case well :). There is an alternative that utilizes SParquet…<\/p>\n CSV File for Proof of Concept (PoC): NYC TLC Green Taxi for December 2016<\/a><\/p>\n The CSV file has 1,224,160 rows and 19 columns, coming in at 107MB uncompressed. Here’s the file schema (using header and inferSchema options in Spark 2.1.1):<\/p>\n Uncompressed CSV of 107MB was reduced to 24MB (Snappy Parquet) and 19MB (GZIP Parquet). But the real power comes in once the data (now in parquet format) is accessed. Parquet is exceptionally fast when accessing specific columns, which is the opposite of row-based file formats, which thrive when accessing an entire row record. Here are simple SQL examples to show the differences:<\/p>\n #3 above is a great example of where Parquet shines, because you’re using pushdown filtering, operating on only specific columns (the rest are ignored), and do not have to parse what you don’t care about (all the other columns\/rows).<\/p>\n What implementation strategies can I use?<\/strong><\/p>\n Some ideas:<\/p>\n By now, you have hopefully learned that Parquet is a powerful data format that facilitates big data analytics at a scale far greater than many traditional limited approaches. Go forth and play with Parquet!<\/p>\n Here’s my blog post for specific optimization tips: http:\/\/garrens.com\/blog\/2017\/04\/08\/getting-started-and-tips-for-using-apache-parquet-with-apache-spark-2-x\/<\/a><\/p>\n Garren Staubli <\/em>is a Big Data Engineer Consultant at Blueprint Consulting Services, and formerly a big data engineer at iSpot.TV.<\/p>\n<\/div>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Apache Spark and Parquet (SParquet) are a match made in scalable data analytics and delivery heaven. Spark brings a wide ranging, powerful computing platform to the equation while Parquet offers a data format that is purpose-built for high-speed big data analytics. If this sounds like fluffy marketing talk, resist the temptation to close this tab,… Continue reading→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[22],"tags":[6,17,25,24,19,2],"_links":{"self":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/141"}],"collection":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/comments?post=141"}],"version-history":[{"count":6,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/141\/revisions"}],"predecessor-version":[{"id":149,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/141\/revisions\/149"}],"wp:attachment":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/media?parent=141"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/categories?post=141"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/tags?post=141"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"http:\/\/garrens.com\/blog\/wp-content\/uploads\/2017\/06\/spark-logo-trademark.png\"){kind=logo}
<\/a><\/p>\n<\/a><\/p>\n\n
\n
|-- VendorID: integer (nullable = true)\r\n |-- lpep_pickup_datetime: timestamp (nullable = true)\r\n |-- lpep_dropoff_datetime: timestamp (nullable = true)\r\n |-- store_and_fwd_flag: string (nullable = true)\r\n |-- RatecodeID: integer (nullable = true)\r\n |-- PULocationID: integer (nullable = true)\r\n |-- DOLocationID: integer (nullable = true)\r\n |-- passenger_count: integer (nullable = true)\r\n |-- trip_distance: nullable = true)\r\n |-- fare_amount: nullable = true)\r\n |-- extra: nullable = true)\r\n |-- mta_tax: nullable = true)\r\n |-- tip_amount: nullable = true)\r\n |-- tolls_amount: nullable = true)\r\n |-- ehail_fee: string (nullable = true)\r\n |-- improvement_surcharge: nullable = true)\r\n |-- total_amount: nullable = true)\r\n |-- payment_type: integer (nullable = true)\r\n |-- trip_type: integer (nullable = true)\r\n<\/pre>\n
--#1\r\n--CSV will read the entire file row-by-row\r\n--Parquet will dump the rows based on their column values\r\n--Winner: Parquet (minimal; because of no parsing)\r\nSELECT *\r\ngreen_tlc\r\n\r\n--#2\r\n--CSV will read the entire file row-by-row\r\n--filter the PULocation column to only ones containing 226 \r\n--and output all rows\/columns that match the filter criteria as the results\r\n--Winner: Parquet (minimal; because of no parsing and push down filtering)\r\nSELECT *\r\ngreen_tlc\r\nWHERE PULocation = 226\r\n\r\n--#3\r\n--Parquet will first find only the relevant \"data blocks\" based on the filter criteria\r\n--and only aggregate the rows\/columns that match the filter criteria\r\n--Winner: Parquet (huge; because of no parsing and only specific columns)\r\nSELECT PULocation, SUM(total_amount)\r\ngreen_tlc\r\nWHERE PULocation IN (77, 102, 107, 226)\r\nGROUP BY PULocation\r\n<\/pre>\n\n