This page is dedicated to resources related to the 7\/29\/17 Understanding Spark event presentation in Bellevue, WA.<\/p>\n

Slides<\/a><\/p>\n

Great [FREE!] resources on all things Spark:
\nhttps:\/\/jaceklaskowski.gitbooks.io\/mastering-apache-spark\/<\/a>
\nhttps:\/\/spark.apache.org\/docs\/latest\/sql-programming-guide.html<\/a><\/p>\n

Databricks was founded by the original creators of Spark and is currently the largest contributor to Apache Spark. As such, they are a phenomenal resource for information and services relating to Spark.<\/p>\n

Datasets: https:\/\/databricks.com\/blog\/2016\/01\/04\/introducing-apache-spark-datasets.html<\/a>
\nCatalyst: https:\/\/www.slideshare.net\/databricks\/a-deep-dive-into-spark-sqls-catalyst-optimizer-with-yin-huai<\/a>
\nhttps:\/\/de.slideshare.net\/SparkSummit\/deep-dive-into-catalyst-apache-spark-20s-optimizer-63071120<\/a>
\nTungsten: https:\/\/databricks.com\/blog\/2015\/04\/28\/project-tungsten-bringing-spark-closer-to-bare-metal.html<\/a>
\nMatrix: https:\/\/databricks.com\/blog\/2016\/07\/14\/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html<\/a><\/p>\n

Personally curated Examples:<\/p>\n

Create mock typed object data<\/strong>
\nimport org.apache.spark.sql.functions._<\/code><\/p>\n

case class CountryGDP(countryCode : String, countryName : String, Year : String, gdp: Double, language : Option[String])
\nval objects = Seq[CountryGDP](
\nCountryGDP(“USA”, “‘Murica”, “2014”, 17393103000000f, None),
\nCountryGDP(“USA”, “‘Murica”, “2015”, 18036648000000f, None),
\nCountryGDP(“USA”, “‘Murica”, “2016”, 18569100000000f, None),
\nCountryGDP(“CHE”, “Switzerland”, “2014”, 702705544908.583, None),
\nCountryGDP(“CHE”, “Switzerland”, “2015”, 670789928809.882, None),
\nCountryGDP(“CHE”, “Switzerland”, “2016”, 659827235193.83, None)
\n)<\/p>\n

Strongly typed Datasets<\/strong>
\nval objectsDS = spark.createDataset(objects)<\/code><\/p>\n

\/\/ typed objects are evaluated at compile time (great for development in IDEs!)
\nval countriesWithLanguages = objectsDS.map(o => {
\nval lang = o.countryCode match {
\ncase “USA” => Some(“English”)
\ncase “CHE” => Some(“Schweizerdeutsch”)
\ncase _ => Some(“Simlish”)
\n}
\no.copy(language = lang)
\n})<\/p>\n

Creating DataFrame and using UDF to transform<\/strong>
\nval rowsDF = spark.createDataFrame(objects)<\/code><\/p>\n

def getLang(countryCode: String): Option[String] = {
\ncountryCode match {
\ncase “USA” => Some(“English”)
\ncase “CHE” => Some(“Schweizerdeutsch”)
\ncase _ => Some(“Simlish”)
\n}
\n}
\nval gl = sqlContext.udf.register(“getLang”, getLang _)<\/p>\n

\/\/ String-based lookups are evaluated at Runtime
\nval rowsDFWithLanguage = rowsDF.withColumn(“language”, gl($”countryCode”))<\/p>\n

Event link: https:\/\/www.eventbrite.com\/e\/understanding-spark-tickets-35440866586#<\/a><\/p>\n

Video recording is here: https:\/\/livestream.com\/metis\/events\/7597562<\/a><\/p>\n

Learn more about parquet<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

This page is dedicated to resources related to the 7\/29\/17 Understanding Spark event presentation in Bellevue, WA. Slides Great [FREE!] resources on all things Spark: https:\/\/jaceklaskowski.gitbooks.io\/mastering-apache-spark\/ https:\/\/spark.apache.org\/docs\/latest\/sql-programming-guide.html Databricks was founded by the original creators of Spark and is currently the largest contributor to Apache Spark. As such, they are a phenomenal resource for information and… Continue reading→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[22],"tags":[17,2],"_links":{"self":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/158"}],"collection":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/comments?post=158"}],"version-history":[{"count":7,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/158\/revisions"}],"predecessor-version":[{"id":225,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/158\/revisions\/225"}],"wp:attachment":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/media?parent=158"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/categories?post=158"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/tags?post=158"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}