Switching between Scala and Python on Spark tips

Posted by Garren on

Switching between Scala and Python on Spark is relatively straightforward, but there are a few differences that can cause some minor frustration. Here are some of the little things I’ve run into and how to adjust for them. PySpark Shell does not support code completion (autocomplete) by default. Why? PySpark uses the basic Python interpreter… Continue reading→

Categories
Apache Spark
Tags
, , , Leave a Comment on Switching between Scala and Python on Spark tips

Real Time Big Data analytics: Parquet (and Spark) + bonus

Posted by Garren on

Apache Spark and Parquet (SParquet) are a match made in scalable data analytics and delivery heaven. Spark brings a wide ranging, powerful computing platform to the equation while Parquet offers a data format that is purpose-built for high-speed big data analytics. If this sounds like fluffy marketing talk, resist the temptation to close this tab,… Continue reading→

Categories
Apache Spark
Tags
, , , , , 3 Comments on Real Time Big Data analytics: Parquet (and Spark) + bonus