Using new PySpark 2.3 Vectorized Pandas UDFs: Lessons

Posted by Garren on

Since Spark 2.3 was officially released 2/28/18, I wanted to check the performance of the new Vectorized Pandas UDFs using Apache Arrow. Following up to my Scaling Python for Data Science using Spark post where I mentioned Spark 2.3 introducing Vectorized UDFs, I’m using the same Data (from NYC yellow cabs) with this code: from… Continue reading→

Categories
Apache Spark 1 Comment on Using new PySpark 2.3 Vectorized Pandas UDFs: Lessons

Snowflake: Getting Started with Walkthrough

Posted by Garren on

What is Snowflake? Snowflake is a new era relational SQL data warehouse built for the cloud that seeks to enable seamless and fully elastic access to business-critical data that satisfies everyone from Analysts to IT to Finance. But why – aren’t there enough Data Warehouses already?! tl;dr Quantity != Quality. Snowflake offers decoupled elastic compute… Continue reading→

Categories
Default Leave a Comment on Snowflake: Getting Started with Walkthrough