Python is the de facto language of data science and engineering, which affords it an outsized community of users. However, when many data scientists and engineers come to Spark with a Python background, unexpected performance potholes can stand in the way of progress. These \u201cPerformance Potholes\u201d include PySpark\u2019s ease of integration with existing packages (e.g. Pandas, SciPy, Scikit Learn, etc), using Python UDFs, and utilizing the RDD APIs instead of Spark SQL DataFrames without understanding the implications. Additionally, Spark 2.3 changes the game even further with vectorized UDFs. In this talk, we will discuss:<\/p>\n
\u2013 How PySpark works broadly (& why it matters)
\n\u2013 Integrating popular Python packages with Spark
\n\u2013 Python UDFs (how to [not] use them)
\n\u2013 RDDs vs Spark SQL DataFrames
\n\u2013 Spark 2.3 Vectorized UDFs<\/p>\n
Session hashtag: #Py9SAIS<\/a><\/p>\n Download full slides here<\/a><\/p>\n Spark + AI Summit session page with video<\/a><\/p>\n<\/div>\n Scaling Python for Data Science using Spark<\/a><\/p><\/blockquote>\n