Why? PySpark uses the basic Python interpreter REPL, so you get the same REPL you’d get by calling python at the command line.<\/p>\n
Fix: Use the iPython REPL by specifying the environment variable
\nPYSPARK_PYTHON=ipython3 before the pyspark command.<\/p>\n
Before: After: This is silly, but I catch myself trying to create variables in python regularly with the Before: After: Just like the val\/var conundrum, println is not a valid keyword in python, but print is!<\/p>\n Before: NameError: name ‘println’ is not defined<\/p>\n Yep, this is one of those frustrating gifts that just keeps on giving [pain].<\/p>\n Scala: Python AttributeError: ‘function’ object has no attribute ‘collect’<\/p>\n After: Python allows both single (‘) quotes and double (“) quotes for strings. Scala uses the single quote to denote more specific types.<\/p>\n Scala scala> 'f' scala> 'foo' scala> \"foo\" == 'foo' Python Switching between Scala and Python on Spark is relatively straightforward, but there are a few differences that can cause some minor frustration. Here are some of the little things I’ve run into and how to adjust for them. PySpark Shell does not support code completion (autocomplete) by default. Why? PySpark uses the basic Python interpreter… Continue reading
\npyspark<\/code><\/p>\n![\"\"](\"http:\/\/garrens.com\/blog\/wp-content\/uploads\/2017\/06\/Screen-Shot-2017-06-27-at-10.28.06-AM.png\")
<\/a><\/p>\n
\nPYSPARK_PYTHON=ipython3 pyspark<\/code><\/p>\n<\/a><\/p>\n\n
val df = spark.read...<\/code> style.<\/p>\n
\n>>> val df = spark.range(100)
\nFile \"\", line 1
\nval df = spark.range(100)
\n^
\nSyntaxError: invalid syntax<\/code><\/p>\n
\n>>> df = spark.range(100)<\/code><\/p>\n\n
\nIn [5]: df.foreach(println)
\n---------------------------------------------------------------------------
\nNameError\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Traceback (most recent call last)
\n<ipython-input-5-3d51e5dc3e2b> in <module>()
\n----> 1 df.foreach(println)<\/code><\/p>\nAfter:
\nIn [6]: df.foreach(print)
\nRow(id=3)
\nRow(id=4)
\nRow(id=2)
\nRow(id=1)
\nRow(id=0)<\/code><\/p>\n\n
\nscala> df.groupBy(\"element\").count.collect.foreach(println)
\n[bar,1]
\n[qux,1]
\n[foo,1]
\n[baz,1]<\/code><\/p>\n
\nBefore:
\nIn [15]: df.groupBy(\"element\").count().foreach(print)
\n---------------------------------------------------------------------------
\nAttributeError Traceback (most recent call last)
\nin ()
\n----> 1 df.groupBy(\"element\").count.collect.foreach(print)<\/code><\/p>\n
\n<\/code>In [17]: df = spark.createDataFrame([(1,\"foo\"), (2, \"bar\"), (3, \"baz\"), (4, \"qux\")]).toDF(\"time\", \"element\")
\nIn [18]: df.groupBy(\"element\").count().foreach(print)
\nRow(element='bar', count=1)
\nRow(element='qux', count=1)
\nRow(element='foo', count=1)
\nRow(element='baz', count=1)<\/code><\/p>\n\n
\nscala> 'f
\nres7: Symbol = 'f<\/p>\n
\nres6: Char = f<\/p>\n
\n<console>:1: error: unclosed character literal
\n'foo'<\/p>\n
\n
\n\"foo\" == 'foo'<\/code><\/p>\n
\n
\nIn [19]: \"foo\" == 'foo'
\nOut[19]: True
\n<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"