apache spark - Why does dropna() not work? -


system: spark 1.3.0 (anaconda python dist.) on cloudera quickstart vm 5.4

here's spark dataframe:

from pyspark.sql import sqlcontext pyspark.sql.types import * sqlcontext = sqlcontext(sc)  data = sc.parallelize([('foo',41,'us',3),                        ('foo',39,'uk',1),                        ('bar',57,'ca',2),                        ('bar',72,'ca',3),                        ('baz',22,'us',6),                        (none,75,none,7)])  schema = structtype([structfield('name', stringtype(), true),                      structfield('age', integertype(), true),                      structfield('country', stringtype(), true),                      structfield('score', integertype(), true)])  df = sqlcontext.createdataframe(data,schema)

data.show()

name age country score foo  41       3     foo  39  uk      1     bar  57  ca      2     bar  72  ca      3     baz  22       6     null 75  null    7

however neither of these work!

df.dropna() df.na.drop()

i message:

>>> df.show() name age country score foo  41       3     foo  39  uk      1     bar  57  ca      2     bar  72  ca      3     baz  22       6     null 75  null    7     >>> df.dropna().show() traceback (most recent call last):   file "<stdin>", line 1, in <module>   file "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 580, in __getattr__     jc = self._jdf.apply(name)   file "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__   file "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value py4j.protocol.py4jjavaerror: error occurred while calling o50.apply. : org.apache.spark.sql.analysisexception: cannot resolve column name "dropna" among (name, age, country, score);     @ org.apache.spark.sql.dataframe$$anonfun$resolve$1.apply(dataframe.scala:162)     @ org.apache.spark.sql.dataframe$$anonfun$resolve$1.apply(dataframe.scala:162)     @ scala.option.getorelse(option.scala:120)     @ org.apache.spark.sql.dataframe.resolve(dataframe.scala:161)     @ org.apache.spark.sql.dataframe.col(dataframe.scala:436)     @ org.apache.spark.sql.dataframe.apply(dataframe.scala:426)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:606)     @ py4j.reflection.methodinvoker.invoke(methodinvoker.java:231)     @ py4j.reflection.reflectionengine.invoke(reflectionengine.java:379)     @ py4j.gateway.invoke(gateway.java:259)     @ py4j.commands.abstractcommand.invokemethod(abstractcommand.java:133)     @ py4j.commands.callcommand.execute(callcommand.java:79)     @ py4j.gatewayconnection.run(gatewayconnection.java:207)     @ java.lang.thread.run(thread.java:745)

has else experienced problem? what's workaround? pyspark seems thing looking column called "na". appreciated!

tl;dr methods na , dropna available since spark 1.3.1.

few mistakes made:

  1. data = sc.parallelize([....('',75,'', 7 )]), intended use '' represent none, however, it's string instead of null

  2. na , dropna both methods on dataframe class, therefore, should call df.

runnable code:

data = sc.parallelize([('foo',41,'us',3),                        ('foo',39,'uk',1),                        ('bar',57,'ca',2),                        ('bar',72,'ca',3),                        ('baz',22,'us',6),                        (none, 75, none, 7)])  schema = structtype([structfield('name', stringtype(), true),                  structfield('age', integertype(), true),                  structfield('country', stringtype(), true),                  structfield('score', integertype(), true)])  df = sqlcontext.createdataframe(data,schema)  df.dropna().show() df.na.drop().show()

Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more