Exception when using UDT in Spark DataFrame -


i'm trying create user defined type in spark sql, receive: com.ubs.ged.risk.stdout.spark.examplepointudt cannot cast org.apache.spark.sql.types.structtype when using example. has made work?

my code:

test("udt serialisation") {     val points = seq(new examplepoint(1.3, 1.6), new examplepoint(1.3, 1.8))     val df = sparkcontextforstdout.context.parallelize(points).todf() }  @sqluserdefinedtype(udt = classof[examplepointudt])  case class examplepoint(val x: double, val y: double)  /**  * user-defined type [[examplepoint]].  */ class examplepointudt extends userdefinedtype[examplepoint] {    override def sqltype: datatype = arraytype(doubletype, false)    override def pyudt: string = "pyspark.sql.tests.examplepointudt"    override def serialize(obj: any): seq[double] = {     obj match {       case p: examplepoint =>         seq(p.x, p.y)     }   }    override def deserialize(datum: any): examplepoint = {     datum match {       case values: seq[_] =>         val xy = values.asinstanceof[seq[double]]         assert(xy.length == 2)         new examplepoint(xy(0), xy(1))       case values: util.arraylist[_] =>         val xy = values.asinstanceof[util.arraylist[double]].asscala         new examplepoint(xy(0), xy(1))     }   }    override def userclass: class[examplepoint] = classof[examplepoint]  }

the usefull stackstrace this:

com.ubs.ged.risk.stdout.spark.examplepointudt cannot cast org.apache.spark.sql.types.structtype java.lang.classcastexception: com.ubs.ged.risk.stdout.spark.examplepointudt cannot cast org.apache.spark.sql.types.structtype     @ org.apache.spark.sql.sqlcontext.createdataframe(sqlcontext.scala:316)     @ org.apache.spark.sql.sqlcontext$implicits$.rddtodataframeholder(sqlcontext.scala:254)

it seems udt needs used inside of class work (as type of field). 1 solution use directly wrap tuple1:

  test("udt serialisation") {     val points = seq(new tuple1(new examplepoint(1.3, 1.6)), new tuple1(new examplepoint(1.3, 1.8)))     val df = sparkcontextforstdout.context.parallelize(points).todf()     df.collect().foreach(println(_))   }

Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more