scala - Spark streaming throws stack overflow exception after running for sometime -


spark streaming starts giving stack overflow exception after running while. stack trace has not been figure out problem , thats why posting here. please help, exception follows

15/05/14 14:02:04 error executor: exception in task 1.0 in stage 134161.0 (tid 7008) java.lang.stackoverflowerror     @ java.io.objectinputstream.readclassdesc(objectinputstream.java:1508)     @ java.io.objectinputstream.readclass(objectinputstream.java:1480)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1330)     @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.readobject(objectinputstream.java:369)     @ scala.collection.immutable.$colon$colon.readobject(list.scala:362)     @ sun.reflect.generatedmethodaccessor6.invoke(unknown source)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:601)     @ java.io.objectstreamclass.invokereadobject(objectstreamclass.java:1004)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1866)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.readobject(objectinputstream.java:369)     @ scala.collection.immutable.$colon$colon.readobject(list.scala:362)     @ sun.reflect.generatedmethodaccessor6.invoke(unknown source)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:601)     @ java.io.objectstreamclass.invokereadobject(objectstreamclass.java:1004)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1866)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)     @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964)     @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888)     @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771)     @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) 

it deadly checkpoint combination: chain of rdds set checkpointed @ same time. if case, understanding actual checkpoint write held , object grows, filling stacks.

more info here: https://issues.apache.org/jira/browse/spark-6847.


Comments

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

php - How can I echo out this array? -