scala - Spark streaming throws stack overflow exception after running for sometime -
spark streaming starts giving stack overflow exception after running while. stack trace has not been figure out problem , thats why posting here. please help, exception follows
15/05/14 14:02:04 error executor: exception in task 1.0 in stage 134161.0 (tid 7008) java.lang.stackoverflowerror @ java.io.objectinputstream.readclassdesc(objectinputstream.java:1508) @ java.io.objectinputstream.readclass(objectinputstream.java:1480) @ java.io.objectinputstream.readobject0(objectinputstream.java:1330) @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.readobject(objectinputstream.java:369) @ scala.collection.immutable.$colon$colon.readobject(list.scala:362) @ sun.reflect.generatedmethodaccessor6.invoke(unknown source) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:601) @ java.io.objectstreamclass.invokereadobject(objectstreamclass.java:1004) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1866) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.readobject(objectinputstream.java:369) @ scala.collection.immutable.$colon$colon.readobject(list.scala:362) @ sun.reflect.generatedmethodaccessor6.invoke(unknown source) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:601) @ java.io.objectstreamclass.invokereadobject(objectstreamclass.java:1004) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1866) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347) @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:1964) @ java.io.objectinputstream.readserialdata(objectinputstream.java:1888) @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1771) @ java.io.objectinputstream.readobject0(objectinputstream.java:1347)
it deadly checkpoint combination: chain of rdd
s set checkpointed @ same time. if case, understanding actual checkpoint write held , object grows, filling stacks.
more info here: https://issues.apache.org/jira/browse/spark-6847.
Comments
Post a Comment