Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Later
-
1.0.0
-
None
-
None
Description
Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector with KryoSerializer throws the following stack trace. Looks like some recursive field in the object. Upgrading to 0.8.1 solved this.
java.lang.StackOverflowError at java.lang.reflect.Field.getDeclaringClass(Field.java:154) at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54) at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38) at java.lang.reflect.Field.get(Field.java:379) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) ...
Code to reproduce:
import breeze.linalg.DenseVector import org.apache.spark.SparkConf import org.apache.spark.serializer.KryoSerializer object SerializerTest { def main(args: Array[String]) { val conf = new SparkConf() .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") .set("spark.kryo.registrator", classOf[MyRegistrator].getName) .set("spark.kryo.referenceTracking", "false") .set("spark.kryoserializer.buffer.mb", "8") val serializer = new KryoSerializer(conf).newInstance() serializer.serialize(DenseVector.rand(10)) } }
Attachments
Issue Links
- relates to
-
SPARK-1997 Update breeze to version 0.9
-
- Resolved
-