Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2200

breeze DenseVector not serializable with KryoSerializer

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Later
    • 1.0.0
    • None
    • MLlib
    • None

    Description

      Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector with KryoSerializer throws the following stack trace. Looks like some recursive field in the object. Upgrading to 0.8.1 solved this.

      java.lang.StackOverflowError
      	at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
      	at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
      	at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
      	at java.lang.reflect.Field.get(Field.java:379)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
      	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
      	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
      ...
      

      Code to reproduce:

      import breeze.linalg.DenseVector
      import org.apache.spark.SparkConf
      import org.apache.spark.serializer.KryoSerializer
      
      object SerializerTest {
        def main(args: Array[String]) {
          val conf = new SparkConf()
            .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
            .set("spark.kryo.registrator", classOf[MyRegistrator].getName)
            .set("spark.kryo.referenceTracking", "false")
            .set("spark.kryoserializer.buffer.mb", "8")
      
          val serializer = new KryoSerializer(conf).newInstance()
          serializer.serialize(DenseVector.rand(10))
        }
      }
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sinisa_lyh Neville Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: