update spark.default.parallelism

CrazyJvm · CrazyJvm · commit ee0fae00ad02 · 2014-04-11T14:54:58.000+08:00
actually, the value 8 is only valid in mesos fine-grained mode :
&lt;code&gt;
  override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
&lt;/code&gt;

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
&lt;code&gt; 
override def defaultParallelism(): Int = {
    conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
  }
&lt;/code&gt;
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -96,7 +96,11 @@ Apart from these, the following properties are also available, and may be useful
 <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
 <tr>
   <td>spark.default.parallelism</td>
-  <td>8</td>
+  <td>
+    · Mesos fine grained mode: 8<br/>
+    · Local mode: core number of the local machine<br/>
+    · Others: total core number of all executor nodes or 2, whichever is larger
+  </td>
   <td>
     Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
     <code>reduceByKey</code>, etc) when not set by user.
