[SPARK-20603][SS][TEST] Set default number of topic partitions to 1 to reduce the load

zsxwing · zsxwing · commit bd5788287957 · 2017-05-05T11:08:26.000-07:00
## What changes were proposed in this pull request? I checked the logs of https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.2-test-maven-hadoop-2.7/47/ and found it took several seconds to create Kafka internal topic `__consumer_offsets`. As Kafka creates this topic lazily, the topic creation happens in the first test `deserialization of initial offset with Spark 2.1.0` and causes it timeout. This PR changes `offsets.topic.num.partitions` from the default value 50 to 1 to make creating `__consumer_offsets` (50 partitions -> 1 partition) much faster. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #17863 from zsxwing/fix-kafka-flaky-test.
diff --git a/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala b/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
@@ -292,6 +292,7 @@ class KafkaTestUtils(withBrokerProps: Map[String, Object] = Map.empty) extends L
     props.put("log.flush.interval.messages", "1")
     props.put("replica.socket.timeout.ms", "1500")
     props.put("delete.topic.enable", "true")
+    props.put("offsets.topic.num.partitions", "1")
     props.putAll(withBrokerProps.asJava)
     props
   }

Original file line number	Diff line number	Diff line change
`@@ -292,6 +292,7 @@ class KafkaTestUtils(withBrokerProps: Map[String, Object] = Map.empty) extends L`
`292`	`292`	`props.put("log.flush.interval.messages", "1")`
`293`	`293`	`props.put("replica.socket.timeout.ms", "1500")`
`294`	`294`	`props.put("delete.topic.enable", "true")`
	`295`	`+ props.put("offsets.topic.num.partitions", "1")`
`295`	`296`	`props.putAll(withBrokerProps.asJava)`
`296`	`297`	`props`
`297`	`298`	`}`