What Does This Do

Instrument the SparkSubmit.runMain(...) function to capture all errors occurring in a spark application

This SparkSubmit class is used by spark to launch the application in non YARN/Mesos environnements. The YARN case is already covered by instrumenting the ApplicationMaster.finish() function (from this PR #5267)

Motivation

We are currently using the interface SparkListener to receive events from spark. However, this interface is only capturing errors that occur during the spark computations, but not in arbitrary user code

def main() = {
  val spark = SparkSession.builder.getOrCreate()
  val df = spark.read.parquet("...")
  
  // Exception will be captured because it's happening during spark computations
  df.map(x => new RuntimeException("Some exception"))

  // Exception currently not captured because it is thrown outside of spark computations
  throw new RuntimeException("Some exception")
}