It should build this automatically when SPARK_HOME is defined right?

I think PYTHONPATH supposed to build automatically when SPARK_HOME is defined by bin/interpreter.sh. But SPARK_YARN_USER_ENV is not taken care of.

Hmm, should we use SparkConf instead then:

https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala

    // Keep this for backwards compatibility but users should move to the config
    sys.env.get("SPARK_YARN_USER_ENV").foreach { userEnvs =>
      YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
    }

I believe all spark.* in the interpreter settings are passed to SparkConf.

https://github.com/apache/incubator-zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java

Is this a bug:

      if (!key.startsWith("spark.") || !val.trim().isEmpty()) {
        logger.debug(String.format("SparkConf: key = [%s], value = [%s]", key, val));
        conf.set(key, val);
      }

Shouldn't it say if (key.startsWith("spark.")?

It should build this automatically when SPARK_HOME is defined right?

That seems right, I didn't need to set PYTHONPATH if SPARK_HOME is set correctly, but I do still need SPARK_YARN_USER_ENV.

However it seems as thought the PYTHONPATH gets created after the zeppelin-env.sh gets run, If I don't set both PYTHONPATH and SPARK_YARN_USER_ENV in that file, It complains that SPARK_YARN_USER_ENV is empty with an array out of bounds exception.

@felixcheung

Is this a bug:

if (!key.startsWith("spark.") || !val.trim().isEmpty()) {
        logger.debug(String.format("SparkConf: key = [%s], value = [%s]", key, val));
        conf.set(key, val);
      }

Shouldn't it say if (key.startsWith("spark.")?

I think it's not a bug. Designed to not pass empty value when the key starts with "spark."
This unittest might helpful to understand.
https://github.com/apache/incubator-zeppelin/blob/master/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java#L168

Hmm, should we use SparkConf instead then:

https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala

// Keep this for backwards compatibility but users should move to the config
    sys.env.get("SPARK_YARN_USER_ENV").foreach { userEnvs =>
      YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
    }

I believe all spark.* in the interpreter settings are passed to SparkConf.

How about put the same comment Keep this for backwards compatibility but users should move to the config in the document and proceed?

How about take care zeppelin-env.sh.template, too?

@Leemoonsoo good idea.

@riverajo @felixcheung @Leemoonsoo
Any changes to be done here?
Is this documentation change still needed?

I'm not sure if we do but the py4j version changes for the last few Spark releases, so if we do, we would need a way to set the right version as per spark version