Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1101 Umbrella for hardening Spark on YARN
  3. SPARK-1064

Make it possible to use cluster's Hadoop jars when running against YARN

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • YARN
    • None

    Description

      YARN applications like MapReduce and Tez rely on the cluster's Hadoop jars instead of distributing their own.

      This has a couple advantages

      • Avoids sending a bunch of bits to every node for each app
      • Only a single version of Hadoop can be running on a cluster at one time, simplifying debugging
      • Easier to upgrade and apply patched versions of Hadoop

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-1233 spark on hadoop 0.23 yarn fails to run: java.lang.NoSuchFieldException: DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              sandyr Sandy Ryza
              sandy Sanford Ryza
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: