Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21063

Spark return an empty result from remote hadoop cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 2.1.0, 2.1.1
    • None
    • Spark Core, SQL
    • None

    Description

      Spark returning empty result from when querying remote hadoop cluster.
      All firewall settings removed.
      Querying using JDBC working properly using hive-jdbc driver from version 1.1.1

      Code snippet is:

      val spark = SparkSession.builder
    .appName("RemoteSparkTest")
    .master("local")
    .getOrCreate()

val df = spark.read
  .option("url", "jdbc:hive2://remote.hive.local:10000/default")
  .option("user", "user")
  .option("password", "pass")
  .option("dbtable", "test_table")
  .option("driver", "org.apache.hive.jdbc.HiveDriver")
  .format("jdbc")
  .load()
 
df.show()

      Result:

      +-------------------+
|test_table.test_col|
+-------------------+
+-------------------+

      All manipulations like: 

      df.select(*).show()

      returns empty result too.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31457 spark jdbc read hive created the wrong PreparedStatement

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-22016 Add HiveDialect for JDBC connection to Hive

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              pbykov Peter Bykov
              Votes:
              4 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: