Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31047

Improve file listing for ViewFileSystem

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • SQL
    • None

    Description

      https://issues.apache.org/jira/browse/SPARK-27801 has improved file listing for DistributedFileSystem, where InMemoryFileIndex.listLeafFiles makes use of DistributedFileSystem's one single listLocatedStatus to namenode. This ticket intends to improve the case where ViewFileSystem is used to manage multiple DistributedFileSystems. It has also overridden the listLocatedStatus method by delegating to the filesystem it resolves to, e.g. DistributedFileSystem.

      Attachments

        Issue Links

          Activity

            People

              mauzhang Manu Zhang
              mauzhang Manu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: