Description
https://issues.apache.org/jira/browse/SPARK-27801 has improved file listing for DistributedFileSystem, where InMemoryFileIndex.listLeafFiles makes use of DistributedFileSystem's one single listLocatedStatus to namenode. This ticket intends to improve the case where ViewFileSystem is used to manage multiple DistributedFileSystems. It has also overridden the listLocatedStatus method by delegating to the filesystem it resolves to, e.g. DistributedFileSystem.
Attachments
Issue Links
- relates to
-
SPARK-27801 InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
-
- Resolved
-
- links to