[Improve] Refactor S3FileCatalog and it's factory #7457

loustler · 2024-08-22T09:06:48Z

Purpose of this pull request

Refactor S3FileCatalog and S3FileCatalogFactory, it's logic exactly same with AbstractFileCatalog

seatunnel/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/catalog/AbstractFileCatalog.java

Lines 37 to 130 in 62b4f16

    
           public abstract class AbstractFileCatalog implements Catalog { 
        
               protected final String catalogName; 
        
               private final HadoopFileSystemProxy hadoopFileSystemProxy; 
        
               private final String filePath; 
        
               protected AbstractFileCatalog( 
        
                       HadoopFileSystemProxy hadoopFileSystemProxy, String filePath, String catalogName) { 
        
                   this.catalogName = catalogName; 
        
                   this.filePath = filePath; 
        
                   this.hadoopFileSystemProxy = hadoopFileSystemProxy; 
        
               } 
        
               @Override 
        
               public void open() throws CatalogException {} 
        
               @Override 
        
               public void close() throws CatalogException {} 
        
               @Override 
        
               public String name() { 
        
                   return catalogName; 
        
               } 
        
               @Override 
        
               public String getDefaultDatabase() throws CatalogException { 
        
                   return null; 
        
               } 
        
               @Override 
        
               public boolean databaseExists(String databaseName) throws CatalogException { 
        
                   return false; 
        
               } 
        
               @Override 
        
               public List<String> listDatabases() throws CatalogException { 
        
                   return null; 
        
               } 
        
               @Override 
        
               public List<String> listTables(String databaseName) 
        
                       throws CatalogException, DatabaseNotExistException { 
        
                   return null; 
        
               } 
        
               @SneakyThrows 
        
               @Override 
        
               public boolean tableExists(TablePath tablePath) throws CatalogException { 
        
                   return hadoopFileSystemProxy.fileExist(filePath); 
        
               } 
        
               @Override 
        
               public CatalogTable getTable(TablePath tablePath) 
        
                       throws CatalogException, TableNotExistException { 
        
                   return null; 
        
               } 
        
               @SneakyThrows 
        
               @Override 
        
               public void createTable(TablePath tablePath, CatalogTable table, boolean ignoreIfExists) 
        
                       throws TableAlreadyExistException, DatabaseNotExistException, CatalogException { 
        
                   hadoopFileSystemProxy.createDir(filePath); 
        
               } 
        
               @SneakyThrows 
        
               @Override 
        
               public void dropTable(TablePath tablePath, boolean ignoreIfNotExists) 
        
                       throws TableNotExistException, CatalogException { 
        
                   hadoopFileSystemProxy.deleteFile(filePath); 
        
               } 
        
               @Override 
        
               public void createDatabase(TablePath tablePath, boolean ignoreIfExists) 
        
                       throws DatabaseAlreadyExistException, CatalogException {} 
        
               @Override 
        
               public void dropDatabase(TablePath tablePath, boolean ignoreIfNotExists) 
        
                       throws DatabaseNotExistException, CatalogException {} 
        
               @SneakyThrows 
        
               @Override 
        
               public void truncateTable(TablePath tablePath, boolean ignoreIfNotExists) 
        
                       throws TableNotExistException, CatalogException { 
        
                   hadoopFileSystemProxy.deleteFile(filePath); 
        
                   hadoopFileSystemProxy.createDir(filePath); 
        
               } 
        
               @SneakyThrows 
        
               @Override 
        
               public boolean isExistsData(TablePath tablePath) { 
        
                   final List<LocatedFileStatus> locatedFileStatuses = 
        
                           hadoopFileSystemProxy.listFile(filePath); 
        
                   return CollectionUtils.isNotEmpty(locatedFileStatuses); 
        
               }

Does this PR introduce any user-facing change?

Nothing

How was this patch tested?

No need tests I think. It's enough already written tests.

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

...3/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/s3/catalog/S3FileCatalog.java

Hisoka-X

LGTM. Thanks @loustler !

Refactor S3FileCatalog and it's factory

1bacc3f

github-actions bot added connectors-v2 file labels Aug 22, 2024

Hisoka-X changed the title ~~Refactor S3FileCatalog and it's factory~~ [Improve] Refactor S3FileCatalog and it's factory Aug 22, 2024

Hisoka-X reviewed Aug 22, 2024

View reviewed changes

...3/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/s3/catalog/S3FileCatalog.java Show resolved Hide resolved

Add a close implementation

46ad455

Hisoka-X approved these changes Aug 23, 2024

View reviewed changes

github-actions bot added approved reviewed labels Aug 23, 2024

hailin0 approved these changes Aug 23, 2024

View reviewed changes

hailin0 merged commit d928e8b into apache:dev Aug 23, 2024

loustler deleted the improve/s3-file-catalog branch August 23, 2024 05:36

hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Aug 29, 2024

[Improve] Refactor S3FileCatalog and it's factory (apache#7457)

81483ca

hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Aug 29, 2024

[Improve] Refactor S3FileCatalog and it's factory (apache#7457)

905fd51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Improve] Refactor S3FileCatalog and it's factory #7457

[Improve] Refactor S3FileCatalog and it's factory #7457

Uh oh!

loustler commented Aug 22, 2024 •

edited

Loading

Uh oh!

Uh oh!

Hisoka-X left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	public abstract class AbstractFileCatalog implements Catalog {

	protected final String catalogName;
	private final HadoopFileSystemProxy hadoopFileSystemProxy;
	private final String filePath;

	protected AbstractFileCatalog(
	HadoopFileSystemProxy hadoopFileSystemProxy, String filePath, String catalogName) {
	this.catalogName = catalogName;
	this.filePath = filePath;
	this.hadoopFileSystemProxy = hadoopFileSystemProxy;
	}

	@Override
	public void open() throws CatalogException {}

	@Override
	public void close() throws CatalogException {}

	@Override
	public String name() {
	return catalogName;
	}

	@Override
	public String getDefaultDatabase() throws CatalogException {
	return null;
	}

	@Override
	public boolean databaseExists(String databaseName) throws CatalogException {
	return false;
	}

	@Override
	public List<String> listDatabases() throws CatalogException {
	return null;
	}

	@Override
	public List<String> listTables(String databaseName)
	throws CatalogException, DatabaseNotExistException {
	return null;
	}

	@SneakyThrows
	@Override
	public boolean tableExists(TablePath tablePath) throws CatalogException {
	return hadoopFileSystemProxy.fileExist(filePath);
	}

	@Override
	public CatalogTable getTable(TablePath tablePath)
	throws CatalogException, TableNotExistException {
	return null;
	}

	@SneakyThrows
	@Override
	public void createTable(TablePath tablePath, CatalogTable table, boolean ignoreIfExists)
	throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
	hadoopFileSystemProxy.createDir(filePath);
	}

	@SneakyThrows
	@Override
	public void dropTable(TablePath tablePath, boolean ignoreIfNotExists)
	throws TableNotExistException, CatalogException {
	hadoopFileSystemProxy.deleteFile(filePath);
	}

	@Override
	public void createDatabase(TablePath tablePath, boolean ignoreIfExists)
	throws DatabaseAlreadyExistException, CatalogException {}

	@Override
	public void dropDatabase(TablePath tablePath, boolean ignoreIfNotExists)
	throws DatabaseNotExistException, CatalogException {}

	@SneakyThrows
	@Override
	public void truncateTable(TablePath tablePath, boolean ignoreIfNotExists)
	throws TableNotExistException, CatalogException {
	hadoopFileSystemProxy.deleteFile(filePath);
	hadoopFileSystemProxy.createDir(filePath);
	}

	@SneakyThrows
	@Override
	public boolean isExistsData(TablePath tablePath) {
	final List<LocatedFileStatus> locatedFileStatuses =
	hadoopFileSystemProxy.listFile(filePath);
	return CollectionUtils.isNotEmpty(locatedFileStatuses);
	}

[Improve] Refactor S3FileCatalog and it's factory #7457

[Improve] Refactor S3FileCatalog and it's factory #7457

Uh oh!

Conversation

loustler commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

loustler commented Aug 22, 2024 •

edited

Loading