Repository navigation
Inconsistency in S3 Transfer Operators #37213
Description
Description
Standardize similar AWS transfer operators to have similar functionality, as well as similar naming conventions.
This would start with updating transfer operators using S3 as a source or destination to have similar behavior and naming.
Use case/motivation
There seems to be substantial discrepancies between transfer operators, especially those using S3 as a source or destination. For example, in the S3ToSFTPOperator, only a single file can be transferred from the SFTP to the S3 bucket. However, in the FTPToS3Operator an arbitrary number of files can be transferred from S3 to an FTP site. To add to this, the S3ToFTPOperator only supports a single file being transferred.
These sort of discrepancies exist for a number of transfer operators. In addition to that, parameter names are quite different from operator to operator, which makes using them quite difficult.
This will allow for a more seamless experience for Airflow users, and ensure common functionality across similar tools.
Related issues
No related issues.
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Feb 7, 2024
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
bricefotzo commentedon Feb 10, 2024
Hi, I would like to bé assigned on this as my first issue, if possible. @jroachgolf84 would it be ok for you?
jroachgolf84 commentedon Jun 17, 2024
@bricefotzo, were you able to complete this issue? If not, I'm looking forward to working on it.
bricefotzo commentedon Jun 18, 2024
@jroachgolf84 I have added those changes 3da0dcf
eladkal commentedon Jul 28, 2024
@bricefotzo do you plan to raise a PR for this issue?
bricefotzo commentedon Jul 28, 2024
Yes, I'll create a PR this week
potiuk commentedon Mar 2, 2026
@bricefotzo We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.