[BEAM-6721] Set numShards dynamically for TextIO.write()#15500
[BEAM-6721] Set numShards dynamically for TextIO.write()#15500pabloem merged 1 commit intoapache:masterfrom
Conversation
|
@lukecwik Hi, could you review this PR? |
|
Any reason not to use flex templates instead? (One of the motivations for this was that plumbing ValueProvider's absolutely everywhere leads to a lot of boilerplate and cruft.) |
|
Hi Robert, it makes sense if Dataflow deprecates classic templates in favor of flex templates. Do they have the plan? |
|
IIRC there is no plan to deprecate classic templates atm, cc: @an2x. |
|
Run Java PreCommit |
1 similar comment
|
Run Java PreCommit |
|
What is the next step on this PR? |
|
If Flex template is recommended for dynamically changeable shards for TextIO, I think this feature is not necessary, but we ask Dataflow Template creator team to migrate to Flex template from Classic template, at least for the templates writing data with TextIO. |
Ack. And @an2x could comment on that. |
|
Run Java PreCommit |
|
I think this is reason enough to include this change. Customers are currently unable to specify this on a supported template, and this would allow us to support them once the template is merged. I'm inclined to just merge this. |
|
Run Java PreCommit |
|
Run Java PreCommit |
|
ES IO test failed yet another time, and I pinged the bug again. +1 to merge if there are no other concerns. |
|
thanks everyone |
5 participants
Currently,
TextIO.Writeonly supportswithNumShards(int)so that it cannot be changed dynamically in Dataflow template jobs. This PR addswithNumShards(ValueProvider<Integer>)to resolve this issue.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunnercompliance status (on master branch)Examples testing status on various runners
Post-Commit SDK/Transform Integration Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.