Repository navigation
Wire DataflowConfiguration.max_num_workers into pipeline options - #73922
garvit-arora wants to merge 3 commits into
Conversation
Pass max_num_workers through to Beam Dataflow pipeline options so the documented parameter takes effect. Relates-to: apache#72982 Generated-by: Cursor
| pipeline_options["project"] = self.dataflow_config.project_id | ||
| pipeline_options["region"] = self.dataflow_config.location | ||
| if self.dataflow_config.max_num_workers is not None: | ||
| pipeline_options["max_num_workers"] = self.dataflow_config.max_num_workers |
There was a problem hiding this comment.
The Java operator passes keys as-is, so this reaches Java as --max_num_workers, but Java reads --maxNumWorkers and would likely reject unknown flags. Would maxNumWorkers work here, like impersonateServiceAccount? convert_camel_to_snake would still give Python and Go max_num_workers.
Java Beam reads --maxNumWorkers; snake_case max_num_workers is rejected as an unknown flag. Generated-by: Cursor
|
Pushed \50bd6056\ to address @yuseok89's review: Java Beam expects --maxNumWorkers, not --max_num_workers. Test expectation updated accordingly. |
Clarify that operator pipeline_options maxNumWorkers wins over dataflow_config.max_num_workers, and add a Dataflow runner regression test. Generated-by: Cursor
|
Pushed 4c2f013 addressing bingqin2 pointers: documented max_num_workers override precedence in DataflowConfiguration, and added a Dataflow runner test asserting pipeline_options maxNumWorkers wins over dataflow_config.max_num_workers. |
yuseok89
left a comment
yuseok89
left a comment
There was a problem hiding this comment.
While this waits for a maintainer review, LGTM from the option wiring side.
Summary
Relates-to: #72982
Test plan