Smarter selection of sdist packages to build #44023
Description
Currently we have an edge case that when we release a new version of provider that other providers depend on, the sdist build might have a problem with fiding the dependent packages in some PRS - when the PRs are modifying only some packages as they are not locally built.
When we build sdist providers, we only build them in some PRs for only affected providers in this PR - and their dependencies. The sdist provider builds are spit into chunks, and the chunks are built independently in parallel (for speed) which means that in some cases the required new version of package will be missing.
For example this happened in #44018 (https://github.com/apache/airflow/actions/runs/11835585982) when modified standard provider,
× No solution found when resolving dependencies:
╰─▶ Because only apache-airflow-providers-common-sql<=1.19.0 is available
and apache-airflow-providers-standard==0.0.1.dev0 depends on
apache-airflow-providers-common-sql>=1.20.0.dev0, we can conclude that
apache-airflow-providers-standard==0.0.1.dev0 cannot be used.
And because only apache-airflow-providers-standard==0.0.1.dev0 is
available and you require apache-airflow-providers-standard, we can
conclude that your requirements are unsatisfiable.
Traceback (most recent call last):
File "/opt/airflow/scripts/in_container/install_airflow_and_providers.py", line 565, in <module>
install_airflow_and_providers()
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/rich_click/rich_command.py", line 152, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/airflow/scripts/in_container/install_airflow_and_providers.py", line 541, in install_airflow_and_providers
run_command(base_install_providers_cmd, github_actions=github_actions, check=True)
File "/opt/airflow/scripts/in_container/in_container_utils.py", line 47, in run_command
result = subprocess.run(cmd, **kwargs)
File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/uv', 'pip', 'install', '--python', '/usr/local/bin/python', '/dist/apache_airflow-3.0.0.dev0.tar.gz', '/dist/apache_airflow_providers_standard-0.0.1.dev0.tar.gz', '/dist/apache_airflow-3.0.0.dev0.tar.gz']' returned non-zero exit status 1.
We should be a bit smarter here and detect (in each chunk separately) which providers are additionally needed to be built - in this case since standard depends on comon.sql >= 1.20.0, the sdist chunk installation should add common.sql to the list of installed providers in the chunk where standard is installed.