How to guarantee the [tool.poetry.dependencies] settings align to requireiments.txt? As well as the poetry.lock lock file to the requireiments.txt?

How to guarantee the [tool.poetry.dependencies] settings align to requireiments.txt? As well as the poetry.lock lock file to the requireiments.txt?

Hi @bowenliang123 , in current stage. we can use poetry add command to keep dependencies consistency as I mentioned in README.md:

In the future, I'd like to suggest that remove requirements.txt and requirements-dev.txt and entirely use poetry to manage dependencies.

Can you figure out at least one way in CI jobs to making sure poetry dependency list satisfying the existed requirements.txt ?

I'm against setting the removal the requirements.txt as final target, considering the following facts:

1. requirements.txt is generally used out of box with python and pip
2. poetry is just one of the build systems with dependencies management
3. For the runtime usage, the build system does not bring any benefits , while it causes reversed dependencies on poetry to the runtime base, like in docker images.

Can you figure out at least one way in CI jobs to making sure poetry dependency list satisfying the existed requirements.txt ?

I'm against setting the removal the requirements.txt as final target, considering the following facts:

1. requirements.txt is generally used out of box with python and pip
2. poetry is just one of the build systems with dependencies management
3. For the runtime usage, the build system does not bring any benefits , while it causes reversed dependencies on poetry to the runtime base, like in docker images.

This PR is just for providing another for developers to manage dependencies, it doesn't impact unit tests, CI and product env. As for your objections, I'd like to discuss with you in another PR(not created yet).

This PR is just for providing another for developers to manage dependencies, it doesn't impact unit tests, CI and product env. As for your opsitions, I'd like to discuss with you in another PR(not created yet).

The introduced support for a new build system should at least respect and align with the existing core mechanisms. Otherwise, it will puzzle the users with the issues caused by inconsistencies in dependencies(which happen often), as the de-facto two different dependencies are listed in the Dify API module.

This PR is just for providing another for developers to manage dependencies, it doesn't impact unit tests, CI and product env. As for your opsitions, I'd like to discuss with you in another PR(not created yet).

The introduced support for a new build system should at least respect and align with the existing core mechanisms. Otherwise, it will puzzle the users with the issues caused by inconsistencies in dependencies(which happen often), as the de-facto two different dependencies are listed in the Dify API module.

Currently, the only supported build system is still not changed. the dependencies in pyproject.toml will be kept update from requirements.txt. So there won't introduce any new issues from this PR.

Hi @bowenliang123, is it possible to migrate to poetry in the future? Actually, currently lint and test are using poetry's configuration file, while package dependencies are using pip. I hope in the future we can consolidate everything with poetry, and poetry can parallelize the dependency fetching process to speed up package installation.

Hi @bowenliang123, is it possible to migrate to poetry in the future? Actually, currently lint and test are using poetry's configuration file, while package dependencies are using pip. I hope in the future we can consolidate everything with poetry, and poetry can parallelize the dependency fetching process to speed up package installation.

Hi @takatost , I am always open to modern build systems like hatch or poetry. All I want to prevent is the inconsistency of dependencies, especially we don't have our decisions in the choice of build system.

I'm suggesting the following changes in this PR to meet the baseline of acceptance for initial support for poetry build system.

1. change the PR title build: initial support for poetry build tool
2. add CI job to check poetry.lock file aligned to the dependency list of [tool.poetry.dependencies] in pyproject.toml, by using poetry check
3. add CI job in api-tests.yml workflow to install dependencies via poetry and run all through all the tests

These actions still don't guarantee the poetry dependencies aligned to requirements.txt, but it should be good enough for initial support. The dependency list of [tool.poetry.dependencies] in pyproject.toml requires more polishment in version declaration like preferring ^ operator rather than >= + <, which could be done in the future.

WDYT? @takatost @MatriQ

I'm suggesting the following changes in this PR to meet the baseline of acceptance for initial support for poetry build system.

1. change the PR title build: initial support for poetry build tool
2. add CI job to check poetry.lock file aligned to the dependency list of [tool.poetry.dependencies] in pyproject.toml, by using poetry check
3. add CI job in api-tests.yml workflow to install dependencies via poetry and run all through all the tests

These actions still don't guarantee the poetry dependencies aligned to requirements.txt, but it should be good enough for initial support. The dependency list of [tool.poetry.dependencies] in pyproject.toml requires more polishment in version declaration like preferring ^ operator rather than >= + <, which could be done in the future.

WDYT? @takatost @MatriQ

It's ok for me. But I'd like to get opinion from @takatost