Skip to content

Draft: Update python generator with correct mypy types and remove sync endpoints in async generator#15878

Closed
Jay-Madden wants to merge 4 commits intoOpenAPITools:masterfrom
Jay-Madden:update-py-asyncio
Closed

Draft: Update python generator with correct mypy types and remove sync endpoints in async generator#15878
Jay-Madden wants to merge 4 commits intoOpenAPITools:masterfrom
Jay-Madden:update-py-asyncio

Conversation

@Jay-Madden
Copy link
Copy Markdown

Draft pr to update the python generator with mypy types, this also modifies the asyncio generator to remove the sync calls and rely on async all the way through.

Disclaimer: This is very very rough poc code, opening a draft to start the discussion

I have also included a new sample python-asyncio in the samples folder. Please let me know if this is not desirable

cc: @wing328

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package 
    ./bin/generate-samples.sh
    ./bin/utils/export_docs_generators.sh
    
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    For Windows users, please run the script in Git BASH.
  • In case you are adding a new generator, run the following additional script :
    ./bin/utils/ensure-up-to-date
    
    Commit all changed files.
  • File the PR against the correct branch: master (6.3.0) (minor release - breaking changes with fallbacks), 7.0.x (breaking changes without fallbacks)
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

@wing328
Copy link
Copy Markdown
Member

wing328 commented Jun 20, 2023

my first suggestion is to move remove sync endpoints in async generator into a separate PR if possible.

@wing328
Copy link
Copy Markdown
Member

wing328 commented Jun 20, 2023

FYI @taxpon (2017/07) @frol (2017/07) @mbohlool (2017/07) @cbornet (2017/09) @kenjones-cisco (2017/11) @tomplus (2018/10) @arun-nalla (2019/11) @spacether (2019/11) @krjakbrjak (2023/02)

@wing328
Copy link
Copy Markdown
Member

wing328 commented Jun 20, 2023

pydnatic also comes with a Mypy plugin: https://docs.pydantic.dev/latest/mypy_plugin/

have you evaluated it?

would it help in our case?

@Jay-Madden
Copy link
Copy Markdown
Author

I have been testing with the mypy plugin enabled I should probably go ahead and add that configuration to the pyproject.toml, sadly it doesn't address the primary problem which is the fact that it doesn't understand the Constrained types https://docs.pydantic.dev/latest/usage/types/#constrained-types

Example: Plugin Enabled

from pydantic import BaseModel, StrictStr, conlist

class Foo(BaseModel):
    bar: conlist(int, min_items=3) = [1]
petstore_api/models/foo.py:29: error: Invalid type comment or annotation  [valid-type]
petstore_api/models/foo.py:29: note: Suggestion: use conlist[...] instead of conlist(...)

Further more comprehensive discussion here about using Annotated as well which works for all cases except nested constraints on pydantic<2 which is why we have the rather verbose and ugly if typing.TYPE_CHECKING solution.

#15464

@Jay-Madden
Copy link
Copy Markdown
Author

my first suggestion is to move remove sync endpoints in async generator into a separate PR if possible.

I can do this, its included here because the interactions between the sync/async endpoints is incredibly confusing type wise, its sort of warping the intended use of async/await syntax and doesn't translate well to static typing at all.

@wing328
Copy link
Copy Markdown
Member

wing328 commented Jun 21, 2023

why we have the rather verbose and ugly if typing.TYPE_CHECKING solution.

what about an option (e.g. supportMypy) to let users choose whether they need Mypy support or not ?

I would imagine some users like to have it enabled while other prefers the output without Mypy.

@Jay-Madden
Copy link
Copy Markdown
Author

what about an option (e.g. supportMypy) to let users choose whether they need Mypy support or not ?

We could do that for sure, but honestly i cant really think of a time you DONT want mypy types in your project because even if you dont use mypy yourself you still get accurate autocomplete etc within your project

@Jay-Madden
Copy link
Copy Markdown
Author

We can also use annotated for all cases except the nested conlist case which removes like 90% of the typechecking checks

@wing328
Copy link
Copy Markdown
Member

wing328 commented Jul 18, 2023

this also modifies the asyncio generator to remove the sync calls and rely on async all the way through.

@Jay-Madden I wonder if you can file a new PR just for this change so that we can get this enhancement included in the upcoming release.

@multani
Copy link
Copy Markdown
Contributor

multani commented Sep 16, 2023

@Jay-Madden I'm also interested to have these 2 topics moving forward (removing the sync call + adding type information to the generated code) and I can dedicate some time on these tasks, is there anything I can do to help here?

I'd be happy to extract the code to remove the sync calls and make it into a dedicated PR for instance.

@wing328
Copy link
Copy Markdown
Member

wing328 commented Sep 16, 2023

I'd be happy to extract the code to remove the sync calls and make it into a dedicated PR for instance.

yes, please file a PR to start with.

@multani
Copy link
Copy Markdown
Contributor

multani commented Sep 16, 2023

It's in #16601, thanks @Jay-Madden for the initial cleanup!

@Jay-Madden
Copy link
Copy Markdown
Author

@multani @wing328

Apologies for letting this fall by the wayside. We took a look at the options and decided to move in a different direction towards k8s first party client-go for our stuff so this fell in priority. Hopefully the initial start is useful to whoever sees this through

@multani
Copy link
Copy Markdown
Contributor

multani commented Sep 19, 2023

I made an initial proposition in #16624 to improve all the generated types.

@Jay-Madden feel free to close this pull request if you don't intend to work on this in the future.

@Jay-Madden
Copy link
Copy Markdown
Author

Closing this PR as the work has been extended by others

@Jay-Madden Jay-Madden closed this Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants