Specialize contextlib.contextmanager #905

Activity

1. JukkaL

   added

   speed

   on Nov 7, 2021
2. ichard26

   added

   area-specialized-opsGenerating faster type-specific code.Generating faster type-specific code.

   on Jan 13, 2023
3. ichard26

   added

   priority-1-normal

   on Feb 21, 2023
4. ilevkivskyi

   added

   priority-0-high

   and removed

   priority-1-normal

   on Jan 22, 2026
5. ilevkivskyi commented on Jan 22, 2026

   

   ilevkivskyi

   on Jan 22, 2026

   Collaborator

   More actions

   Now that mypy is faster this can bring potentially big perf win for mypy, so I raised priority to high.
6. JukkaL commented on Jan 22, 2026

   

   JukkaL

   on Jan 22, 2026

   CollaboratorAuthor

   More actions

   Since general context managers are pretty hard to optimize, we could instead add librt.contextlib.contextmanager that only works for sufficiently simple context managers.

   First, it should be easy to support context managers with a single yield statement at the top level block (generate a class with suitable __enter__ and __exit__ methods):

   @contextmanger
   def foo(...) -> ...:
        ...
        yield
        ...

   Second, yield in try/finally is also common:

   @contextmanager
   def foo(...) -> ...:
       ...
       try:
           yield
       finally:
           ...

   Again, we should be able to compile this into a class with __enter__ and __exit__.

   The third most common use case seems to be yield within a with statement. This might not be too hard either, but I'm not sure.
7. JukkaL commented on Jan 22, 2026

   

   JukkaL

   on Jan 22, 2026

   CollaboratorAuthor

   More actions

   Or maybe I'm overthinking this -- maybe it's sufficient to compile a normal generator and call the generator methods directly when the context manager is used.
8. ilevkivskyi commented on Jan 22, 2026

   

   ilevkivskyi

   on Jan 22, 2026

   Collaborator

   More actions

   @JukkaL Yes, I think simply avoiding a bunch of slow Python calls should be enough (although I didn't do any profiling I intuitively feel this is where the problem occurs). I don't think we will need to look into the generator body, just make sure all calls (i.e. with ... -> .send() etc) are fast C-level calls.
9. ilevkivskyi commented on Jan 22, 2026

   

   ilevkivskyi

   on Jan 22, 2026

   Collaborator

   More actions

   That said, if we are going this way, it looks like we will need to replicate all the tedious dances with exceptions in __exit__() from here https://github.com/python/cpython/blob/main/Lib/contextlib.py#L145
10. hauntsaninja commented on Jan 27, 2026

    

    hauntsaninja

    on Jan 27, 2026

    Last edited by hauntsaninja

    Collaborator

    More actions

    I took a shot at it, but (assuming I did it correctly) it's slower than the original... I think exception handling is slow, looks like #476 is related
11. ilevkivskyi commented on Jan 29, 2026

    

    ilevkivskyi

    on Jan 29, 2026

    Collaborator

    More actions

    I took a shot at it, but (assuming I did it correctly) it's slower than the original...

    TBH I am surprised. We have special handling for StopIteration (normally we don't raise it and instead return a special value, stop_iter_ptr). And any other exceptions should happen very rarely. I guess it would make sense to look into what you did in details.