Jump to: navigation, search

Eventlet-removal

Why Eventlet Needs to Be Removed

Eventlet, though once popular for concurrent execution using "greenthreads," now presents several limitations and issues that make it an outdated solution for modern projects like OpenStack:

1. Incompatibility with Recent Versions of Python:

Eventlet struggles to keep up with the evolution of Python, particularly with recent versions (such as Python 3.12 and beyond). Some of its internal functionalities, such as "monkey patching" standard libraries, no longer work correctly or cause unexpected behavior. This limits projects from adopting the latest improvements in Python, including critical security patches.

2. Inefficient Resource and Thread Management:

The greenthreads used by Eventlet can lead to excessive memory consumption and poor resource management, especially under heavy loads or in large-scale applications. Given that OpenStack operates in a distributed and highly parallel environment, this can become a significant bottleneck for overall performance.

3. Maintenance and Security Issues:

Eventlet suffers from a lack of active maintenance and does not receive regular updates to address security vulnerabilities and compatibility issues. This exposes OpenStack to potential security risks if flaws in Eventlet are not addressed promptly.

4. Outdated Approach to Concurrency Programming:

With the emergence of more modern solutions like `asyncio` or `concurrent.futures`, which are natively integrated into Python, using Eventlet feels outdated. These newer libraries offer better concurrency management, improved performance, and tighter integration with the current Python ecosystem.

5. Inability to Adopt New Linux Distributions:

Many modern Linux distributions, such as Ubuntu, RHEL, and others, now ship with newer versions of Python that are not supported by Eventlet. Without migrating away from Eventlet, OpenStack will face challenges in adopting these newer distributions. This will limit the ability to stay current with the latest infrastructure improvements and security enhancements provided by modern operating systems.

Potential Impacts if the Migration is Not Completed

Failure to migrate away from Eventlet could have significant consequences for OpenStack-based projects:

1. Blockage of Python Updates:

Continuing to use Eventlet may cause OpenStack to be stuck on older versions of Python. This prevents the adoption of new features, performance improvements, and —most importantly—security patches offered by newer versions of Python.

2. Security Vulnerabilities Risk:

Given Eventlet's lack of maintenance, unresolved security flaws could leave OpenStack systems vulnerable to attacks. Failing to migrate to modern solutions exposes the infrastructure to major security risks.

3. Degradation of Performance and Scalability:

OpenStack handles large workloads in distributed environments, and Eventlet’s inefficient resource management can cause slowdowns and failures. Without migration, this could limit OpenStack's ability to scale smoothly and handle increasingly larger data volumes.

4. Inability to Adopt New Linux Distributions:

If the migration from Eventlet is not completed, OpenStack will struggle to support modern Linux distributions that come with newer versions of Python. Distributions like Ubuntu and RHEL now ship with versions of Python that are incompatible with Eventlet. This will hinder the ability to upgrade to these distributions, which are critical for staying up-to-date with the latest security fixes and performance improvements.

5. Increased Maintenance Costs:

Continuing to rely on an outdated technology will increase maintenance costs as it becomes harder to find fixes or workarounds for technical issues. This will lead to a growing technical debt that will be difficult to manage in the long term.

To ensure the longevity, stability, and security of OpenStack, it is crucial to migrate to modern solutions like `asyncio` or `concurrent.futures`. This will not only overcome Eventlet's current limitations but also ensure that OpenStack remains capable of meeting performance, security, and scalability demands. Moreover, migrating is essential for supporting the latest Linux distributions like Ubuntu and RHEL, which include newer versions of Python that are not compatible with Eventlet.

The following sections aims to guide you during your migration journey.

Migration's Milestones

  • Adapting common libraries (Oslo, etc...) - e.g: introducing an Async Engine Facade to oslo.db, adapting oslo.service to not relying on Eventlet;
  • Services migration - removing Eventlet usages from OpenStack services;
  • Common libraries migration - common libraries depends on Eventlet because at a higher level - at the services level - Eventlet is required, we cannot remove this kind of Eventlet occurence from the common libraries while services are not fully migrated;
  • Removing Eventlet from our global requirements;
  • Abandoning Eventlet.

Releases

Target Releases

Fully removing Eventlet from OpenStack will surely take several series, we target, at least, the H or I series.

Workload Regulation

  • SLURP releases can be used to introduce "freeze" periods where development slows down to focus on stability, documentation, and training. By example, SLURP releases could used to slow down our migration activity, and to be more focused on observation and documentation. SLURP releases can be also used to deprecate things that need to be deprecated or to remove useless workaround related to Eventlet;
  • use non SLURP releases to introduce breaking changes. E.g, such a replacement of Eventlet WSGI by the selected alternative (see the specialized working groups for further details about breaking changes)

Save the Date

List of the coming events related to the Eventlet removal that could interest the reader.

Recurring Events

Global

Meetings are no longer continued due to the lack of participants.

Here is the etherpad that tracked the meetings https://etherpad.opendev.org/p/flamingo-eventlet-tracking.

Nova

The Nova core team agreed to have a weekly high bandwidth sync meeting about the ongoing Eventlet removal work. Here are the details:

[public]Nova Eventlet removal sync

Every Wednesday: 14:30-15:00 UTC (16:30 – 17:00 CEST)

Video call link: https://meet.google.com/bcy-uqoz-hje

Cross Project Events

PTG Oct 2025: Tuesday 28 3pm UTC https://etherpad.opendev.org/p/oct2025-ptg-eventlet

Project-specific events

If your team or project schedule a specific event related to the Eventlet removal initiative do not hesitate to share your dates with the community.

Here are the coming project-specific events: - <none>

Want to Discuss?

#eventlet-removal

Follow the #eventlet-removal tag. It will be used everywhere to tag all the related things to this topic:

  • gerrit patches
  • mailing list threads
  • OpenStack OFTC IRC channel (#openstack-eventlet-removal)
  • ...

The Pop-up team

This community goal is going to be complex, for this reason It is imperative to take account of its impact on our community. That's the role of the pop-up team. We need a way to manage the community, and we need to establish measures which aim to improve the working condition of our community. That's the role of this pop-up team to provide such measures and conditions, while guaranteeing a good management of the community around this topic.

Our mantras

Our mantras are way to lead our decisions related to this community goal:

  1. It is crucial to avoid having multiple ways to manage things. A unified approach will simplify maintenance and enhance user experience.
  2. This is a community goal, so all community voices count. People should look at this goal with a global overview and not only with a team or a project centric point of view.
  3. Our decision should be driven by sustainability. We are in a way to drop a black box, we should not replace it by another black box.
  4. If a feature is by default included in the stdlib, then we must avoid relying on an esoteric third party library that provide the same kind of feature.

Our pillars

The elements below present the different perspectives that the pop-up should embody.

Collaborative governance

  • Establish consensus-based governance where major decisions are voted on after public consultations via proposals (blueprints, specs) or kind of RFCs;
  • Form a steering committee with representatives from different technical teams (nova, neutron, glance, swift, oslo, etc.) to ensure a balanced vision and prevent biases.

Code of conduct

Develop a code of conduct that defines respectful, inclusive, and productive collaboration rules. Enforce sanctions for harmful or disrespectful behavior to protect the integrity of the community.

Specialized working groups

Set up specialized working groups to choose and select the specific technical details, for example which python server to use, which library to use (FastAPI vs Flask, HTTPX vs Requests vs AIOHttp, etc...). For now we need working groups specialized:

  • HTTP decisions (server choice, application choice (fastapi, etc..), specific pattern definition);
  • Task management, threading vs others (Asyncio, awaitlet), for background tasks, deferred tasks, etc…;
  • General Patterns definition (timeout management; etc).

Those specialized working groups would be responsible to audit critical changes. Some people are more skilled on this or that specific technical point, these persons should, as far as possible, help to audit critical changes related to their center of interest. The finality of these specialized working groups is to define the global technical roadmap.

Find the list of specialized working groups here https://wiki.openstack.org/wiki/Eventlet-removal#Specialized_Working_Groups_2

Mentorship systems

  • Create a mentorship program to guide new or less experienced members. For example by providing some meetups related to the technical roadmap defined by the specialized working groups, to present a specific aspect of this migration;
  • Pair senior developers or principal engineers with each sub-project to ensure effective knowledge transfer. Encourage the creation of a role of technical mentor for each team responsible for spreading the knowledge around his teammates.

Transparency measures

The pop-up would be responsible to regularly publish progress reports on the project's advancement and resource usage (time, contributions, migrated components, bugs fixed); The pop-up would be responsible for creating a centralized dashboard to track the status of tasks and deadlines. The pop-up would be responsible for communicating regularly through IRC meetings and mailing list newsletters.

Decentralized management

Encourage sub-communities/teams (e.g., Glance, Cinder, etc.) to make independent decisions within their domains while following a shared global roadmap.

Workload regulation

  • Introduce "freeze" periods where development slows down to focus on stability, documentation, and training. For example, SLURP releases could be used to slow down our migration activity, and to be more focused on observation and documentation. SLURP releases are also a good occasion to deprecate things to plan their removal;
  • Use non SLURP releases to introduce breaking changes.


All These pillars are ways to be more efficient, enhance collaboration, and help avoid tensions or bottlenecks. More details about the pop-up team are available in the links below:

Specialized Working Groups

Specialized working groups are responsible for defining the various aspects of the global roadmap. The main role of these working groups is to avoid having multiple ways to manage things. The groups are responsible for producing a unified approach that will simplify maintenance and enhance user experience. Everybody is welcome. If you feel concerned about a specific aspect, then do not hesitate to show your interest on. Find below the specialized working groups identified for now, this list may varie in the coming months. If you feel some questions are aspects are missing, then feel free to ping us and to initiate the discussion.

The HTTP SGI Working Group

The role of this group is to define the right alternative to the WSGI features from Eventlet. That mean defining which:

The Tasks/Process Working Group

This working group will be responsible for defining solutions for each type of task in use in our deliverables. Deferred tasks, Background task, Service, Process, etc... One aspect of this is the current refactor of oslo.service, see https://review.opendev.org/c/openstack/oslo-specs/+/927503.

The Architecture, Interoperability and Technical Standardization Working Group

This working group will be responsible for defining the working models that best fits our needs. Where to put threads or where to put async. How to ensure that client and libraries remains compatible with all the services. The new oslo.db engine facade fall into this group. Discussions around the oslo.messaging executor fall into this group. How to manage timeout in an unified way, etc... This group will be responsible for defining the how to and the guidance.

Getting Started

Please visit the migration guide available at https://removal.eventlet.org You will find migration scenarios and snippets for alternatives like threading and asyncio.

Roles and Responsibilities

To help teams to manage the organisational aspects of the migration, we propose the following roles and responsibilities. OpenStack is maintained by several companies and organisations, the following structure aims to provide a minimal template for teams organisation to ensure that everyone is on the same page and have a clear understanding on how we work.

The Eventlet migration is complex, without a bit of structure your team might suffer of that migration. The future of your deliverables depends on it.

We strongly encourage teams to adopt this structure.

Technical Lead Role

The technical lead will be responsible for:

  • Supervision: Monitoring the technical progress of the migration within the team.
  • Support: Providing assistance to other developers and troubleshooting any technical issues related to the migration.
  • Reporting: Submitting regular progress reports, listing impacted deliverables, patterns of Eventlet identified, and participating in monthly meetings with the team product manager.
  • Technology Updates: Staying up-to-date on the latest technical news and updates related to the migration, and sharing this information with the team.
  • Coordination: Working closely with the product manager to ensure that the team has the necessary resources to complete the migration.
  • Communication: Acting as the main point of contact for questions or issues within the team regarding the migration process.

Product Manager’s Role

The product manager is responsible for coordinating and facilitating the Eventlet removal within their team. They ensure that the necessary resources and time are allocated for the transition and that progress is tracked regularly. Key Responsibilities:

  • Prioritization and Planning: Ensure the migration is prioritized in the team's roadmap and schedules. Set clear objectives and deadlines for each stage of the migration.
  • Support for the Team: Provide the technical lead and the team with the necessary resources and time to complete the migration. Address any organizational or logistical obstacles that might slow down progress.
  • Progress Monitoring: Review and monitor progress reports submitted by the technical lead. Ensure the team meets the deadlines for audits, migrations, and report submissions.
  • Coordination: Collaborate with other product managers to ensure a smooth and coordinated transition across all teams. Facilitate knowledge sharing between teams to optimize the migration process.
  • Communication: Attend monthly meetings with the technical lead to discuss progress, challenges, and any resource needs. Provide regular updates to stakeholders about the state of the migration.

Responsibilities For Developers

  • Task 1: Identify which Eventlet version is currently in use in your deliverables.You can simply inspect your requirements. You can also use beagle on all your deliverables, see the related sections.
  • Task 2: If the version is lower than 0.36.1, prepare the Eventlet requirement upgrade.
  • Task 3: Audit your code to identify all dependencies related to Eventlet within the next three weeks.
  • Task 4: Prepare a progress report that includes:
  1. A list of impacted deliverables;
  2. Eventlet patterns used within these deliverables (based on the patterns defined above).
  • Task 5: Join us at the PTG sessions and then lets discuss together about your use cases.
  • Task 6: Start migrating identified Eventlet components to available alternatives after the audit is complete. The target is to have this migration started within the current quarter.

Responsibilities For Product Managers

  • Task 1: Prioritize this migration in your team's roadmap. From easiest to most complex deliverables to migrate.
  • Task 2: Within the next two weeks, identify a technical lead (preferably a senior developer or principal senior developer) who will oversee the migration.
  • Task 3: Ensure that your teams have the necessary resources and time to complete the migration.
  • Task 4: Attend the coming PTG sessions and monthly follow-up meetings with the technical lead to discuss progress, blockers, and next steps.

Tracking Progress

First have a look to https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22. This page regroup all the patches related to the Eventlet removal initiative and labelled with the official topic (eventlet-removal).

Proposed Key Performance Indicators (KPI)

These KPIs aim to provide managers (product managers) with a way to track and measure progress, both from a team-based perspective and from a global overview.

1. Eventlet Deactivation Rate in Deliverables:

  • Objective: Track the progress of the number of deliverables where Eventlet has been completely disabled and replaced.
  • KPI: Percentage of deliverables where Eventlet is replaced by alternative solutions (`asyncio`, `awaitlet`, `concurrent.futures`, etc.).
  • Calculation: (Number of migrated deliverables / Total number of identified deliverables) x 100.


2. Reduction in the Number of `monkey_patch()` Occurrences:

  • Objective: Assess the gradual removal of calls to `eventlet.monkey_patch()`, which can cause unwanted behaviors.
  • KPI: Number of `monkey_patch()` occurrences still present in the code.
  • Calculation: Total number of `monkey_patch()` occurrences identified before and after each migration phase.


3. Response Time and Performance Post-Migration:

  • Objective: Verify whether the performance of the deliverables improves or remains stable after migrating to `asyncio` or other solutions.
  • KPI: Average response time of services post-migration (in ms).
  • Calculation: Measure the response time before and after migration to check if performance improves.


4. Percentage of Automated Tests Passed After Migration:

  • Objective: Ensure the stability and quality of the migrated code by verifying that unit and functional tests pass successfully.
  • KPI: Percentage of automated tests passed after migration.
  • Calculation: (Number of tests passed post-migration / Total number of tests executed) x 100.


5. Number of Bugs or Incidents Post-Migration:

  • Objective: Track the stability of the project post-migration by monitoring the number of bugs or incidents related to the migration.
  • KPI: Number of bugs or incidents identified related to the Eventlet migration.
  • Calculation: Total number of bugs/incidents opened during and after migration.


6. Adoption of New Python Versions:

  • Objective: Verify if the teams are able to adopt new Python versions in the migrated deliverables.
  • KPI: Percentage of deliverables using a recent version of Python (e.g., Python 3.12).
  • Calculation: (Number of deliverables using the latest Python version / Total number of deliverables) x 100.


7. Meeting Project Migration Deadlines:

  • Objective: Ensure that the migration project adheres to the initial schedule.
  • KPI: Number of deadlines met compared to the migration plan.
  • Calculation: (Number of deliverables migrated within the deadline / Total number of planned deliverables) x 100.

Useful Links

FAQ

Is this initiative is an official initiative?

Yes, this initiative is supported by the OpenStack TC. This initiative is an official OpenStack community goal. This goal has been accepted months ago and is now going to be a selected goal.

What will happen to eventlet once the openstack migration is complete?

Eventlet will be simply abandoned officially. No more maintenance will be given to the official Eventlet repository. The official Eventlet repository will be archived.

Why not simply maintaining Eventlet?

We have many reasons for that:

  1. The main reason is that, since years, to keep running Eventlet started to fork deprecated and removed versions of old CPython stdlib modules. With coming versions of Python this kind of workaround will surely have to repeat. This trend seems already coming with Python 3.13 and the threading module (see the discussion: https://github.com/eventlet/eventlet/pull/966#issuecomment-2344252826). We do not want to transform Eventlet into a poorly maintained fork of CPython. A poor full of security breaches. Moving away of that trend would almost require a complete rewrite of Eventlet.
  2. The second reason is more about resources. Eventlet is maintained by 1 or 2 part time core maintainers, that's not enough to keep its head above water, especially if we take account of the previous reason. We do not want to repeat the log4shell story.
  3. The third reason is that when the migration will be done, even though the migration was difficult given our deepest Eventlet usages, then we will be safe for a decades. We want to encourage a sustainable solution.

Teams efforts

Here is a list of documents/patches created by teams related to their migration. If your team own documents or patches and if you do not see them in the list below please feel free to add it. It could help other team to manage their own removal.

The list: 

   - https://etherpad.opendev.org/p/neutron-eventlet-deprecation
   - https://bugs.launchpad.net/neutron/+bugs?field.tag=eventlet-deprecation
  • Nova: 
   - https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22+project:openstack/nova
   - https://gibizer.github.io/categories/eventlet/


Else you can also have a look to the official gerrit tracker page https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22.

Map of the Impacted Deliverables

Here is a curated list of the Openstack deliverables currently (as of 2024 Oct. 4th) using Eventlet and which must be migrated:

Existing spec repos where Eventlet appear

Archives

Past PTG Events

Flamingo PTG

Eventlet cross-poject session: 

 - Date: 2025 8th April (Tuesday)
 - Etherpad: https://etherpad.opendev.org/p/apr2025-ptg-eventlet
 - Recording: https://www.youtube.com/watch?v=uwtDJpATPik & https://www.youtube.com/watch?v=Epq9EaePzIM

Epoxy PTG

TC/Cross-project session: 

  - Date: 2024 21st October (Monday)  
  - Time: 1500 UTC  
  - Meeting link: https://meetpad.opendev.org/oct2024-ptg-os-tc
  - Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-os-tc

Eventlet Removal Open Discussion Hours: 

  - Date: 2024 23rd October (Wednesday)  
  - Time: 1300-1500 UTC  
  - Meeting link: https://meetpad.opendev.org/oct2024-ptg-eventlet-removal
  - Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-eventlet-removal

Glance - Eventlet Removal Session 

 - Date: 2024 24th October (Thursday)
 - Time: 16:00-16:30 UTC
 - Meeting link: https://meetpad.opendev.org/oct2024-ptg-glance
 - Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-glance#L174
Retrieved from "https://wiki.openstack.org/w/index.php?title=Eventlet-removal&oldid=186871"