Skip to content

Conversation

@JR-1991
Copy link
Member

@JR-1991 JR-1991 commented Mar 1, 2024

Changes this PR introduces

This pull request adds async support to pyDataverse based on the previously merged switch from requests to httpx. Asynchronous requests significantly improve pyDataverse's performance by using concurrent requests. The implementation provides a seamless usage to work either sync or async by the usage of a context manager.

Small benchmark

  • 100 calls to api/info/version to demo.dataverse.org
  • Synchronous --> ~35s
  • Asynchronous --> ~0.5s 🏎️

Overview

  • Added _sync_request and _async_request base methods
  • Added Context manager to API to enable async requests
  • Small test to verify async requests are working

Example

Synchronous

native_api = NativeApi("https://demo.dataverse.org")
response = native_api.get_info_version()

Asynchronous

import asyncio
from pyDataverse.api import NativeApi


async def run_async(native_api):
    async with native_api:
        tasks = [native_api.get_info_version() for _ in range(100)]
        responses = await asyncio.gather(*tasks)

if __name__ == "__main__":
    native_api = NativeApi("https://demo.dataverse.org")
    asyncio.run(run_async(native_api))

@JR-1991 JR-1991 added type:feature New feature pkg:api api related activities prio:medium status:for review Waiting to be reviewd labels Mar 1, 2024
@JR-1991 JR-1991 self-assigned this Mar 1, 2024
@pdurbin pdurbin mentioned this pull request Mar 4, 2024
47 tasks
@pdurbin pdurbin added this to the 0.3.2 milestone Mar 4, 2024
Copy link

@haeussma haeussma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the example with python=3.11.

Notices missing dependency urllib3 which is not specified in the requirements but imported in src/pyDataverse/__init__.py.

Except for that the example works as expected.

Copy link

@haeussma haeussma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the example with python=3.11.

Notices missing dependency urllib3 which is not specified in the requirements but imported in src/pyDataverse/__init__.py.

Except for that the example works as expected.

@JR-1991
Copy link
Member Author

JR-1991 commented Apr 22, 2024

@haeussma thanks for the review! I have added urllib3 to the list of dependencies.

@JR-1991 JR-1991 merged commit 072acb7 into master Apr 24, 2024
@JR-1991 JR-1991 deleted the async-requests branch August 14, 2025 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pkg:api api related activities prio:medium status:for review Waiting to be reviewd type:feature New feature

Projects

Development

Successfully merging this pull request may close these issues.

4 participants