Skip to content

RFC: integration with pytest(-benchmark) #567

Open

Description

@nzjrs

I've got a relatively large project with many benchrmarks written as pytest fixtures and benchmarks collected with pytest-benchmark

Would it be feasible to use pytest to discover and execute these benchmarks? In particular, I find the operation of @pytest.mark.parameterize to work very nicely with pytest-benchmark and allow easy comparative bench-marking. Do you think it is feasible to integrate with pytest in this way?

Activity

  1. pv commented on Oct 29, 2017

    @pv
    Collaborator

    Feasible in principle, yes.

    However, at the moment I suspect it's not a good idea to burden the development with supporting multiple benchmark runners. If you need integration now, a simpler solution is to whip up a script that converts pytest-benchmark json files to asv json --- that's probably doable in a few hours.

  2. nzjrs commented on Oct 30, 2017

    @nzjrs
    Author

    cool, is the format of the asv json files defined somewhere?

  3. klauer commented on Jul 23, 2018

    @klauer

    While it's very far from perfect, I worked on some code that allowed for generation of asv-compatible json using pytest-benchmark. Here's some links:

    Example generated from pytest-benchmark+asv,
    ugly asv/pytest-benchmark bridge code, and
    required lines in conftest

    If there's some interest, maybe this could be cleaned up and put in a package of its own.

  4. added
    ideaLow-priority enhancement suggestion
    on Jun 1, 2019
  5. removed
    enhancementTriaged as an enhancement request
    on Jun 30, 2019
  6. 1heart commented on Oct 22, 2021

    @1heart

    For those searching, it looks like RapidsAI has some sort of pytest-benchmark + asv solution using a CLI-based asv "database" viewer.

  7. vyasr commented on Nov 4, 2021

    @vyasr

    @pv I work on RAPIDS and my colleague @rlratzel is the maintainer of the asvdb package mentioned in the previous comment. Our use case is basically what's been described here: we have a large test suite already written with different tools (including pytest-benchmark) and we wanted to take advantage of asv's awesome viewer features without having to rewrite our entire test suite. Our solution is basically what you proposed a few comments ago on this thread. We developed asvdb as a programmatic means for interacting with asv's JSON database, and we separately built the rapids-pytest-benchmark pytest plugin that builds on top of pytest-benchmark and asvdb to allow running a pytest-benchmark and immediately dumping to asv by adding an extra command line argument.

    I agree that full pytest-benchmark integration with asv is probably out of scope for asv at this stage. However, I assume that asv already has some sort of internal machinery for working with its JSON format that probably overlaps with our asvdb. Would you be open to maybe exposing some of those internals or allowing us to upstream asvdb into asv so that there's a canonical programmatic method for interacting with asv's JSON "database"? Having that feature baked directly into asv would be preferable to having it reside in a third-party, not officially supported package, and it's likely that you have internals that asvdb is largely replicating in order to write these files. If you're interested in, we could collaborate on getting asvdb (or a subset of it) merged into asv. We'd be happy to maintain the rapids-pytest-benchmark plugin separately since the actual pytest-benchmark->asv conversion seems out of scope for asv and would be straightforward to maintain if the asv-writing functionality was part of asv.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ideaLow-priority enhancement suggestion

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions