Skip to content

servers.List() pagination is extremely inefficient #3867

Closed

Description

@majewsky

What happened?

I was investigating remarkably bad performance in a component that runs github.com/gophercloud/gophercloud/v2/openstack/compute/v2/servers.List(), and the profiling showed that the problem is with how pagination is implemented inside of it. It unmarshals the response body no less than four times:

  1. once into map[string]any during pagination.PageResultFrom() for use in generic NextPageURL/IsEmpty implementations (which do not end up being used)
  2. once in ServersPage's custom NextPageURL implementation
  3. once in ServersPage's custom IsEmpty implementation
  4. once in ServersPage.Extract() to get the actual []servers.Server

While this is clearly suboptimal, usually it's not all that bad because most JSON responses from OpenStack are not all that large. But servers.List() contains server objects which not only have an extremely large number of attributes, some of those attributes are also genuinely huge (esp. userdata).

For my own usecase, I mitigated the problem by using a custom pagination implementation that bypasses the standard types from package pagination entirely. I don't think that it will be possible for the standard pagination implementation to be as efficient (because of how func Extract always does a separate unmarshal), but it might be possible to avoid the three separate unmarshals for NextPageURL and IsEmpty.

Version

v2 (current)

Activity

  1. added theissue type on Jul 22, 2026
  2. changed the title [-]servers.ListDetail() pagination is extremely inefficient[/-] [+]servers.List() pagination is extremely inefficient[/+] on Jul 22, 2026
  3. blacksoxx commented on Oct 11, 2026

    @blacksoxx
    Contributor

    Measured on main with an EachPage handler that calls ExtractServers: each page is serialised three times and parsed four times, since IsEmpty and NextPageURL both go through ExtractInto, which re-encodes the body PageResultFrom already decoded. Reading servers and servers_links from the decoded map removes those two round trips, ExtractServers stays the only full decode. PR coming for the servers package; the generic decode in PageResultFrom stays since AllPages depends on it.

  4. added a commit that references this issue on Nov 12, 2026
    f36aacc
  5. added a commit that references this issue on Nov 12, 2026
    5aa2f49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions