Make cursor opaque to the client #418

What problem does your feature solve?

We are currently debating a potential change to the event ID encoding, due to the need to correctly order the different arrays of events that are coming out of core: #414

Currently, id is utilized by the getEvents API as a pagination cursor, so it is user-facing and derivable/interpretable by end-users. If we were to change the encoding logic, this presents two problems:

• Mid-flight requests during deploys will fail in unexpected ways. For example, imagine a client requests events and is holding a cursor value. RPC then deploys a new version, with the new event_id encoding. The client then requests the next page of events by passing in the cursor value. Depending on how the encoding changes, this could either return wrong/confusing results, or just error out in some way.
• Clients may be deriving their own cursors. Currently, id is returned in the events response, and it's clear that this is the same value as the cursor. It's possible that clients have identified this and thus may have built logic into their application assuming they can grab any id from any response and pass it in as the cursor.

Other cursor/pagination problems that this could solve include:

• Clients may send different filters/parameters on subsequent requests that contain a cursor, which could lead to confusing results. Clients also must drop startLedger/endLedger parameters to pass API validation when a cursor is passed, so it's an easy mistake to make that they also drop additional filters/parameters.
• The endLedger if passed, is effectively "lost" on the second page of a pagination session. For example, if I query events with startLedger=1, endLedger=1, limit=5, and there are 6 events in ledger 1, then my second request may be cursor=<x>, limit=5 and will include events from ledger 2.

What would you like to see?

Change our cursor values to be opaque to the end-user, and encode information about the query parameters of the original request in said cursor.

For example, the cursor could simply be a base64 encoded string representation of the following key/value pairs:

{
    version: 1,
    startLedger: ...,
    endLedger: ...,
    filters: ....,
    cursor: ...
}

This gives us a few advantages:

• Makes us forwards-compatible against any changes we may need to make to the cursor values in the future. Application code could translate cursor versions across deployment boundaries. For example, if the event_id encoding switched between v1/v2, RPC could still accept v1 cursors for a time but only return v2 cursors, ensuring that clients that are mid-stream/mid-pagination are not disrupted when RPC deploys happen. It would also be possible to do this when introducing this change, as we know if the cursor looks like an old event_id encoding that it's an old cursor version.
• Discourages end-users from assigning or deriving meaning from our cursor/event id
• Removes the need for the client to pass in all of their filter parameters on every single request. If they have a live cursor they effectively have a stateful scroll id that they can use to continue their scrolling session.
• Allows endLedger to be used.

What alternatives are there?

• Do nothing. It's probably unlikely that we'd need to change the event ID encoding more than once. And even if we do, it's just a temporary breakfast for clients paginating across version updates. A client that had an old cursor would just have to backtrack to the most recent ledger and start paging again from there. We should ensure these kinds of requests throw some error or are otherwise easily identifiable to the end-user, rather than returning real (incorrect) results.