With some advice from PlanetScale I believe I've matched the logic vtgate uses to find tablets, minus the logic for checking replication lag (Freno will do this) and minimum tablet count 👍

Path I followed through the code:

1. https://github.com/vitessio/vitess/blob/master/go/vt/vtgate/tabletgateway.go *TabletGateway.withRetry(...)
2. https://github.com/vitessio/vitess/blob/master/go/vt/discovery/healthcheck.go *HealthCheckImpl.GetHealthyTabletStats(...)
3. https://github.com/vitessio/vitess/blob/master/go/vt/discovery/replicationlag.go *FilterStatsByReplicationLag(...)
   • This new logic aims to copy this block: https://github.com/vitessio/vitess/blob/master/go/vt/discovery/replicationlag.go#L113-L119

This looks to be right. Mind you that I'm not that familiar with realtime stats in Vitess.

After some testing with Vitess the logic in this PR won't work as it stands. When a node is considered "unhealthy" by Vitess, the following is returned by the vtctld API:

$ curl -ks https://<vtctld hostname>/api/keyspace/test_ks/tablets/?cells=dc1 \
    | jq '.[] | select( .hostname == "<hostname>" ).stats'
{
  "realtime": {
    "seconds_behind_master": 30
  },
  "serving": false,
  "up": true
}

This response was gathered from a node that exceeds the -discovery_high_replication_lag_minimum_serving=# threshold

While Vitess won't send reads to a replica in serving: false, this behaviour means the logic in this PR will cause Freno to ignore nodes that lag beyond -discovery_high_replication_lag_minimum_serving=#, hiding the lag vs throttling when replication is overwhelmed because valid lagging nodes can become serving: false

This means:

1. This PR needs to consider serving: false AND serving: true for Freno probes
2. There will still be cases where a broken node can cause Freno to throttle (although the replica is not serving in Vitess):
   • Replication SQL thread hits an error
   • Replication was running but is stopped later on (eg: STOP SLAVE, restart of mysqld, etc)

In my testing vtctld didn't "notice" situations where replication is broken after previously being healthy, only the replication lag caused by the problem is reported as seconds_behind_master without a last_error set. This could mean that in some situations we're unable to differentiate a node with "broken" replication from a valid "lagging" one

If Vitess periodically updated the Realtime stats last_error field with the health of the SQL replication thread, this would allow us to know when replication is broken vs overwhelmed with writes. Currently replication is not running is returned on a new tablet that has never been "seen" replicating, but not when a previously-healthy tablet hits a problem

cc @tomkrouper / @shlomi-noach for thoughts

Re-requesting review from @shlomi-noach, @drogart and @tomkrouper