Change is the only constant...

A few months ago, after adding yet another DB into ProxySQL, we started to experience instance crash. The case was interesting and challenging due to following reasons:

Happening only with new DB type we had just added to ProxySQL for the 1st time – having already all the other DB types processing billions of queries per day through ProxySQL we never had such problems.
Hitting only after 2-3 hours of run time, where there is nothing obvious pointing to any specific processing happening around that point in time.

“Improbable” from the post title may be a bit of exaggeration, but as you will see what is needed to reproduce the problem, it is the most unlikely to happen bug I was dealing with so far 😁. Hence the idea to write down the summary of how the investigation was conducted – hopefully helping you with future troubleshooting sessions, as the risk that you will hit the same bug in your environment is fairly small (it was confirmed to happen at least in 8.0.28 and still existing in 8.0.35).

Before migrating from MySQL 5.7 to 8.0 I had created automated benchmarking framework which proved to give us repeatable results. It is based on sysbench and we perform set of runs with different threads from 16 to 256 with the step of 16, using OLTP Read Write workload and Percona’s TPCC-like one. I may write the other time a bit more on all the details and the process we are using, but in the context of this post the most important thing to know is that we can rely on its results and between different runs with the same conditions we were achieving less than 0.7% difference (averaged across runs for all threads, in terms of transactions per second).

Change is the only constant…

A short tale about troubleshooting improbable bug

Significant performance regression since MySQL 8.0.30 (up to 8.2.0 so far)