Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Autcomplete for ClickHouse CLI based on a small transformer and several markov models

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

I will add full documentation in the future in this PR. For now:

It is an autocomplete system that predicts the next token of user input in ClickHouse CLI. On a high level, it works like this:

There is a transformer that predicts the type of the next token (Literal, Operator, Identifier or it can predict Keywords like SELECT, etc). Then on each of the types (Literal, Operator, Identifier), there is a dedicated markov model that predicts the value of the token itself. Also, there is a Markov Model for not preprocessed tokens, just for bare queries. If the latter markov model is highly sure about the next word (p>0.8) its prediction is placed on top of the predictions of the other machinery.
This approach allows us to predict words with markov model that we are very sure of and if we are not sure it allows us NOT to predict nonsense (because we are bound to the specific type of token).

Here is a short video (this is an old one, it works a bit better now).

I will be updating this PR description and the code because it is still ongoing. There will be a very verbose description of the system a bit later as this is part of my course work and I need to do it anyway :).

Please have a look and if you have any suggestions/comments I will be happy to update the code/explain my choices.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

• Allow: All Required Checks
• Allow: Stateless tests
• Allow: Stateful tests
• Allow: Integration Tests
• Allow: Performance tests
• Allow: All Builds
• Allow: batch 1, 2 for multi-batch jobs
• Allow: batch 3, 4, 5, 6 for multi-batch jobs

• Exclude: Style check
• Exclude: Fast test
• Exclude: All with ASAN
• Exclude: All with TSAN, MSAN, UBSAN, Coverage
• Exclude: All with aarch64, release, debug

• Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
• Exclude: AST fuzzers

• Do not test
• Woolen Wolfdog
• Upload binaries for special builds
• Disable merge-commit
• Disable CI cache