Skip to content

clarinet fuzz command and heterogeneous test-suites #398

@moodmosaic

Description

@moodmosaic

This was started as two separate GitHub issues, but then I thought I should merge them. — The idea is:

clarinet test can support property tests, and those tests may be written in either TypeScript or Clarity
clarinet fuzz can then turn those tests into fuzz tests

Context

Clarinet tests are essentially Deno tests. This has the interesting side-effect of being able to use what's already available in JS/TS ecosystem when testing Clarity code. (For example, fast-check and dspec can be used, as we've done here with @LNow.)

A typical workflow is declaring functions in Clarity and then writing tests in TypeScript. Clarinet can then discover those tests (with the help of Deno) and run them. It would be practical however to also discover tests written in Clarity and run those as well.

Discoverability

In addition to existing test suite(s) in TypeScript, Clarinet can execute as a test any Clarity function that meets the following criteria:

  • The method is public
  • The method name starts with test (configurable in Clarinet.toml)

(If a public function named beforeEach is present it can be executed before each test is run. Such a function can exist in the style of testing @jcnelson does here, may be discussed also on a separate thread, however.)

type prefix arguments semantics
concrete test no single execution with concrete values
property test yes multiple executions with randomly generated concrete values (smaller values)
fuzz test yes multiple executions with randomly generated concrete values (biased)*

*Also configurable, in Clarinet.toml.

Fuzz mode can be enabled via new clarinet fuzz command. Essentially it can work the same as clarinet test but use a different fast-check configuration.

Libraries

Both test suites (TypeScript, and Clarity (hypothetically)) can be discovered and run from the context of Deno, and in the case of property tests and fuzz tests, the generated data can be provided by fast-check.

I have been using (and getting an inside look into) fast-check recently. It has all the modern features of a prop/fuzz testing library, e.g. model testing, integrated shrinking, control over the scope of generated values, and many other useful functions.

Pros and cons

What are the advantages and disadvantages of having the option to write tests in Clarity?

  • One disadvantage that @LNow pointed out to me is that this can screw the execution costs.
  • An advantage can be that there's less context switching; you write your contract in Clarity and your tests in Clarity as well. (No need to marshal stuff between different languages (as done here for example).
  • The possibility to encode (some of the) smart contract audits in code that (continuously) runs and checks those tests, instead of having all the audits based on a specific git commit and PDF file.
  • (more pros and cons to be added)...

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions