We use the word repro with two different meanings:

1. It is a pytest marker:

Current markers include:

repro: All available reproducibility tests (all repro_ test markers but repro_determinism_restart).

2. It is a !test keyword:

Usage is as follows:

!test TYPE [commit]

Where TYPE is a test suite that we support. Currently, this consists of repro.

However, the meaning is different. In the second case, it runs the tests configured in that repository as reproducability tests, e.g. the ci.json could say:

"reproducibility": {
        "default": {
            "markers": "repro and (not slow)"
        },
        "dev-MC_100km_jra_ryf": {
            "markers": "repro"
        },
        "dev-MC_100km_jra_ryf+wombatlite": {
            "markers": "repro_historical and repro_determinism"
        },
}

So - when users run the !test repro command in a comment, they might run the tests which use the pytest marker repro or they might not. Users can't run !test repro_determinism.

I think this is confusing!

@CodeGat @minghangli-uni @aidanheerdegen