test

Spglib test-suite

The main Spglib test-suite is a standalone CMake project that can be executed on a pre-installed Spglib package or as part of the main Spglib project. On top of this, there are language-dependent test-suites, e.g. pytest for the Python bindings, and a tmt test layer that orchestrates all tests across each test-suite.

Requirements

CMake >= 3.11
Googletest (downloaded via FetchContent if not available)
Optional:
- tmt: to execute the tmt tests locally. Currently only Fedora-like systems are supported
- [test] python optional dependency: to execute python tests

Folder structure

We organize the test-suite as follows:

unit: Unit tests that test small individual components of the code-base
integration: Integration tests that test the interplay between the units and the external dependencies. (Currently the api binding tests are not covered here, but in functional)
functional: Functional tests that test the physical correctness and how the user would actually interact with the library
example: Standalone example project tests. These are symlinked to ../example folder
package: Packaging tests that check the supported methods that the project can be imported

Ctest layer

All tests are either implemented within the ctest test-suite layer or they are indirectly called by it. There can be discrepancy in what and how the tests are executed, e.g. the project source being tested in package tests, and these differences are being covered by the tmt test layer.

Tests on the ctest layer are implemented in a few methods:

as Googletest individual tests: unit, functional, integration
as standalone CMake projects invoked by ctest --build-and-test: example, package
as standalone test-suite invoked directly by ctest: pytest

How to build the ctest project

There are two ways of building and executing the ctest test-suite:

Build as part of the main Spglib project with SPGLIB_WITH_TESTS=ON. This tests the main Spglib project that is being built
Build as standalone project, i.e. running the typical cmake workflow in this test folder
```
$ cmake -B ./build
$ cmake --build ./build
$ ctest --test-dir ./build
```
This tests the Spglib project that is already installed on the system via find_package, or that FetchContent points to

How to run specific tests

You can find a full list of available tests by running:

$ ctest --test-dir ./build -N

The -N can also be used to check the effect of the test filtering by regex/labels.

The tests are also organized by labels which you can find a list of using:

$ ctest --test-dir ./build --print-labels

To select specific tests, you can either use regex -R (simplest being just the name of the test), or the labels -L:

$ ctest --test-dir ./build -R "SymmetrySearch.test_spg_get_symmetry"
$ ctest --test-dir ./build -L unit_tests

How to add tests

First you have to consider what is being tested and if it is a unit/functional/integration etc. test. After that, depending on the type of test you either need to:

Add or expand a <gtest_testsuite_name>.cpp file with appropriate link in the folder's CMakeLists.txt file. The content of the <gtest_testsuite_name>.cpp looks like:

#include <gtest/gtest.h>

extern "C" {
  #include "spglib.h"
}

TEST(<gtest_testsuite_name>, <test_name>) {
    // ASSERT_EQ(a, b)
    // ASSERT_NE(a, b)
    // ASSERT_TRUE(a)
}

Add an example CMake project. It should be buildable by itself with optional/minimal integrated tests. Use the tmt test layer to do more complex test checks.
Add or expand a language-specific test. Make sure the test-suite is included as an add_test inside ctest

Other notes

VSCode setting

Although Googletest is automatically prepared, you may need to install it locally to highlight structs and functions.

Below is an example of c_cpp_properties.json for Mac.

{
    "configurations": [
        {
            ...
            "includePath": [
                "${workspaceFolder}/**",
                "${workspaceFolder}/cmake-build-release-gcc/_deps/googletest-src/googletest/include",
                "${workspaceFolder}/cmake-build-release-llvm/_deps/googletest-src/googletest/include"
            ],
            ...
        }
    ]
}

The launch.json below runs a single test with a debugger:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Launch(lldb): #381",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/build/test/test_suite",
            "args": ["--gtest_filter=MagneticDataset.test_with_slightly_distorted_positions"],
            "cwd": "${workspaceFolder}/build/test",
        },
    ]
}

Utility functions for preparing structures

When you find a doubtful cell with Python API, the following python snippets will be useful to prepare a test case for Python testing and C testing.

def print_python_cell(cell):
    if len(cell) == 3:
        lattice, positions, numbers = cell
        magmoms = None
    elif len(cell) == 4:
        lattice, positions, numbers, magmoms = cell

    contents = []

    # lattice
    contents.append("lattice = np.array([")
    for i in range(3):
        contents.append(f"    [{lattice[i][0]:.8f}, {lattice[i][1]:.8f}, {lattice[i][2]:.8f}],")
    contents.append("])")

    # positions
    contents.append("positions = np.array([")
    for i in range(len(positions)):
        contents.append(
            f"    [{positions[i][0]:.8f}, {positions[i][1]:.8f}, {positions[i][2]:.8f}],"
        )
    contents.append("])")

    # numbers
    contents.append("numbers = np.array([" + ", ".join(map(str, numbers)) + "])")

    # magmoms
    if magmoms:
        contents.append("magmoms = np.array([")
        for i in range(len(positions)):
            contents.append(f"    [{magmoms[i][0]:.8f}, {magmoms[i][1]:.8f}, {magmoms[i][2]:.8f}],")
        contents.append("])")

    print("\n".join(contents))

def print_cpp_cell(cell):
    if len(cell) == 3:
        lattice, positions, numbers = cell
        magmoms = None
    elif len(cell) == 4:
        lattice, positions, numbers, magmoms = cell

    contents = []

    # lattice
    lattice_col = lattice.T
    contents.append("double lattice[3][3] = {")
    for i in range(3):
        contents.append(
            f"    {{ {lattice_col[i][0]:.8f}, {lattice_col[i][1]:.8f}, {lattice_col[i][2]:.8f} }},"
        )
    contents.append("};")

    # positions
    contents.append("double positions[][3] = {")
    for i in range(len(positions)):
        contents.append(
            f"    {{ {positions[i][0]:.8f}, {positions[i][1]:.8f}, {positions[i][2]:.8f} }},"
        )
    contents.append("};")

    # numbers
    contents.append("int types[] = {" + ", ".join(map(str, numbers)) + "};")

    # magmoms
    if magmoms:
        contents.append("double spins[] = {")
        for i in range(len(positions)):
            contents.append(f"    {magmoms[i][0]:.8f}, {magmoms[i][1]:.8f}, {magmoms[i][2]:.8f},")
        contents.append("};")

    contents.append(f"int num_atoms = {len(positions)};")

    print("\n".join(contents))

Exposing API for unit tests

Normally all functions outside of the exported spglib.h are not exposed, but for the purposes of unit testing, we need access to these. Currently, we don't know a clear way of resolving this issue, but we know two ways of addressing this:

Building Spglib as a static library for linking to the test-suite
Use a macro SPG_API_TEST that expands to SPG_API when building unit tests

We have opted for the latter approach for now, so keep in mind when writing unit tests to include SPG_API_TEST directive.

Tmt test-suite

The tmt test layer is primarily for orchestrating and running all the tests in arbitrary environments, e.g. in the CI, post packaging, in the Fedora-CI, etc. Although this test-suite is currently limited to fedora-like environments, there are plans to expand this to arbitrary systems soon.

The tmt tests-suite is composed of plans (a collection of tests with a common preparation step) and individual tests, all of which are defined in the .fmf files.

Introspecting tmt tests/plans

After installing tmt from PyPI or other sources, you can do basic inspection of the tests/plans, even if the OS does not support actual execution of the tests.

The .fmf files are simply yaml files with folder structure inheritance, and to view the actual generated yaml-like content use:

$ fmf show --path ./

:::{note} The --path in fmf is equivalent to the --root in tmt and it points to the fmf tree root (the folder that contains .fmf/version file) that you wish to introspect. :::

To view the actual plans/tests use:

$ tmt --root ./ tests show
$ tmt --root ../.distro plans show

Executing tests

The tmt tests are primarily executed within the testing-farm environment through packit or other downstream like Fedora-CI. But you can also interact with it directly, e.g.

$ tmt --root ../.distro run --all plans --name full

:::{note} The tests are currently assuming Spglib is already installed on the system. Standalone plans that compile Spglib as a preparation step is yet to be designed. :::

You can alter these executions by adding individual steps, e.g. if you want to change the provision step so that you run in a container you can add a provision step in the middle like:

$ tmt --root ../.distro run --all provision --how container --image=fedora:rawhide plans --name full

Designing tmt tests/plans

At the bare minimum, you can simply design a test that executes the test-suite, see pytest.fmf for example. For more complex testing we use beakerlib, which basically provides a structured way of executing bash commands and check the results and stdout. These tests are found as <test>.sh, the simplest reference being that of ctest.sh.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

Spglib test-suite

Requirements

Folder structure

Ctest layer

How to build the ctest project

How to run specific tests

How to add tests

Other notes

VSCode setting

Utility functions for preparing structures

Exposing API for unit tests

Tmt test-suite

Introspecting tmt tests/plans

Executing tests

Designing tmt tests/plans

Resources

Name		Name	Last commit message	Last commit date
parent directory ..
.fmf		.fmf
example		example
functional		functional
integration		integration
package		package
unit		unit
CMakeLists.txt		CMakeLists.txt
README.md		README.md
ctest.fmf		ctest.fmf
ctest.sh		ctest.sh
pytest.fmf		pytest.fmf
utils.c		utils.c
utils.h		utils.h

FilesExpand file tree

test

Directory actions

More options

Directory actions

More options

Latest commit

History

test

Folders and files

parent directory

README.md

Spglib test-suite

Requirements

Folder structure

Ctest layer

How to build the ctest project

How to run specific tests

How to add tests

Other notes

VSCode setting

Utility functions for preparing structures

Exposing API for unit tests

Tmt test-suite

Introspecting tmt tests/plans

Executing tests

Designing tmt tests/plans

Resources