Use Python's unittest for validating bundles and configurations? #98
Description
Currently we're using Go, which means it's hard to build a single validator that can process multiple versions of the spec, because you'd have to use tedious interface{} manipulation or load separate packages with structures for each supported version.
We're also not using a test suite, which means it's hard to test multiple aspects of the configuration and report on several errors (the current code just dies on the first error).
By using Python's unittest, we get more convenient handling of generic JSON and an established test framework that doesn't need a lot of boilerplate. I've started working up this approach here if folks want to kick the tires. Examples of potentially tricky things and how this approach lets us handle them easily:
- Skipping particular tests if the configuration has an unrecognized version (wking/oci-runtime-config-validator@c607380f).
- Applying the Windows-specific mount-nesting restriction (here).
- Applying the 0.5.0-specific (and previous, but I haven't bothered supporting them) relative
root.pathrequirement (here).
Example compact output with the current tip (wking/oci-runtime-config-validator@cd0facf0a. I still haven't finished process, later entries in config.md or anything from the platform-specific files):
$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest
...s.............
----------------------------------------------------------------------
Ran 17 tests in 0.006s
OK (skipped=1)
and verbose output:
$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest -v
test_configuration (test.test_bundle.TestBundle)
config.json MUST reside in the root of the bundle directory. ... ok
test_root (test.test_bundle.TestBundle)
The bundle directory MUST contain the root filesystem. ... ok
test_destination (test.test_mounts.TestMounts)
destination (string, required). ... ok
test_destination_nesting (test.test_mounts.TestMounts)
Mount destinations MUST not be nested within another mount. ... skipped 'the destination-nesting restriction only applies to Windows for specification version 1.0.0-rc1.'
test_options (test.test_mounts.TestMounts)
options (list of strings, optional). ... ok
test_source (test.test_mounts.TestMounts)
source (string, required). ... ok
test_type (test.test_mounts.TestMounts)
type (string, required). ... ok
test_args (test.test_process.TestProcess)
args (array of strings, required). ... ok
test_cwd (test.test_process.TestProcess)
cwd (string, required). ... ok
test_env (test.test_process.TestProcess)
env (array of strings, optional). ... ok
test_process (test.test_process.TestProcess)
process (object, required). ... ok
test_terminal (test.test_process.TestProcess)
terminal (bool, optional). ... ok
test_path (test.test_root.TestRoot)
path (string, required). ... ok
test_readonly (test.test_root.TestRoot)
readonly (bool, optional). ... ok
test_syntax (test.test_syntax.TestSyntax)
All configuration JSON MUST be encoded in UTF-8. ... ok
test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version. ... ok
test_semantic_version (test.test_version.TestVersion)
ociVersion (string, required) MUST be in SemVer v2.0.0 format. ... ok
----------------------------------------------------------------------
Ran 17 tests in 0.007s
OK (skipped=1)
And with an unrecognized version:
$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest
.sssssssssssss.F.
======================================================================
FAIL: test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/wking/src/opencontainers/oci-runtime-config-validator/test/test_version.py", line 39, in test_recognized_version
.format(util.VERSION))
AssertionError: '1.0.0-rc2' not found in ['1.0.0-rc1', '0.5.0'] : Unrecognized configuration version. Either your configuration does not match an OCI specification or the test suite has not been taught to process the version you are using.
----------------------------------------------------------------------
Ran 17 tests in 0.002s
FAILED (failures=1, skipped=13)
$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest -v
test_configuration (test.test_bundle.TestBundle)
config.json MUST reside in the root of the bundle directory. ... ok
test_root (test.test_bundle.TestBundle)
The bundle directory MUST contain the root filesystem. ... skipped 'cannot validate an unrecognized version'
test_destination (test.test_mounts.TestMounts)
destination (string, required). ... skipped 'cannot validate an unrecognized version'
test_destination_nesting (test.test_mounts.TestMounts)
Mount destinations MUST not be nested within another mount. ... skipped 'cannot validate an unrecognized version'
test_options (test.test_mounts.TestMounts)
options (list of strings, optional). ... skipped 'cannot validate an unrecognized version'
test_source (test.test_mounts.TestMounts)
source (string, required). ... skipped 'cannot validate an unrecognized version'
test_type (test.test_mounts.TestMounts)
type (string, required). ... skipped 'cannot validate an unrecognized version'
test_args (test.test_process.TestProcess)
args (array of strings, required). ... skipped 'cannot validate an unrecognized version'
test_cwd (test.test_process.TestProcess)
cwd (string, required). ... skipped 'cannot validate an unrecognized version'
test_env (test.test_process.TestProcess)
env (array of strings, optional). ... skipped 'cannot validate an unrecognized version'
test_process (test.test_process.TestProcess)
process (object, required). ... skipped 'cannot validate an unrecognized version'
test_terminal (test.test_process.TestProcess)
terminal (bool, optional). ... skipped 'cannot validate an unrecognized version'
test_path (test.test_root.TestRoot)
path (string, required). ... skipped 'cannot validate an unrecognized version'
test_readonly (test.test_root.TestRoot)
readonly (bool, optional). ... skipped 'cannot validate an unrecognized version'
test_syntax (test.test_syntax.TestSyntax)
All configuration JSON MUST be encoded in UTF-8. ... ok
test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version. ... FAIL
test_semantic_version (test.test_version.TestVersion)
ociVersion (string, required) MUST be in SemVer v2.0.0 format. ... ok
======================================================================
FAIL: test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/wking/src/opencontainers/oci-runtime-config-validator/test/test_version.py", line 39, in test_recognized_version
.format(util.VERSION))
AssertionError: '1.0.0-rc2' not found in ['1.0.0-rc1', '0.5.0'] : Unrecognized configuration version. Either your configuration does not match an OCI specification or the test suite has not been taught to process the version you are using.
----------------------------------------------------------------------
Ran 17 tests in 0.003s
FAILED (failures=1, skipped=13)
This approach would also likely work fine in other languages and test frameworks that make it easy to handle generic JSON. I'm familiar with Python, but if the OCI community prefers a different language, I'm game to try. So:
- Does this sound like a useful direction? If so,
- Is it worth pulling into ocitools? Go is a reasonable language for compiled binaries like
runtimetest(certainly a better choice for that than Python), and multi-language repositories can get awkward. The validation tasks (configuration validation and runtime validation) seem to decouple well, so I'd rather not stuff both into a single repository. But I can live with a single repo if it's the maintainer preference ;).