Assert Contains with list of ref values for metrics #1321
Description
It would be nice to have a list of reference values for a given metric as a means for testing status. For example let's have the following test, where we define x metric with value of 1 and we have a list of reference values [ 1, 2, 4, 8] for validation
buildspecs:
assert_contains_example:
type: script
executor: generic.local.bash
description: Test for assert equality
vars:
X: 1
run: |
echo "X: $X"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
status:
assert_contains:
- name: x
ref: [1, 2, 4, 8]
Similarly, we can have the inverse with assert_not_contains
buildspecs:
assert_not_contains_example:
type: script
executor: generic.local.bash
description: Test for assert equality
vars:
X: 1
run: |
echo "X: $X"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
status:
assert_not_contains:
- name: x
ref: [1, 2, 4, 8]
Some other ideas for assert_contains and assert_not_contains could be following:
in/not_inassert_in,assert_not_in