Skip to content

Commit ff951bd

Browse files
nikitawootten-nistwendellpiezaj-stein-nist
committed
Feature profile resolution unittesting a (#1770)
* Profile resolution unit tests and specification updates (#1219) * Updates to tests including whitespace scrub test * Adding oXygen project * Integration work on specs and unit tests / XSLT and Schematron * Profile resolution testing infrastructure, plus some tests * Some test files with expected results * Updated and improved XSpec generation and tests - now driven directly from SpecML source * Adjusted gitignore; extended spec with more bindings; illustrator XSLT * Added XSLT producing Markdown summary of requirements w/ examples; updates * Added a small utility for converting JSON "prop" fields to "props" fields (emended 1.0.2 syntax) * Updated to Profile Resolution spec (with example files now tagged) and readme.md * Updated profile resolution (spec) readme with clarification regarding unit testing * Adding XSLT provisionally to produce a unit testing harness after @david-waltermire-nist model, with example and schema; also cleanup * Removed working file included by accident * More unit tests with edits to spec for tracking. * Catching up various outstanding profile resolution work * Small update to oXygen setup * Update metaschema submodule to existing commit. * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754. * XSpec test suites script & CI (#1754) * XSpec test suites script & CI * Helper script to build and tag the OSCAL env * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754. * Save XSpec test summary + pull appropriate tag * Escape TEST_SUITE in XSpec runner Co-authored-by: A.J. Stein <[email protected]> * Update test script to address shellcheck guidance * Grab branch from GHA * Fix for builds on the M1 --------- Co-authored-by: A.J. Stein <[email protected]> * Fixed logic for when a tag does not exist --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>
1 parent d43713a commit ff951bd

File tree

124 files changed

+11985
-239
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

124 files changed

+11985
-239
lines changed

‎.github/workflows/xspec-tests.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: OSCAL XSpec Test Suite
2+
on:
3+
push:
4+
branches:
5+
- main
6+
- develop
7+
- "feature-*"
8+
- "release-*"
9+
paths:
10+
- /src
11+
- "**.xsl"
12+
- "**.xpl"
13+
- "**.xspec"
14+
pull_request:
15+
branches:
16+
- main
17+
- develop
18+
- "feature-*"
19+
- "release-*"
20+
paths:
21+
- /src
22+
- "**.xsl"
23+
- "**.xpl"
24+
- "**.xspec"
25+
workflow_call: {}
26+
jobs:
27+
xspec-tests:
28+
name: Run XSpec tests
29+
runs-on: ubuntu-22.04
30+
steps:
31+
- uses: actions/checkout@v3
32+
with:
33+
submodules: true
34+
- name: Login to Docker Hub
35+
uses: docker/login-action@v2
36+
with:
37+
username: ${{ secrets.DOCKERHUB_USERNAME }}
38+
password: ${{ secrets.DOCKERHUB_TOKEN }}
39+
- name: Pull the correct image
40+
shell: bash
41+
# Produces a tagged image oscal-common-env:selected
42+
run: ./build/pull-oscal-env-dockerfile.sh "${{ github.head_ref || github.ref_name }}"
43+
- name: Run XSpec tests
44+
shell: bash
45+
run: |
46+
set -o pipefail # propagate return code
47+
docker run \
48+
-v $(pwd):/oscal \
49+
-e TEST_DIR=/oscal/xspec \
50+
oscal-common-env:selected \
51+
/oscal/src/utils/util/resolver-pipeline/testing/test.sh \
52+
| tee summary.csv || {
53+
if [ "$?" = 83 ]; then
54+
# For now we only fail when tests fail to compile
55+
echo "Some test suites failed to compile, failing..."
56+
exit 1
57+
fi
58+
}
59+
- name: Upload artifacts
60+
uses: actions/upload-artifact@v3
61+
with:
62+
name: xspec-output
63+
path: |
64+
xspec/*.html
65+
summary.csv

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,5 @@ node_modules/
5555
/docs/scratch-dir
5656
/.skipbuild
5757
/.runbuild
58+
59+
/summary.csv

‎build/build-oscal-env-dockerfile.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/usr/bin/env bash
2+
# Build and tag the oscal-common-env dockerfile
3+
#
4+
# By default the tag is the sanitized branch name, but can be overidden
5+
# by an argument.
6+
7+
set -Eeuo pipefail
8+
9+
IMAGE="csd773/oscal-common-env"
10+
BRANCH=$(git branch --show-current)
11+
BRANCH_SANITIZED=${BRANCH/\//_}
12+
13+
TAG="${1:-$BRANCH_SANITIZED}"
14+
15+
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
16+
17+
docker build \
18+
--label "branch=${BRANCH}" \
19+
--label "commit_sha=$(git rev-parse HEAD)" \
20+
--label "dirty=$(git diff --quiet && echo 'false' || echo 'true')" \
21+
--label "[email protected]" \
22+
--label "author=$(git config user.email)" \
23+
--platform linux/amd64 \
24+
-f "$SCRIPT_DIR/Dockerfile" \
25+
-t "$IMAGE:$TAG" \
26+
"$SCRIPT_DIR"
27+
28+
echo "Built and tagged $IMAGE:$TAG, to push run:"
29+
echo " docker push $IMAGE:$TAG"

‎build/pull-oscal-env-dockerfile.sh‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/env bash
2+
# Pull the oscal-common-env from the registry using a defined set of tags, and
3+
# retag the first working image with a new tag.
4+
# This is intended to be used in CI/CD environments where rebuilding the
5+
# oscal-common-env would be expensive, but an escape hatch to allow for a
6+
# "special" image for a run is preferred.
7+
8+
set -Eeuo pipefail
9+
10+
# Allow the user to override the "branch" name (note that this still sanitizes the input)
11+
if [ "${1-}" ]; then
12+
BRANCH=${1}
13+
else
14+
BRANCH=$(git branch --show-current)
15+
fi
16+
17+
# Docker tags cannot have "/" in them
18+
SANITIZED_BRANCH=${BRANCH/\//_}
19+
20+
TAGS=("${SANITIZED_BRANCH}" "develop")
21+
22+
SOURCE_IMAGE="csd773/oscal-common-env"
23+
24+
# the output image and tag to write to
25+
OUTPUT_IMAGE="oscal-common-env"
26+
OUTPUT_TAG="selected"
27+
OUTPUT_REF="${OUTPUT_IMAGE}:${OUTPUT_TAG}"
28+
29+
for TAG in "${TAGS[@]}"; do
30+
REF="${SOURCE_IMAGE}:${TAG}"
31+
docker pull "${REF}" && {
32+
docker tag "${REF}" "${OUTPUT_REF}"
33+
echo "Successsfully pulled ${REF} and retagged it as ${OUTPUT_REF}"
34+
exit 0
35+
} || echo "Pulling tag ${REF} failed..."
36+
done
37+
38+
echo "Failed to pull any images in"
39+
exit 1

‎src/release/content-upgrade/json-prop-to-props.xsl‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"
3+
xpath-default-namespace="http://www.w3.org/2005/xpath-functions"
4+
xmlns="http://www.w3.org/2005/xpath-functions">
5+
6+
<!--
7+
Command line invocation:
8+
9+
$ java -jar $SAXON_HOME/saxon-he-10.2.jar -xsl:json-prop-to-props.xsl -it json-input=YOUR.json
10+
11+
where YOUR.json is your JSON file and the jar file is Saxon 10 or later.
12+
13+
The process reads the file at this URL, parses it as JSON, maps it through a filter, and reports it back.
14+
15+
-->
16+
17+
<xsl:output method="text"/>
18+
19+
<xsl:mode on-no-match="shallow-copy"/>
20+
21+
<!-- The input JSON file -->
22+
<xsl:param name="json-input" select="'url to json file'"/>
23+
24+
<!-- The initial template that process the JSON -->
25+
<xsl:template name="xsl:initial-template">
26+
<xsl:variable name="mapped-result">
27+
<xsl:apply-templates select="json-to-xml(unparsed-text($json-input))"/>
28+
</xsl:variable>
29+
<!--<xsl:copy-of select="$mapped-result"/>-->
30+
<xsl:value-of select="xml-to-json($mapped-result)"/>
31+
</xsl:template>
32+
33+
<xsl:template match="array[@key='prop']">
34+
<array key="props">
35+
<xsl:apply-templates/>
36+
</array>
37+
</xsl:template>
38+
39+
</xsl:stylesheet>

‎src/specifications/profile-resolution/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
!oscal-specs_share.xpr
2+
requirement-tests/output-actual/

‎src/specifications/profile-resolution/example-set.xspec‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!-- autogenerated 2021-12-14T16:02:57.305-05:00 following model in example-set.xspec-->
3+
<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec"
4+
xmlns:opr="http://csrc.nist.gov/ns/oscal/profile-resolution"
5+
xmlns:o="http://csrc.nist.gov/ns/oscal/1.0"
6+
stylesheet="../../utils/util/resolver-pipeline/oscal-profile-RESOLVE.xsl"
7+
run-as="external"
8+
xmlns:ov="http://csrc.nist.gov/ns/oscal/test/variable">
9+
<x:helper stylesheet="lib/oscal-profile-test-helper.xsl"/>
10+
<!-- old tests are out of whack need update to current functionality /spec -->
11+
<!-- but the XSpec provides a functional model -->
12+
<x:scenario label="Base test">
13+
<!-- Alternative model for test scrubbing both the result and the expected
14+
result for comparison -->
15+
<x:context href="profile-resolution-examples/base-test_profile.xml"/>
16+
<x:variable name="ov:expected-output"
17+
href="profile-resolution-examples/output-expected/base-test_profile_RESOLVED.xml"/>
18+
<x:expect label="Base test - fully resolved"
19+
test="opr:scrub($x:result)" select="opr:scrub($ov:expected-output)"/>
20+
</x:scenario>
21+
22+
<x:scenario label="Testing base-test_profile.xml">
23+
24+
<x:context href="profile-resolution-examples/base-test_profile.xml"/>
25+
<x:expect label="Resolution of base-test_profile.xml"
26+
select="opr:scrub(.)"
27+
href="profile-resolution-examples/output-expected/base-test_profile_RESOLVED.xml"/>
28+
</x:scenario>
29+
<x:scenario label="Testing base2-test_profile.xml">
30+
<x:context href="profile-resolution-examples/base2-test_profile.xml"/>
31+
<x:expect label="Resolution of base2-test_profile.xml"
32+
select="opr:scrub(.)"
33+
href="profile-resolution-examples/output-expected/base2-test_profile_RESOLVED.xml"/>
34+
</x:scenario>
35+
<x:scenario label="Testing broken_profile.xml" pending="galtm">
36+
<x:context href="profile-resolution-examples/broken_profile.xml"/>
37+
<x:expect label="Resolution of broken_profile.xml"
38+
select="opr:scrub(.)"
39+
href="profile-resolution-examples/output-expected/broken_profile_RESOLVED.xml"/>
40+
</x:scenario>
41+
<x:scenario label="Testing circular_profile.xml">
42+
<x:context href="profile-resolution-examples/circular_profile.xml"/>
43+
<x:expect label="Resolution of circular_profile.xml"
44+
select="opr:scrub(.)"
45+
href="profile-resolution-examples/output-expected/circular_profile_RESOLVED.xml"/>
46+
</x:scenario>
47+
<x:scenario label="Testing exclude-call-test_profile.xml">
48+
<x:context href="profile-resolution-examples/exclude-call-test_profile.xml"/>
49+
<x:expect label="Resolution of exclude-call-test_profile.xml"
50+
select="opr:scrub(.)"
51+
href="profile-resolution-examples/output-expected/exclude-call-test_profile_RESOLVED.xml"/>
52+
</x:scenario>
53+
<x:scenario label="Testing full-test_profile.xml">
54+
<x:context href="profile-resolution-examples/full-test_profile.xml"/>
55+
<x:expect label="Resolution of full-test_profile.xml"
56+
select="opr:scrub(.)"
57+
href="profile-resolution-examples/output-expected/full-test_profile_RESOLVED.xml"/>
58+
</x:scenario>
59+
<x:scenario label="Testing home_profile.xml">
60+
<x:context href="profile-resolution-examples/home_profile.xml"/>
61+
<x:expect label="Resolution of home_profile.xml"
62+
select="opr:scrub(.)"
63+
href="profile-resolution-examples/output-expected/home_profile_RESOLVED.xml"/>
64+
</x:scenario>
65+
<x:scenario label="Testing import-twice_profile.xml">
66+
<x:context href="profile-resolution-examples/import-twice_profile.xml"/>
67+
<x:expect label="Resolution of import-twice_profile.xml"
68+
select="opr:scrub(.)"
69+
href="profile-resolution-examples/output-expected/import-twice_profile_RESOLVED.xml"/>
70+
</x:scenario>
71+
<x:scenario label="Testing include-all-no-children-test_profile.xml">
72+
<x:context href="profile-resolution-examples/include-all-no-children-test_profile.xml"/>
73+
<x:expect label="Resolution of include-all-no-children-test_profile.xml"
74+
select="opr:scrub(.)"
75+
href="profile-resolution-examples/output-expected/include-all-no-children-test_profile_RESOLVED.xml"/>
76+
</x:scenario>
77+
<x:scenario label="Testing include-all-test_profile.xml">
78+
<x:context href="profile-resolution-examples/include-all-test_profile.xml"/>
79+
<x:expect label="Resolution of include-all-test_profile.xml"
80+
select="opr:scrub(.)"
81+
href="profile-resolution-examples/output-expected/include-all-test_profile_RESOLVED.xml"/>
82+
</x:scenario>
83+
<x:scenario label="Testing include-call-with-children-test_profile.xml">
84+
<x:context href="profile-resolution-examples/include-call-with-children-test_profile.xml"/>
85+
<x:expect label="Resolution of include-call-with-children-test_profile.xml"
86+
select="opr:scrub(.)"
87+
href="profile-resolution-examples/output-expected/include-call-with-children-test_profile_RESOLVED.xml"/>
88+
<x:expect label="Includes grandparent in result" test="$x:result//o:control/@id = 'c3'"/>
89+
<x:expect label="Includes grandchild in result" test="$x:result//o:control/@id = 'c3.a-1'"/>
90+
<x:expect label="Is missing a control not included" test="not( $x:result//o:control/@id = 'c2' )"/>
91+
</x:scenario>
92+
<x:scenario label="Testing include-loose-param-test_profile.xml">
93+
<x:context href="profile-resolution-examples/include-loose-param-test_profile.xml"/>
94+
<x:expect label="Resolution of include-loose-param-test_profile.xml"
95+
select="opr:scrub(.)"
96+
href="profile-resolution-examples/output-expected/include-loose-param-test_profile_RESOLVED.xml"/>
97+
</x:scenario>
98+
<x:scenario label="Testing include-match-test_profile.xml">
99+
<x:context href="profile-resolution-examples/include-match-test_profile.xml"/>
100+
<x:expect label="Resolution of include-match-test_profile.xml"
101+
select="opr:scrub(.)"
102+
href="profile-resolution-examples/output-expected/include-match-test_profile_RESOLVED.xml"/>
103+
</x:scenario>
104+
<x:scenario label="Testing merge-implicit-keep_profile.xml">
105+
<x:context href="profile-resolution-examples/merge-implicit-keep_profile.xml"/>
106+
<x:expect label="Resolution of merge-implicit-keep_profile.xml"
107+
select="opr:scrub(.)"
108+
href="profile-resolution-examples/output-expected/merge-implicit-keep_profile_RESOLVED.xml"/>
109+
</x:scenario>
110+
<x:scenario label="Testing merge-keep-resources_profile.xml">
111+
<x:context href="profile-resolution-examples/merge-keep-resources_profile.xml"/>
112+
<x:expect label="Resolution of merge-keep-resources_profile.xml"
113+
select="opr:scrub(.)"
114+
href="profile-resolution-examples/output-expected/merge-keep-resources_profile_RESOLVED.xml"/>
115+
</x:scenario>
116+
<x:scenario label="Testing merge-keep_profile.xml">
117+
<x:context href="profile-resolution-examples/merge-keep_profile.xml"/>
118+
<x:expect label="Resolution of merge-keep_profile.xml"
119+
select="opr:scrub(.)"
120+
href="profile-resolution-examples/output-expected/merge-keep_profile_RESOLVED.xml"/>
121+
</x:scenario>
122+
<x:scenario label="Testing modify-adds_profile.xml">
123+
<x:context href="profile-resolution-examples/modify-adds_profile.xml"/>
124+
<x:expect label="Resolution of modify-adds_profile.xml"
125+
select="opr:scrub(.)"
126+
href="profile-resolution-examples/output-expected/modify-adds_profile_RESOLVED.xml"/>
127+
</x:scenario>
128+
</x:description>

0 commit comments

Comments
 (0)