fix(349)!: export plain models and nested lists, refuse model unions by Thanatos9404 · Pull Request #414 · awslabs/stickler

Thanatos9404 · 2026-10-16T18:36:04Z

Pull Request

Issue Reference

Refs #349 (parts B and C).

Not in this PR: part A is #407, the List[Dict] row is #405, and the singular Dict row is left to #360, which already exports a mapping field as an object on both paths.

Description of Changes

to_json_schema() and to_stickler_config() fell back to "string" / "str" for any annotation missing from their type tables, and the guard meant to stop a nested model reaching that fallback checked isinstance(field_type, type), which is False for Union[Card, Bank]. For the shapes below, each exporter now writes a type its importer reads back as one that holds the same data, or raises:

shape	what both exporters write now
plain `BaseModel`, or a list of them	free-form `{"type": "object"}` / `"dict"` (`List[dict]` for a list), re-imported as `dict` with the declared comparator, threshold, weight and clip
list of lists, including `List[List[Dict]]`	nested arrays on the JSON path; the config keeps writing `List[List]`
union with a model arm (`Union[Card, Bank]`, `Union[Plain, str]`), also as a list element or inside `Optional` / `Annotated`	`ValueError` on both exporters
StructuredModel inside a nested list (`List[List[Card]]`)	`ValueError` on both exporters

Changes Made

structured_model.py: the StructuredModel guard moves into _refuse_scalar_export(), which strips Optional and Annotated at every level and also refuses a union with a model arm and a StructuredModel inside a nested list. In to_json_schema(), a list of lists or of plain models still goes through the primitive list branch, and its items is then replaced by _json_type_schema() (nested arrays, objects for mappings and plain models). Scalar elements, scalar unions and mapping elements are untouched. In to_stickler_config(), a list of plain models is written List[dict]; it was List[<ClassName>], which model_from_json() refused with Unknown type.
json_schema_field_converter.py: field_to_property() and field_to_stickler_config() call that guard, and write a plain model as "object" / "dict".
New tests/structured_object_evaluator/test_export_roundtrip_shapes.py, 45 tests.
docs/docs/Advanced/model-export.md: one paragraph under the "at a glance" table naming what these shapes export as, what is not kept, and what raises.
CHANGELOG.md: one **Breaking:** entry under [Unreleased] / Changed.

Why These Changes Are Needed

The issue's model, one field per model so the union does not stop the others. e403b9b is dev, 8ade791 is this branch; "loads" means the re-imported model accepts the issue's DATA value.

field	`to_json_schema()` on dev	on this branch	`to_stickler_config()` on dev	on this branch
`child: Plain`	`string`, ValidationError	`object`, loads	`str`, ValidationError	`dict`, loads
`either: Union[Card, Bank]`	`string`, ValidationError	raises `ValueError`	`str`, ValidationError	raises `ValueError`
`nested: List[List[float]]`	items `string`, ValidationError	items `array` of `number`, loads	`List[List]`, loads	unchanged
`meta: Dict[str, Any]`	`string`, ValidationError	unchanged (left to #360)	`str`, ValidationError	unchanged (left to #360)
`rows: List[Dict[str, str]]`	items `string`, ValidationError	unchanged (left to #405)	`List[Dict]`, loads	unchanged
`kid: Card`	`object`, loads	unchanged	`structured_model`, loads	unchanged

field_to_property() on the issue's C table: Union[Card, Bank] went from {"type": "string"} to ValueError, Plain from "string" to "object", and Card still raises. The schema-only loop from part C (from_json_schema(S).to_json_schema() with the two-arm anyOf) wrote "string" on dev and then could not reload {"iban": "DE89"}; it now raises at the export. List[List[Dict[str, str]]], which the #405 review pointed here, now round-trips on the JSON path and scores the same (0.9706 before and after).

I also exported 40 annotations on both trees and compared the exported type / items and whether the re-import loads a sample value. 11 changed: four plain-model spellings (Plain, Optional[Plain], List[Plain], List[Optional[Plain]]), four lists of lists, and three unions with a model arm, which now raise. The other 29 are unchanged, including every mapping spelling, every scalar (date, Decimal, UUID, Enum, Literal, Any, bytes), Card, List[Card], List[Dict[str, str]], Union[str, int], List[Union[str, int]], Tuple and Set.

Two choices worth flagging:

A plain BaseModel exports as an object, not a raise. The acceptance list asks the guard to cover it. On e403b9b, raising there fails 10 tests in TestTheExportedConfigurationMatchesTheEngine, which pin that to_json_schema() reports the comparator the engine uses for a plain-model field. The engine scores a plain model the way it scores a mapping (is_object_grade_annotation), so a free-form object reloads the data and scores the same, which the tests check. Its own field types are not exported. If you would rather it raise, it is a small change plus those 10 tests.
A model union, and a StructuredModel inside a nested list, raise on both paths. JSON Schema could express the union: on dev, a hand-built anyOf of Card.to_json_schema() and Bank.to_json_schema() re-imports as Union[Card, Bank] and scores the same. A model inside a nested list would also need importer work: a nested array of Card.to_json_schema() re-imports as List[List[DynamicModel]], without Card's name or settings. The Stickler config has no type for either (its List[List] re-scored List[List[Card]] from 0.9857 to 0.9706 on dev), so the JSON path raises too, to keep one rule for both exports. An anyOf export for the union can follow if you want it.

Testing

Unit tests added/updated
Integration tests added/updated
Manual testing completed
All existing tests pass (apart from the 6 tests that need the POSIX-only signal.SIGALRM, details below)

Windows, Python 3.12.5, uv sync --frozen --extra llm, pytest tests -n 6. inspect.getfile(stickler) pointed at the tree named in each row.

	dev `e403b9b`	branch `8ade791`
`test_export_roundtrip_shapes.py` (45)	38 failed, 7 passed	45 passed
full suite	2909 passed, 6 failed, 10 skipped, 1 xfailed	2954 passed, 6 failed, 10 skipped, 1 xfailed

The 6 failures are the same on both: test_realistic_deep_nesting.py, AttributeError: module 'signal' has no attribute 'SIGALRM'.

On dev, the new tests fail for the reported reasons: 22 because nothing raised (nine union spellings and the nested-model case on both exporters, the schema-only loop, field_to_property()); 6 because the config wrote List[Plain], which model_from_json() refuses; 2 with ValidationError on the re-imported model; and 8 type assertions on "string" / "str" / List[Plain]. The 7 that pass on dev are guards: the JSON re-import resolves the same comparator, threshold, weight and clip for each of the five fields (the extensions already survived; only the type was wrong), the config still writes List[List] for a list of lists (the documented limit), and field_to_property() still refuses a bare StructuredModel.

The round-trip test has one value wrong in every field; the original and both re-imports give the same field scores and confusion matrix. test_value_types_are_not_exported pins the documented limit (a plain model with n: int coerces "5"; the re-imported dict keeps "5").

ruff check and ruff format --check pass on the three changed Python files; ruff check . reports the same 16 findings on both trees, all in examples/. mkdocs build succeeds on both trees with the same 9 warnings.

Reviewers

@sromoam

Checklist

Code follows the project's coding standards
Self-review of code completed
Documentation updated (if applicable)
CHANGELOG.md entry added under [Unreleased], or not needed because
nothing user-visible changed (see RELEASING.md)
Tests added/updated for new functionality
All CI checks are passing
Ready for review

Additional Notes

Other open PRs. Merged locally on top of this branch, fix(333): export a List[Dict] element as an object so the schema loads its own data #405, fix(json-schema)!: read model keys on ["object","null"] and check $defs bodies (#340, #349 part A) #407 and fix(importers): resolve an unstated threshold and clip as the Python path does (#334, #348) #408 merge without conflicts, and this branch with fix(333): export a List[Dict] element as an object so the schema loads its own data #405 and fix(json-schema)!: read model keys on ["object","null"] and check $defs bodies (#340, #349 part A) #407 passes the full suite (3007 passed, the same 6 SIGALRM failures). fix(306): accept default_factory on ComparableField #360 conflicts only in the package README, as it already does with dev; this branch keeps off the lines fix(306): accept default_factory on ComparableField #360 rewrites for mappings, and with fix(306): accept default_factory on ComparableField #360 merged every list shape still exports its default_factory=list default on both paths, because the new list shapes share the primitive list branch.
Behaviour to know about:
- A plain-model field declared without ComparableField (child: Optional[Plain] = None) exports no x-aws-stickler-* keys, so it re-imports with the importer's defaults (ExactComparator at 1.0) and a one-of-two-wrong pair scores 0.0 instead of 0.5. On dev this export could not reload the data at all. Same cause as the bare-field note in fix(333): export a List[Dict] element as an object so the schema loads its own data #405 (Comparator, threshold and clip resolve differently on the JSON Schema and Stickler-config paths than on the Python path, so #300 and #316 landed on one entry point each #348).
- A plain model inside a nested list comes back as a dict, which the engine scores slightly differently there: List[List[Plain]] went from 0.9762 to 0.9773 in my test pair on both paths (the config path already did this on dev).
Not changed here: List[Union[Dict, str]] (the other shape the fix(333): export a List[Dict] element as an object so the schema loads its own data #405 review pointed here) and scalar unions such as Union[str, int] still export as "string"; to_stickler_config() still writes List[Optional[X]] as List[X] and a list of lists without its inner type.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

to_json_schema() and to_stickler_config() fell back to "string" / "str" for any annotation missing from their type tables. A plain BaseModel, a list of plain models and a list of lists all took that fallback, so the re-imported field was str (or List[str], or an unknown type in the config) and could not load the data the original model accepted. The guard meant to stop a nested model reaching the fallback checked `isinstance(field_type, type)`, which is False for Union[Card, Bank], so that union exported as "string" too. from_json_schema() builds exactly that union from a multi-arm anyOf, so a schema-only round trip broke. - A plain BaseModel is scored as one object, as a mapping is, so it exports as a free-form {"type": "object"} ("dict" in a Stickler config, List[dict] for a list) and re-imports as dict with its comparison settings. Its own field types are not exported. - A list of lists exports as nested arrays, so List[List[float]] and List[List[Dict]] reload. The element goes through the existing primitive list branch, so that branch stays the one place that writes a list property. - to_stickler_config() has no type for a union with a model arm, or for a StructuredModel inside a nested list, so both exporters raise ValueError for these and follow one rule. JSON Schema could express the union (anyOf), and that export could follow; a model inside a nested list would also need importer work. Optional and Annotated wrappers are stripped at every level before the check. Mapping fields and List[Dict] elements are left alone here: the first is handled in awslabs#360, the second in awslabs#405. Refs awslabs#349 (parts B and C)

sromoam

Thanks @Thanatos9404. Refusing model unions instead of silently writing "string" is the right call, and the nested-array export is a real improvement. I measured everything below on 7faefc0 with the import pinned to the PR tree.

Two things to fix, both about valid models that now fail to export:

Annotated around a nested model is refused. The export dispatch (structured_model.py:1847) strips Optional but not Annotated. A supported nested field therefore skips the recursive branch and reaches _refuse_scalar_export. For example, card: Optional[Annotated[Card, Field(description="d")]] (how pydantic stores Annotated[Card, Field(...)] | None) and cards: List[Annotated[Card, "m"]] both raise "Cannot export Card as a scalar" on both exporters. Running ConfigurationHelper.strip_annotation_wrappers in the dispatch, for both the field and the list element, should send them down the recursive path.
A single-level list is called a nested list. _refuse_scalar_export sets nested=True after removing one list level (:2011), but field_to_property and field_to_stickler_config pass it the whole field annotation. So items: Optional[Annotated[List[Card], Field(description="d")]] raises "has no type for a StructuredModel inside a nested list". Please add these two shapes to test_export_roundtrip_shapes.py. They're what let this slip through.

Not blocking, but please narrow the CHANGELOG and test docstring claim ("neither exporter writes a structured field as a scalar"). The check doesn't look inside container arms or containers yet:

Union[List[Card], str] and Union[Dict[str, Card], str] still export as "string".
List[List[Dict[str, Card]]] and Tuple[Card, ...] don't raise.
A List[List[Union[str, int]]] leaf still falls back to "string" (:2054).

I'll file those as a follow-up so this PR doesn't grow. The helper consolidation (_json_type_schema and the four plain-model overrides) can wait too.

Both exporters stripped Optional but not Annotated before dispatching, so a nested model declared as Optional[Annotated[Card, Field(...)]] or List[Annotated[Card, "m"]] skipped the recursive branch and was refused as a scalar. The field and the list element now go through strip_annotation_wrappers, so they export as the model, and an annotated scalar exports as its own type instead of a string. _refuse_scalar_export now takes the field's whole annotation and counts list levels, so a single List[Card] is no longer reported as a nested list. Both list branches pass it the whole field. The CHANGELOG entry and the test docstring now say which shapes the refusal does not reach yet. Refs awslabs#349

Thanatos9404 · 2026-10-19T18:24:05Z

Thanks for measuring this so carefully. Both fixes are in bcc24de: the export dispatch now runs strip_annotation_wrappers on the field and on the list element, so Optional[Annotated[Card, Field(...)]], List[Annotated[Card, "m"]] and Optional[Annotated[List[Card], Field(...)]] take the recursive path on both exporters, and _refuse_scalar_export now gets the whole field annotation and only calls two list levels a nested list. Those three shapes are in test_export_roundtrip_shapes.py, and as a side effect an annotated scalar such as Optional[Annotated[int, Field(...)]] now exports as integer / int instead of "string" (one test pins that). I also narrowed the CHANGELOG entry and the test docstring with your examples of what the refusal does not look inside yet.

sromoam

Thanks @Thanatos9404, this is exactly what was needed. I re-measured on bcc24de with the import pinned to the PR tree.

Optional[Annotated[Card, Field(description="d")]], List[Annotated[Card, "m"]] and Optional[Annotated[List[Card], Field(description="d")]] now take the recursive path on both exporters. Each round-trips through from_json_schema() and model_from_json() with the same scores as the original.
Having _refuse_scalar_export count list depth on the whole annotation is a cleaner rule than before, and the single-level test pins it.
Nearby shapes (Optional[Card], List[Optional[Card]], outer Annotated, nested primitive lists, plain models) still round-trip, and the model-union and nested-list refusals still refuse. The suite passes on its own and merged with dev.

The narrowed CHANGELOG reads well, and I'll track the container cases in #426.

One heads-up, no action needed here: this overlaps with #405 in the to_stickler_config() list branch. Whichever lands second keeps #405's List[Dict] mapping branch first, then your plain-model dict fallback.

Approving. Thanks again for the careful follow-up.

github-actions · 2026-10-19T21:24:51Z

Security Scan Results

ASH Security Scan Report

Report generated: 2026-10-19T21:24:26+00:00
Time since scan: 3 minutes

Scan Metadata

Project: ASH
Scan executed: 2026-10-19T21:21:05+00:00
ASH version: 3.1.2

Summary

Scanner Results

The table below shows findings by scanner, with status based on severity thresholds and dependencies:

Severity levels:
- Suppressed (S): Findings that have been explicitly suppressed and don't affect scanner status
- Critical (C): Highest severity findings that require immediate attention
- High (H): Serious findings that should be addressed soon
- Medium (M): Moderate risk findings
- Low (L): Lower risk findings
- Info (I): Informational findings with minimal risk
Duration (Time): Time taken by the scanner to complete its execution
Actionable: Number of findings at or above the threshold severity level that require attention
Result:
- PASSED = No findings at or above threshold
- FAILED = Findings at or above threshold
- MISSING = Required dependencies not available
- SKIPPED = Scanner explicitly disabled
- ERROR = Scanner execution error
Threshold: The minimum severity level that will cause a scanner to fail
- Thresholds: ALL, LOW, MEDIUM, HIGH, CRITICAL
- Source: Values in parentheses indicate where the threshold is set:
  - global (global_settings section in the ASH_CONFIG used)
  - config (scanner config section in the ASH_CONFIG used)
  - scanner (default configuration in the plugin, if explicitly set)
Statistics calculation:
- All statistics are calculated from the final aggregated SARIF report
- Suppressed findings are counted separately and do not contribute to actionable findings
- Scanner status is determined by comparing actionable findings to the threshold

Scanner	Suppressed	Critical	Low	Actionable	Result	Threshold
bandit	0	3	5928	3	FAILED	MEDIUM (global)
cdk-nag	0	0	0	0	PASSED	MEDIUM (global)
cfn-nag	0	0	0	0	MISSING	MEDIUM (global)
checkov	0	0	0	0	PASSED	MEDIUM (global)
detect-secrets	0	1	0	1	SKIPPED	MEDIUM (global)
grype	0	0	0	0	MISSING	MEDIUM (global)
npm-audit	0	0	0	0	PASSED	MEDIUM (global)
opengrep	0	0	0	0	MISSING	MEDIUM (global)
semgrep	3	0	0	0	PASSED	MEDIUM (global)
syft	0	0	0	0	MISSING	MEDIUM (global)

Top 2 Hotspots

Files with the highest number of security findings:

Finding Count	File Location
2	tests/comparators/test_anls_leaf_semantics.py
2	tests/structured_object_evaluator/test_rollup_node_semantics.py

Detailed Findings

Show 4 actionable findings

Finding 1: SECRET-HEX-HIGH-ENTROPY-STRING

Severity: HIGH
Scanner: detect-secrets
Rule ID: SECRET-HEX-HIGH-ENTROPY-STRING
Location: tests/comparators/test_anls_leaf_semantics.py:89

Description:
Secret of type 'Hex High Entropy String' detected in file 'tests/comparators/test_anls_leaf_semantics.py' at line 89

Code Snippet:

Secret of type Hex High Entropy String detected

Finding 2: B108

Severity: HIGH
Scanner: bandit
Rule ID: B108
Location: tests/comparators/test_anls_leaf_semantics.py:210-212

Description:
Probable insecure usage of temp file/directory.

Code Snippet:

uuid.UUID(int=1),
            pathlib.Path("/tmp/x"),
            b"bytes",

Finding 3: B307

Severity: HIGH
Scanner: bandit
Rule ID: B307
Location: tests/structured_object_evaluator/test_rollup_node_semantics.py:1362-1368

Description:
Use of possibly insecure function - consider using safer ast.literal_eval.

Code Snippet:

def published_label(node: dict, section: str, object_lists: set) -> str:
            counts_objects = eval(  # noqa: S307
                cond.group(1),
                {},
                {"data": node, "section": section, "OBJECT_LISTS": object_lists},
            )
            return eval(unit.group(1), {}, {"counts_objects": counts_objects})  # noqa: S307

Finding 4: B307

Severity: HIGH
Scanner: bandit
Rule ID: B307
Location: tests/structured_object_evaluator/test_rollup_node_semantics.py:1367-1369

Description:
Use of possibly insecure function - consider using safer ast.literal_eval.

Code Snippet:

)
            return eval(unit.group(1), {}, {"counts_objects": counts_objects})  # noqa: S307

Report generated by Automated Security Helper (ASH) at 2026-10-19T21:24:27+00:00

… optional elements nullable Review follow-ups on #405: - docs: a list of lists exports nested arrays since #414; drop "or another list" from the string-export sentence. - to_stickler_config() writes List[Optional[Dict]] for an optional mapping element, so the re-imported model accepts None elements again. - A declared ComparableField whose annotation resolves only at model_rebuild() exported Levenshtein with clip on while the engine scored ANLS* with clip off (0.95 vs 0.5 after the round trip). Mapping lists now always export get_comparison_info(), declared or not, and the stale comparator_config is dropped.

Thanatos9404 requested a review from sromoam as a code owner October 16, 2026 18:36

sromoam requested changes Oct 19, 2026

View reviewed changes

sromoam added this to the 1.0.1 (Minor) milestone Oct 19, 2026

This was referenced Oct 19, 2026

Schema export: a model inside a container union arm, dict or tuple still exports as a scalar or free-form object #426

Open

fix(333): export a List[Dict] element as an object so the schema loads its own data #405

Merged

sromoam approved these changes Oct 19, 2026

View reviewed changes

sromoam merged commit d90c7f6 into awslabs:dev Oct 19, 2026
8 checks passed

sromoam mentioned this pull request Oct 30, 2026

release(v1.0.1): HTML report security fixes and scoring corrections #473

Closed

7 tasks

digalaviz mentioned this pull request Nov 2, 2026

release(v1.0.1): HTML report security fixes and scoring corrections #475

Merged

1 of 7 tasks

Navigation Menu

Repository navigation

fix(349)!: export plain models and nested lists, refuse model unions - #414

fix(349)!: export plain models and nested lists, refuse model unions#414
sromoam merged 2 commits into
awslabs:devfrom
Thanatos9404:bugfix/349-roundtrip-union

Thanatos9404 commented Oct 16, 2026

Uh oh!

sromoam left a comment

Uh oh!

Thanatos9404 commented Oct 19, 2026

Uh oh!

sromoam left a comment

Uh oh!

github-actions Bot commented Oct 19, 2026

Finding 1: SECRET-HEX-HIGH-ENTROPY-STRING

Finding 2: B108

Finding 3: B307

Finding 4: B307

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Thanatos9404 commented Oct 16, 2026

Pull Request

Issue Reference

Description of Changes

Changes Made

Why These Changes Are Needed

Testing

Reviewers

Checklist

Additional Notes

Uh oh!

sromoam left a comment

Choose a reason for hiding this comment

Uh oh!

Thanatos9404 commented Oct 19, 2026

Uh oh!

sromoam left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 19, 2026

Security Scan Results

ASH Security Scan Report

Scan Metadata

Summary

Scanner Results

Top 2 Hotspots

Detailed Findings

Finding 1: SECRET-HEX-HIGH-ENTROPY-STRING

Finding 2: B108

Finding 3: B307

Finding 4: B307

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants