Repository navigation
Blitzy: Add structural conversion API (tomlkit.convert) for bidirectional table/inline/dotted-key conversions - #1
Closed
blitzy[bot] wants to merge 15 commits into
Conversation
Introduce ConversionError, a TOMLKitError subclass that carries a key_path attribute identifying the requested dotted key path. It is raised when a structural conversion cannot be performed for a target key path, and is kept distinct from the existing ConvertError used by item(). Document the upcoming tomlkit.convert structural conversion API (to_inline_table, to_standard_table, to_dotted_keys, to_super_table) and the new ConversionError under the unreleased section of the changelog.
Implement the tomlkit/convert.py leaf module providing bidirectional conversion between TOML's three structural forms — standard header tables, inline tables, and dotted-key assignments — while preserving values and migrating comments: - to_inline_table: standard Table -> InlineTable (recurses nested tables; no-op if already inline; ConversionError on non-Table or any AoT descendant). - to_standard_table: InlineTable -> [header] Table (recurses nested inline tables; migrates the inline key's comment onto the header; no-op if already standard). - to_dotted_keys: flatten a Table/InlineTable into dotted-key assignments in the parent, honoring max_depth (None = unlimited, 1 = immediate children); emits the header comment as a standalone comment before the first dotted key. - to_super_table: group dotted keys sharing a prefix into a new [prefix] table; promotes a preceding standalone comment to the header; ConversionError when no entries match. All functions mutate doc in place, return the same instance, resolve key paths via a shared _resolve helper (raising ConversionError with a populated key_path for nonexistent keys or non-table intermediates), and preserve parse(dumps(doc)) round-trip integrity by reusing the existing Container trivia-preserving mutation primitives.
Resolves all 13 review findings (2 CRITICAL, 10 MAJOR, 1 MINOR) that made the structural conversion API NOT APPROVED. Rewrites tomlkit/convert.py so every conversion is transactional, value/comment/round-trip preserving, and linear-time. No changes to CHANGELOG.md or exceptions.py were needed: the CHANGELOG's preservation claims are now made TRUE by the implementation, and ConversionError was already correct. CRITICAL - #1 Out-of-order / tuple-mapped targets: _resolve now recognizes fragmented (multi-index) logical tables and either handles the complete target or rejects atomically (doc left byte-identical), instead of operating on the first physical fragment and producing an unparsable document. - #2 R3 AoT-under-dotted-key: to_dotted_keys pre-validates and rejects AoT descendants with ConversionError before any mutation (no partial edits). MAJOR - #3 R3 empty table/branch no longer lost: emits an empty inline `{}` so the key survives the flatten. - #4 Builders now transform the full ordered body stream, preserving standalone comments, blank lines, and nested inline comments (R2/R3/R4); only inline per-item comment loss (documented, engine-level) remains. - python-poetry#5 R3 keeps the resolved index (transaction-safe indexed replacement); the migrated header comment stays adjacent, immediately before the first dotted key, rather than being reordered after later siblings/tables. - python-poetry#6 to_super_table inserts the new [prefix] table at the first match rather than appending at the end. - python-poetry#7 (CWE-20) key-path segmentation rejects empty components (`a.`, `.a`, `a..b`) across all four functions. - python-poetry#8 Comment migration copies text verbatim (multi-hash `### keep` preserved) instead of stripping leading hashes. - python-poetry#9 Mutation uses only sanctioned Container primitives (_replace_at, _insert_at, _remove_at, remove, _raw_append); no direct body/_map/dict surgery and no swallowed KeyError. - python-poetry#10 (CWE-400) super-table grouping is O(m) linear (was O(m^2)); super-Table children keep their dotted key so fragments merge without duplicate headers. - python-poetry#11 Dotted-key separators preserved (`a.x = 1`, no `x = 1` leak) and EOF newline handling corrected for all builders. MINOR - python-poetry#12 Module and per-function docstrings corrected to describe actual behavior (no overstated guarantees). Validation: ruff check + ruff-format (pinned 0.15.6) clean, McCabe C901 <=10, py_compile OK; full existing suite 964 passed (zero regressions); dedicated findings harness 46/46 plus per-requirement harnesses all green.
…, F-2)
Resolve two CRITICAL QA findings and one INFO finding in tomlkit.convert,
all rooted in to_dotted_keys (R3).
Root cause: to_dotted_keys inserted literal DottedKey body entries via
parent._insert_at, leaving the parent Container._map keyed by the full
dotted STRING (e.g. {'a.x': 0, 'a.y': 1}). Serialization was byte-correct
and round-tripped, but the flattened result was structurally broken.
F-1 (CRITICAL): the converted values were inaccessible and the map stale --
doc["a"], "a" in doc, doc.unwrap(), dict(doc), value updates, and deletes
all raised NonExistentKey.
F-2 (CRITICAL): the R3 result did not compose with to_super_table (R4)
in memory; to_dotted_keys("a", d) then to_super_table("a", d) raised
ConversionError.
Fix: build the same canonical representation the parser produces for dotted
keys (Container._handle_dotted_key) -- a single super-table keyed by the
leading segment (Table(Container(True), Trivia(), False,
is_super_table=True)) with _dotted child keys, children attached via
Table.raw_append so both the _map and dict view stay consistent. The parent
map is now keyed by the leading segment ({'a': 0}), so the flattened document
stays addressable, unwrap-able, editable, and composes with to_super_table.
Empty tables still collapse to an empty inline table (a = {}) with the header
comment preserved as a leading standalone line. Header and between-entry
comments are migrated, not dropped.
INFO: documented in the module docstring that a dotted-STRING path splits on
".", so segments containing a dot/space/quote must be passed via the sequence
form (e.g. ["a b"]).
Removed now-dead helpers (_dotted_key, _leaf_assignment_key, _apply_pending,
_flatten) and the unused DottedKey import; added small single-purpose helpers
(each within McCabe complexity 10) with full docstrings.
Verification: full existing suite (964 passed) unchanged; ruff check, ruff
format, and C901 (McCabe<=10) all clean; F-1/F-2/INFO re-verified at runtime
(exact QA repro + inline/nested variants + full standard->inline->standard->
dotted->super chain); 36-case golden battery + fuzz differential confirm
byte-exact serialization and identical error behavior vs the prior code; R1/
R2/R4/R5/resolver and ConvertError backward-compat regression-checked (67
checks). Only tomlkit/convert.py changed.
Add four single-line imports from tomlkit.convert (to_dotted_keys, to_inline_table, to_standard_table, to_super_table) and the matching four __all__ entries so the structural conversion functions are callable as tomlkit.to_inline_table(...) etc. Purely additive: __version__ stays 0.14.0 and all existing tomlkit.api imports and __all__ entries are unchanged.
Document the new tomlkit.convert module in the API reference by adding a "TOML Conversions" section with an automodule directive between the existing "TOML Items" and "TOML Exceptions" sections. This renders the four public conversion functions (to_inline_table, to_standard_table, to_dotted_keys, to_super_table). ConversionError is intentionally not added here; it is auto-documented by the existing tomlkit.exceptions automodule block. Purely additive change matching the repository's section conventions (16-dash underline, 3-space-indented options, single-blank-line spacing, LF endings).
F1 (R3): place the flattened table's header comment as a standalone comment in the PARENT container immediately before the dotted replacement (via new _insert_standalone_comment helper) instead of burying it inside the dotted super-table, so to_super_table (the in-memory inverse) restores the header. F2 (R3): preserve nested table header comments during flattening - _emit_dotted_entry now carries value.trivia.comment as the recursion's leading comment, and emits it as a standalone comment before the collapsed inline assignment at a max_depth boundary. F3 (docs): narrow the module-docstring note to literal-dot ambiguity only; key segments containing spaces or quote characters resolve correctly as dotted-string paths. F4 (R4): require genuine dotted-key provenance in _find_super_matches and _descend_super (Key.is_dotted() at the lead fragment and every traversed prefix segment) so ordinary [a]/[a.b] header tables are no longer matched and to_super_table raises ConversionError when no dotted assignments share the prefix. All conversions retain in-place mutation, identity return, and parse(dumps(doc)) round-trip integrity. ruff/ruff-format/McCabe<=10 clean; full suite 964 passed.
…A Issue #1) and document ConversionError.key_path (QA Issue #2) Issue #1 (MAJOR): The four tomlkit.convert functions annotate params/returns with the bare name TOMLDocument, which was registered in two py-domain locations (the 'Public API' automodule:: tomlkit and the dedicated 'TOML Document' autoclass), causing 8 'more than one target found for cross-reference TOMLDocument' warnings that fail the fail_on_warning docs build (AAP §0.6.3). Exclude TOMLDocument from the automodule:: tomlkit directive so it is documented once, canonically, in its dedicated section; the bare annotation now resolves unambiguously. Convert warnings 8 -> 0. Issue #2 (INFO): Add an :ivar key_path: field to the ConversionError docstring so the key_path attribute contract (AAP R5) is documented in the class prose, not only via the constructor signature. Out-of-scope (AAP §0.6.2, documented not fixed): api.py register_encoder RST warnings; CHANGELOG python-poetry#456 parser bullet paren. Verified: RTD Sphinx 7.4.7 build -> 0 tomlkit.convert warnings; ConversionError renders key_path; full test suite 964 passed (no regressions).
…sion API Add pytest coverage for the new tomlkit.convert module (to_inline_table, to_standard_table, to_dotted_keys, to_super_table) and the ConversionError exception. 23 flat test functions cover: - R1 to_inline_table: standard->inline, no-op, non-Table error, AoT-descendant error, nested recursion - R2 to_standard_table: inline->standard, no-op, non-InlineTable error, inline key comment -> header comment migration, nested recursion - R3 to_dotted_keys: flatten Table/InlineTable into parent dotted keys, wrong type error, max_depth=None vs max_depth=1, header comment -> standalone comment - R4 to_super_table: group dotted keys by prefix, unmatched-prefix error, preceding standalone comment -> header comment promotion - Cross-cutting: resolver errors (nonexistent key, non-table intermediate), key-sequence path support, and the return-identity contract for all four Every error branch asserts ConversionError.key_path equals the requested dotted path; every mutating happy path asserts return identity (result is doc) and byte-exact round-trip integrity (dumps(parse(dumps(doc))) == dumps(doc)).
Fix all eight code-review findings in the structural conversion API and complete its acceptance/regression test matrix. tomlkit/convert.py (F1-F7): - F1 (CRITICAL, R2 to_standard_table): promote inline-table ancestors to standard tables before converting, so a [header] is never emitted inside an inline parent (previously produced unparsable TOML / InvalidNumberError). - F2 (CRITICAL, R3 to_dotted_keys): same inline-ancestor promotion for the dotted-key path (previously produced EmptyKeyError inside an inline parent). - F3 (MAJOR, R4 to_super_table): descend through existing standard header tables to resolve a nested prefix, restoring bidirectionality with R3. - F4 (MAJOR, R4): detect and group literal (is_multi) DottedKey body entries, not only canonical super-table fragments. - F5 (MAJOR, R4): promote only the standalone comment immediately preceding the first match; keep genuinely-inner fragment comments and leave unrelated parent-level comments with their siblings. - F6 (MAJOR, R3): stop rejecting AoT descendants (AAP reserves AoT rejection for R1); preserve an AoT as a prefixed header while flattening compatible children, emitting dotted keys before any header child. - F7 (MAJOR, R3): preserve a comment-only table's header and inner standalone comments (and blank lines) instead of dropping them as "empty". tests/test_convert.py (F8): - Add 20 deterministic tests (47 -> 67) covering every checkpoint obligation and a regression test per F1-F7 reproducer: facade re-export + unique __all__, ConversionError MRO/default/custom message/key_path, empty and literal-empty prefixes, failure atomicity (R1/R3/R4), max_depth=2 and multi-level recursion, deep nested AoT, comment non-duplication, and sibling preservation. All conversions mutate in place, return the same doc, migrate comments per direction, and satisfy dumps(parse(dumps(doc))) == dumps(doc) round-trip integrity. Full suite: 1031 passed, zero regressions. ruff check, ruff-format, and McCabe (<=10) all clean.
Extend tests/test_convert.py with two structural-conversion cases that
were previously untested: to_inline_table preserving a standalone comment
by rendering a multi-line inline table, and to_super_table grouping a
multi-segment dotted prefix that has no header-table ancestors. Also
assert the ConversionError.key_path value on the wrong-type and no-match
atomicity tests.
Format the register_encoder "Example" block in tomlkit/api.py as a
reStructuredText literal block ("Example::" plus a blank line) so the
documentation builds cleanly under fail_on_warning.
Wrong-typed key_path/dotted_prefix (None/int/float/bool/bytes) and a non-int max_depth previously raised a bare TypeError that lacked .key_path and was not a TOMLKitError, so an 'except ConversionError' guard could not catch it and an uncaught traceback leaked internal paths/helper names. - _segments: type-guard key_path; reject non-(str|Sequence) and bytes-like inputs with ConversionError(str(key_path)). - to_dotted_keys: validate max_depth is None or int before any mutation (atomic), raising ConversionError with the resolved key_path (bool still accepted as an int subclass). - tests/test_convert.py: add regression coverage for wrong-type key_path across all four functions, TOMLKitError catchability, bytes rejection, non-int max_depth, and bool max_depth support. Satisfies the AAP 0.7 error contract (all failure modes raise ConversionError with a populated key_path). No behavior change for valid inputs; 1038 tests pass, ruff/McCabe/format clean, docs -W build warning-free.
Resolve five QA-reported defects in the structural conversion API and add regression coverage, all confined to the in-scope files tomlkit/convert.py and tests/test_convert.py. B-1 (to_inline_table comment loss): _replace_at does not copy trivia on a Table->InlineTable classification change, so header comments were dropped. Migrate the target table's header comment onto the inline assignment via _copy_comment, and re-emit nested standard-table header comments as standalone lines inside a multi-line inline table (_inline_needs_multiline now forces multi-line when a nested table carries a header comment; _build_multiline_inline captures and emits it through the new _inline_comment_entry helper). B-2 (live-model ordering): Container is a dict subclass whose live iteration (list(doc)/keys()/items()/values()/unwrap()) follows dict insertion order, while value/serialization/re-parse follow body order. The mutation primitives re-add converted keys at the end of the dict, diverging the two orders. Add _body_key_order/_resync_dict_order/_resync_tree and call _resync_tree(doc) before returning from all four functions, realigning dict order to body first-occurrence order without touching stored values, _map, or _body. B-3 (unsafe key_path handling): the resolver ran arbitrary __str__/__repr__ on caller-supplied path objects and silently coerced non-string sequence elements (e.g. int 1 -> "1"). Add _SAFE_STR_TYPES exact-type gate with _safe_path_repr and _safe_seq_repr, and validate each sequence element via _normalize_sequence, rejecting non-str/Key elements with a ConversionError carrying a safe, bounded key_path. F-1 wrong-type-path behavior preserved. B-4 (CRLF -> invalid inline TOML): InlineTable.as_string strips "\n" but not "\r", leaving a bare CR inside a compact inline table (standards-invalid). Strip CR/LF from a scalar value's line-ending trivia in the compact builder via _strip_inline_linebreaks so the rendered inline table is valid. D-1 (inaccurate docstrings): to_super_table/to_dotted_keys/_descend_header_ancestors claimed an "exact/true inverse at any nesting depth". Reword to describe to_super_table as the semantic grouping counterpart of to_dotted_keys, with a caveat that values are preserved but the concrete pre-flattening nesting shape is not necessarily restored. Docs still build warning-free. T-1 (regression coverage): add 17 tests (input validation, header-comment and CRLF fidelity, live-order consistency across R1-R4, grouping-counterpart semantics), rename one existing super-table test to reflect true semantics, and import re for bare-CR assertions. All conversions continue to mutate doc in place and return the same instance, preserve parse(dumps(doc)) round-trip integrity, keep every function within McCabe complexity 10, and pass ruff/ruff-format. Full suite: 1055 passed, zero warnings under -W error.
Resolve CRITICAL QA finding S1: after to_super_table() grouped dotted keys into a [prefix] header table, its grouped children could not be deleted -- del/pop raised KeyError while the entry had ALREADY been removed from the container body, a non-atomic failure that silently dropped the entry from the serialized output. Root cause: Table/InlineTable subclass dict and keep their own key->item storage alongside the backing Container. _build_super_leaf populated the leaf via low-level Container appends (to preserve exact trivia) which updated the container but left the leaf table's own dict empty; AbstractTable.remove deletes from the container first, then dict.__delitem__ on the empty own dict raised KeyError. Fix: add _sync_table_dict() to seed a freshly built table's own dict from its body (exactly as AbstractTable.__init__ does), invoked at the end of _build_super_leaf. Only missing keys are added and the walk recurses into nested tables, so already-consistent tables are untouched and the serialized bytes are unchanged. Grouped children are now deletable and atomically so. Tests: add 8 regression tests covering del/pop of grouped children at every nesting level (leaf single-seg, grandchild, nested super, literal multiseg), pop(existing)->value, pop(missing,default)->default, pop(missing) atomicity, update/extend, and byte-exact round-trip after deletion. Verified these fail against the pre-fix code with the exact S1 KeyError signature. Files: tomlkit/convert.py, tests/test_convert.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a structural conversion API to
tomlkitas a new, purely additivetomlkit.convertmodule. Four functions convert in place between TOML's three nested-data forms — standard header tables ([header]), inline tables ({ ... }), and dotted-key assignments (a.b.c = value) — while preserving values, migrating comments, and maintaining round-trip integrity.What's included
tomlkit/convert.py(new, 1,639 L):to_inline_table,to_standard_table,to_dotted_keys(…, max_depth=None),to_super_table, plus a sharedkey_pathresolver and helpers factored to keep McCabe ≤ 10.tomlkit/exceptions.py: newConversionError(TOMLKitError)with akey_pathattribute — distinct from the pre-existingConvertError.tomlkit/__init__.py: four re-exports +__all__entries (callable astomlkit.to_inline_table(...)).tests/test_convert.py(new, 1,577 L): 99 tests at 97% coverage.docs/api.rst+CHANGELOG.md: "TOML Conversions" automodule and an[unreleased] → Addedentry.Contracts upheld
func(...) is doc).dumps(parse(dumps(doc))) == dumps(doc)after every conversion.ConversionErrorwith populatedkey_pathfor nonexistent key, non-table intermediate, wrong-type target, AoT descendant, and unmatched prefix.Validation
-W error.ruff check/ruff format --checkclean; McCabe ≤ 10;compileallexit 0.Reviewer notes
tomlkit/api.pyreceived a docstring-only RST fix (Example:→Example::) to satisfy the docs-no-warnings gate — no functional/behavioral change.[unreleased]reconciliation, release/publish, and optional non-gating mypy cleanup (16 type-narrowing findings, runtime-correct).Status: 86.5% complete — functionally complete and validated; remaining work is human review, CI-matrix confirmation, and release mechanics.