Skip to content

Blitzy: Add structural conversion API (tomlkit.convert) for bidirectional table/inline/dotted-key conversions - #1

Closed
blitzy[bot] wants to merge 15 commits into
instance_dd05eebc8ed9e30fc6c223088a5a450cb54c1cabfrom
blitzy-8820af43-47d3-4280-b032-f8e1f55bdcf9
Closed

blitzy[bot] wants to merge 15 commits into
instance_dd05eebc8ed9e30fc6c223088a5a450cb54c1cabfrom
blitzy-8820af43-47d3-4280-b032-f8e1f55bdcf9

Conversation

@blitzy

@blitzy blitzy Bot commented Jul 16, 2026

Copy link
Copy Markdown

Summary

Adds a structural conversion API to tomlkit as a new, purely additive tomlkit.convert module. Four functions convert in place between TOML's three nested-data forms — standard header tables ([header]), inline tables ({ ... }), and dotted-key assignments (a.b.c = value) — while preserving values, migrating comments, and maintaining round-trip integrity.

What's included

  • tomlkit/convert.py (new, 1,639 L): to_inline_table, to_standard_table, to_dotted_keys(…, max_depth=None), to_super_table, plus a shared key_path resolver and helpers factored to keep McCabe ≤ 10.
  • tomlkit/exceptions.py: new ConversionError(TOMLKitError) with a key_path attribute — distinct from the pre-existing ConvertError.
  • tomlkit/__init__.py: four re-exports + __all__ entries (callable as tomlkit.to_inline_table(...)).
  • tests/test_convert.py (new, 1,577 L): 99 tests at 97% coverage.
  • docs/api.rst + CHANGELOG.md: "TOML Conversions" automodule and an [unreleased] → Added entry.

Contracts upheld

  • In-place mutation returning the same document instance (func(...) is doc).
  • Byte-exact round-trip: dumps(parse(dumps(doc))) == dumps(doc) after every conversion.
  • Direction-dependent comment migration (3 directions).
  • ConversionError with populated key_path for nonexistent key, non-table intermediate, wrong-type target, AoT descendant, and unmatched prefix.
  • Zero new dependencies (stdlib-only imports).

Validation

  • 1,063 / 1,063 tests pass (99 feature + 284 regression + 680 TOML-compliance); zero regressions; passes under -W error.
  • ruff check / ruff format --check clean; McCabe ≤ 10; compileall exit 0.
  • Sphinx docs build with 0 warnings (Read the Docs toolchain).

Reviewer notes

  • Minor scope deviation: tomlkit/api.py received a docstring-only RST fix (Example: → Example::) to satisfy the docs-no-warnings gate — no functional/behavioral change.
  • Remaining (human path-to-production, ~12h): peer review, full CI interpreter matrix (py38–py312/pypy3; validated on 3.13 only), merge + [unreleased] reconciliation, release/publish, and optional non-gating mypy cleanup (16 type-narrowing findings, runtime-correct).

Status: 86.5% complete — functionally complete and validated; remaining work is human review, CI-matrix confirmation, and release mechanics.

blitzyai added 15 commits July 15, 2026 22:19
Introduce ConversionError, a TOMLKitError subclass that carries a
key_path attribute identifying the requested dotted key path. It is
raised when a structural conversion cannot be performed for a target
key path, and is kept distinct from the existing ConvertError used by
item().

Document the upcoming tomlkit.convert structural conversion API
(to_inline_table, to_standard_table, to_dotted_keys, to_super_table)
and the new ConversionError under the unreleased section of the
changelog.
Implement the tomlkit/convert.py leaf module providing bidirectional
conversion between TOML's three structural forms — standard header
tables, inline tables, and dotted-key assignments — while preserving
values and migrating comments:

- to_inline_table: standard Table -> InlineTable (recurses nested
  tables; no-op if already inline; ConversionError on non-Table or
  any AoT descendant).
- to_standard_table: InlineTable -> [header] Table (recurses nested
  inline tables; migrates the inline key's comment onto the header;
  no-op if already standard).
- to_dotted_keys: flatten a Table/InlineTable into dotted-key
  assignments in the parent, honoring max_depth (None = unlimited,
  1 = immediate children); emits the header comment as a standalone
  comment before the first dotted key.
- to_super_table: group dotted keys sharing a prefix into a new
  [prefix] table; promotes a preceding standalone comment to the
  header; ConversionError when no entries match.

All functions mutate doc in place, return the same instance, resolve
key paths via a shared _resolve helper (raising ConversionError with a
populated key_path for nonexistent keys or non-table intermediates),
and preserve parse(dumps(doc)) round-trip integrity by reusing the
existing Container trivia-preserving mutation primitives.
Resolves all 13 review findings (2 CRITICAL, 10 MAJOR, 1 MINOR) that made
the structural conversion API NOT APPROVED. Rewrites tomlkit/convert.py so
every conversion is transactional, value/comment/round-trip preserving, and
linear-time. No changes to CHANGELOG.md or exceptions.py were needed: the
CHANGELOG's preservation claims are now made TRUE by the implementation, and
ConversionError was already correct.

CRITICAL
- #1 Out-of-order / tuple-mapped targets: _resolve now recognizes fragmented
  (multi-index) logical tables and either handles the complete target or
  rejects atomically (doc left byte-identical), instead of operating on the
  first physical fragment and producing an unparsable document.
- #2 R3 AoT-under-dotted-key: to_dotted_keys pre-validates and rejects AoT
  descendants with ConversionError before any mutation (no partial edits).

MAJOR
- #3 R3 empty table/branch no longer lost: emits an empty inline `{}` so the
  key survives the flatten.
- #4 Builders now transform the full ordered body stream, preserving
  standalone comments, blank lines, and nested inline comments (R2/R3/R4);
  only inline per-item comment loss (documented, engine-level) remains.
- python-poetry#5 R3 keeps the resolved index (transaction-safe indexed replacement); the
  migrated header comment stays adjacent, immediately before the first
  dotted key, rather than being reordered after later siblings/tables.
- python-poetry#6 to_super_table inserts the new [prefix] table at the first match rather
  than appending at the end.
- python-poetry#7 (CWE-20) key-path segmentation rejects empty components (`a.`, `.a`,
  `a..b`) across all four functions.
- python-poetry#8 Comment migration copies text verbatim (multi-hash `### keep` preserved)
  instead of stripping leading hashes.
- python-poetry#9 Mutation uses only sanctioned Container primitives (_replace_at,
  _insert_at, _remove_at, remove, _raw_append); no direct body/_map/dict
  surgery and no swallowed KeyError.
- python-poetry#10 (CWE-400) super-table grouping is O(m) linear (was O(m^2)); super-Table
  children keep their dotted key so fragments merge without duplicate headers.
- python-poetry#11 Dotted-key separators preserved (`a.x = 1`, no `x = 1` leak) and EOF
  newline handling corrected for all builders.

MINOR
- python-poetry#12 Module and per-function docstrings corrected to describe actual
  behavior (no overstated guarantees).

Validation: ruff check + ruff-format (pinned 0.15.6) clean, McCabe C901 <=10,
py_compile OK; full existing suite 964 passed (zero regressions); dedicated
findings harness 46/46 plus per-requirement harnesses all green.
…, F-2)

Resolve two CRITICAL QA findings and one INFO finding in tomlkit.convert,
all rooted in to_dotted_keys (R3).

Root cause: to_dotted_keys inserted literal DottedKey body entries via
parent._insert_at, leaving the parent Container._map keyed by the full
dotted STRING (e.g. {'a.x': 0, 'a.y': 1}). Serialization was byte-correct
and round-tripped, but the flattened result was structurally broken.

F-1 (CRITICAL): the converted values were inaccessible and the map stale --
doc["a"], "a" in doc, doc.unwrap(), dict(doc), value updates, and deletes
all raised NonExistentKey.

F-2 (CRITICAL): the R3 result did not compose with to_super_table (R4)
in memory; to_dotted_keys("a", d) then to_super_table("a", d) raised
ConversionError.

Fix: build the same canonical representation the parser produces for dotted
keys (Container._handle_dotted_key) -- a single super-table keyed by the
leading segment (Table(Container(True), Trivia(), False,
is_super_table=True)) with _dotted child keys, children attached via
Table.raw_append so both the _map and dict view stay consistent. The parent
map is now keyed by the leading segment ({'a': 0}), so the flattened document
stays addressable, unwrap-able, editable, and composes with to_super_table.
Empty tables still collapse to an empty inline table (a = {}) with the header
comment preserved as a leading standalone line. Header and between-entry
comments are migrated, not dropped.

INFO: documented in the module docstring that a dotted-STRING path splits on
".", so segments containing a dot/space/quote must be passed via the sequence
form (e.g. ["a b"]).

Removed now-dead helpers (_dotted_key, _leaf_assignment_key, _apply_pending,
_flatten) and the unused DottedKey import; added small single-purpose helpers
(each within McCabe complexity 10) with full docstrings.

Verification: full existing suite (964 passed) unchanged; ruff check, ruff
format, and C901 (McCabe<=10) all clean; F-1/F-2/INFO re-verified at runtime
(exact QA repro + inline/nested variants + full standard->inline->standard->
dotted->super chain); 36-case golden battery + fuzz differential confirm
byte-exact serialization and identical error behavior vs the prior code; R1/
R2/R4/R5/resolver and ConvertError backward-compat regression-checked (67
checks). Only tomlkit/convert.py changed.
Add four single-line imports from tomlkit.convert (to_dotted_keys,
to_inline_table, to_standard_table, to_super_table) and the matching
four __all__ entries so the structural conversion functions are
callable as tomlkit.to_inline_table(...) etc.

Purely additive: __version__ stays 0.14.0 and all existing
tomlkit.api imports and __all__ entries are unchanged.
Document the new tomlkit.convert module in the API reference by adding a
"TOML Conversions" section with an automodule directive between the
existing "TOML Items" and "TOML Exceptions" sections. This renders the
four public conversion functions (to_inline_table, to_standard_table,
to_dotted_keys, to_super_table).

ConversionError is intentionally not added here; it is auto-documented by
the existing tomlkit.exceptions automodule block. Purely additive change
matching the repository's section conventions (16-dash underline,
3-space-indented options, single-blank-line spacing, LF endings).
F1 (R3): place the flattened table's header comment as a standalone comment
in the PARENT container immediately before the dotted replacement (via new
_insert_standalone_comment helper) instead of burying it inside the dotted
super-table, so to_super_table (the in-memory inverse) restores the header.

F2 (R3): preserve nested table header comments during flattening -
_emit_dotted_entry now carries value.trivia.comment as the recursion's leading
comment, and emits it as a standalone comment before the collapsed inline
assignment at a max_depth boundary.

F3 (docs): narrow the module-docstring note to literal-dot ambiguity only;
key segments containing spaces or quote characters resolve correctly as
dotted-string paths.

F4 (R4): require genuine dotted-key provenance in _find_super_matches and
_descend_super (Key.is_dotted() at the lead fragment and every traversed
prefix segment) so ordinary [a]/[a.b] header tables are no longer matched and
to_super_table raises ConversionError when no dotted assignments share the
prefix.

All conversions retain in-place mutation, identity return, and
parse(dumps(doc)) round-trip integrity. ruff/ruff-format/McCabe<=10 clean;
full suite 964 passed.
…A Issue #1) and document ConversionError.key_path (QA Issue #2)

Issue #1 (MAJOR): The four tomlkit.convert functions annotate params/returns
with the bare name TOMLDocument, which was registered in two py-domain
locations (the 'Public API' automodule:: tomlkit and the dedicated 'TOML
Document' autoclass), causing 8 'more than one target found for
cross-reference TOMLDocument' warnings that fail the fail_on_warning docs
build (AAP §0.6.3). Exclude TOMLDocument from the automodule:: tomlkit
directive so it is documented once, canonically, in its dedicated section;
the bare annotation now resolves unambiguously. Convert warnings 8 -> 0.

Issue #2 (INFO): Add an :ivar key_path: field to the ConversionError
docstring so the key_path attribute contract (AAP R5) is documented in the
class prose, not only via the constructor signature.

Out-of-scope (AAP §0.6.2, documented not fixed): api.py register_encoder RST
warnings; CHANGELOG python-poetry#456 parser bullet paren.

Verified: RTD Sphinx 7.4.7 build -> 0 tomlkit.convert warnings; ConversionError
renders key_path; full test suite 964 passed (no regressions).
…sion API

Add pytest coverage for the new tomlkit.convert module (to_inline_table,
to_standard_table, to_dotted_keys, to_super_table) and the ConversionError
exception. 23 flat test functions cover:

- R1 to_inline_table: standard->inline, no-op, non-Table error, AoT-descendant
  error, nested recursion
- R2 to_standard_table: inline->standard, no-op, non-InlineTable error, inline
  key comment -> header comment migration, nested recursion
- R3 to_dotted_keys: flatten Table/InlineTable into parent dotted keys, wrong
  type error, max_depth=None vs max_depth=1, header comment -> standalone comment
- R4 to_super_table: group dotted keys by prefix, unmatched-prefix error,
  preceding standalone comment -> header comment promotion
- Cross-cutting: resolver errors (nonexistent key, non-table intermediate),
  key-sequence path support, and the return-identity contract for all four

Every error branch asserts ConversionError.key_path equals the requested dotted
path; every mutating happy path asserts return identity (result is doc) and
byte-exact round-trip integrity (dumps(parse(dumps(doc))) == dumps(doc)).
Fix all eight code-review findings in the structural conversion API and
complete its acceptance/regression test matrix.

tomlkit/convert.py (F1-F7):
- F1 (CRITICAL, R2 to_standard_table): promote inline-table ancestors to
  standard tables before converting, so a [header] is never emitted inside
  an inline parent (previously produced unparsable TOML / InvalidNumberError).
- F2 (CRITICAL, R3 to_dotted_keys): same inline-ancestor promotion for the
  dotted-key path (previously produced EmptyKeyError inside an inline parent).
- F3 (MAJOR, R4 to_super_table): descend through existing standard header
  tables to resolve a nested prefix, restoring bidirectionality with R3.
- F4 (MAJOR, R4): detect and group literal (is_multi) DottedKey body entries,
  not only canonical super-table fragments.
- F5 (MAJOR, R4): promote only the standalone comment immediately preceding
  the first match; keep genuinely-inner fragment comments and leave unrelated
  parent-level comments with their siblings.
- F6 (MAJOR, R3): stop rejecting AoT descendants (AAP reserves AoT rejection
  for R1); preserve an AoT as a prefixed header while flattening compatible
  children, emitting dotted keys before any header child.
- F7 (MAJOR, R3): preserve a comment-only table's header and inner standalone
  comments (and blank lines) instead of dropping them as "empty".

tests/test_convert.py (F8):
- Add 20 deterministic tests (47 -> 67) covering every checkpoint obligation
  and a regression test per F1-F7 reproducer: facade re-export + unique
  __all__, ConversionError MRO/default/custom message/key_path, empty and
  literal-empty prefixes, failure atomicity (R1/R3/R4), max_depth=2 and
  multi-level recursion, deep nested AoT, comment non-duplication, and
  sibling preservation.

All conversions mutate in place, return the same doc, migrate comments per
direction, and satisfy dumps(parse(dumps(doc))) == dumps(doc) round-trip
integrity. Full suite: 1031 passed, zero regressions. ruff check, ruff-format,
and McCabe (<=10) all clean.
Extend tests/test_convert.py with two structural-conversion cases that
were previously untested: to_inline_table preserving a standalone comment
by rendering a multi-line inline table, and to_super_table grouping a
multi-segment dotted prefix that has no header-table ancestors. Also
assert the ConversionError.key_path value on the wrong-type and no-match
atomicity tests.

Format the register_encoder "Example" block in tomlkit/api.py as a
reStructuredText literal block ("Example::" plus a blank line) so the
documentation builds cleanly under fail_on_warning.
Wrong-typed key_path/dotted_prefix (None/int/float/bool/bytes) and a non-int
max_depth previously raised a bare TypeError that lacked .key_path and was not
a TOMLKitError, so an 'except ConversionError' guard could not catch it and an
uncaught traceback leaked internal paths/helper names.

- _segments: type-guard key_path; reject non-(str|Sequence) and bytes-like
  inputs with ConversionError(str(key_path)).
- to_dotted_keys: validate max_depth is None or int before any mutation
  (atomic), raising ConversionError with the resolved key_path (bool still
  accepted as an int subclass).
- tests/test_convert.py: add regression coverage for wrong-type key_path across
  all four functions, TOMLKitError catchability, bytes rejection, non-int
  max_depth, and bool max_depth support.

Satisfies the AAP 0.7 error contract (all failure modes raise ConversionError
with a populated key_path). No behavior change for valid inputs; 1038 tests
pass, ruff/McCabe/format clean, docs -W build warning-free.
Resolve five QA-reported defects in the structural conversion API and add
regression coverage, all confined to the in-scope files tomlkit/convert.py
and tests/test_convert.py.

B-1 (to_inline_table comment loss): _replace_at does not copy trivia on a
Table->InlineTable classification change, so header comments were dropped.
Migrate the target table's header comment onto the inline assignment via
_copy_comment, and re-emit nested standard-table header comments as standalone
lines inside a multi-line inline table (_inline_needs_multiline now forces
multi-line when a nested table carries a header comment; _build_multiline_inline
captures and emits it through the new _inline_comment_entry helper).

B-2 (live-model ordering): Container is a dict subclass whose live iteration
(list(doc)/keys()/items()/values()/unwrap()) follows dict insertion order,
while value/serialization/re-parse follow body order. The mutation primitives
re-add converted keys at the end of the dict, diverging the two orders. Add
_body_key_order/_resync_dict_order/_resync_tree and call _resync_tree(doc)
before returning from all four functions, realigning dict order to body
first-occurrence order without touching stored values, _map, or _body.

B-3 (unsafe key_path handling): the resolver ran arbitrary __str__/__repr__ on
caller-supplied path objects and silently coerced non-string sequence elements
(e.g. int 1 -> "1"). Add _SAFE_STR_TYPES exact-type gate with _safe_path_repr
and _safe_seq_repr, and validate each sequence element via _normalize_sequence,
rejecting non-str/Key elements with a ConversionError carrying a safe, bounded
key_path. F-1 wrong-type-path behavior preserved.

B-4 (CRLF -> invalid inline TOML): InlineTable.as_string strips "\n" but not
"\r", leaving a bare CR inside a compact inline table (standards-invalid).
Strip CR/LF from a scalar value's line-ending trivia in the compact builder
via _strip_inline_linebreaks so the rendered inline table is valid.

D-1 (inaccurate docstrings): to_super_table/to_dotted_keys/_descend_header_ancestors
claimed an "exact/true inverse at any nesting depth". Reword to describe
to_super_table as the semantic grouping counterpart of to_dotted_keys, with a
caveat that values are preserved but the concrete pre-flattening nesting shape
is not necessarily restored. Docs still build warning-free.

T-1 (regression coverage): add 17 tests (input validation, header-comment and
CRLF fidelity, live-order consistency across R1-R4, grouping-counterpart
semantics), rename one existing super-table test to reflect true semantics, and
import re for bare-CR assertions.

All conversions continue to mutate doc in place and return the same instance,
preserve parse(dumps(doc)) round-trip integrity, keep every function within
McCabe complexity 10, and pass ruff/ruff-format. Full suite: 1055 passed,
zero warnings under -W error.
Resolve CRITICAL QA finding S1: after to_super_table() grouped dotted keys
into a [prefix] header table, its grouped children could not be deleted --
del/pop raised KeyError while the entry had ALREADY been removed from the
container body, a non-atomic failure that silently dropped the entry from the
serialized output.

Root cause: Table/InlineTable subclass dict and keep their own key->item
storage alongside the backing Container. _build_super_leaf populated the leaf
via low-level Container appends (to preserve exact trivia) which updated the
container but left the leaf table's own dict empty; AbstractTable.remove
deletes from the container first, then dict.__delitem__ on the empty own dict
raised KeyError.

Fix: add _sync_table_dict() to seed a freshly built table's own dict from its
body (exactly as AbstractTable.__init__ does), invoked at the end of
_build_super_leaf. Only missing keys are added and the walk recurses into
nested tables, so already-consistent tables are untouched and the serialized
bytes are unchanged. Grouped children are now deletable and atomically so.

Tests: add 8 regression tests covering del/pop of grouped children at every
nesting level (leaf single-seg, grandchild, nested super, literal multiseg),
pop(existing)->value, pop(missing,default)->default, pop(missing) atomicity,
update/extend, and byte-exact round-trip after deletion. Verified these fail
against the pre-fix code with the exact S1 KeyError signature.

Files: tomlkit/convert.py, tests/test_convert.py
@blitzy blitzy Bot closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant