Skip to content

Commit eefb024

Browse files
johannschopplichjohannschopplich
committed
feat: opt-in key folding and path expansion (closes #86)
1 parent e1f5d13 commit eefb024

File tree

14 files changed

+647
-12
lines changed

14 files changed

+647
-12
lines changed

‎README.md‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
[![CI](https://github.com/toon-format/toon/actions/workflows/ci.yml/badge.svg)](https://github.com/toon-format/toon/actions)
66
[![npm version](https://img.shields.io/npm/v/@toon-format/toon.svg)](https://www.npmjs.com/package/@toon-format/toon)
7-
[![SPEC v1.4](https://img.shields.io/badge/spec-v1.4-lightgray)](https://github.com/toon-format/spec)
7+
[![SPEC v1.5](https://img.shields.io/badge/spec-v1.5-lightgray)](https://github.com/toon-format/spec)
88
[![npm downloads (total)](https://img.shields.io/npm/dt/@toon-format/toon.svg)](https://www.npmjs.com/package/@toon-format/toon)
99
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
1010

@@ -80,6 +80,7 @@ See [benchmarks](#benchmarks) for concrete comparisons across different data str
8080
- 🍱 **Minimal syntax:** removes redundant punctuation (braces, brackets, most quotes)
8181
- 📐 **Indentation-based structure:** like YAML, uses whitespace instead of braces
8282
- 🧺 **Tabular arrays:** declare keys once, stream data as rows
83+
- 🔗 **Optional key folding (v1.5):** collapses single-key wrapper chains into dotted paths (e.g., `data.metadata.items`) to reduce indentation and tokens
8384

8485
[^1]: For flat tabular data, CSV is more compact. TOON adds minimal overhead to provide explicit structure and validation that improves LLM reliability.
8586

@@ -736,6 +737,9 @@ cat data.toon | npx @toon-format/cli --decode
736737
| `--length-marker` | Add `#` prefix to array lengths (e.g., `items[#3]`) |
737738
| `--stats` | Show token count estimates and savings (encode only) |
738739
| `--no-strict` | Disable strict validation when decoding |
740+
| `--key-folding <mode>` | Key folding mode: `off`, `safe` (default: `off`) - collapses nested chains (v1.5) |
741+
| `--flatten-depth <number>` | Maximum segments to fold (default: `Infinity`) - requires `--key-folding safe` (v1.5) |
742+
| `--expand-paths <mode>` | Path expansion mode: `off`, `safe` (default: `off`) - reconstructs dotted keys (v1.5) |
739743

740744
### Examples
741745

@@ -752,6 +756,9 @@ npx @toon-format/cli data.json --delimiter "|" --length-marker -o output.toon
752756
# Lenient decoding (skip validation)
753757
npx @toon-format/cli data.toon --no-strict -o output.json
754758

759+
# Key folding for nested data (v1.5)
760+
npx @toon-format/cli data.json --key-folding safe -o output.toon
761+
755762
# Stdin workflows
756763
echo '{"name": "Ada", "age": 30}' | npx @toon-format/cli --stats
757764
cat large-dataset.json | npx @toon-format/cli --delimiter "\t" > output.toon
@@ -797,6 +804,40 @@ user:
797804
name: Ada
798805
```
799806

807+
### Key Folding (Optional)
808+
809+
New in v1.5: Optionally collapse single-key wrapper chains into dotted paths to reduce tokens. Enable with `keyFolding: 'safe'`.
810+
811+
Standard nesting:
812+
813+
```
814+
data:
815+
metadata:
816+
items[2]: a,b
817+
```
818+
819+
With key folding:
820+
821+
```
822+
data.metadata.items[2]: a,b
823+
```
824+
825+
Round-trip with path expansion:
826+
827+
```ts
828+
import { decode, encode } from '@toon-format/toon'
829+
830+
const original = { data: { metadata: { items: ['a', 'b'] } } }
831+
832+
const toon = encode(original, { keyFolding: 'safe' })
833+
// → "data.metadata.items[2]: a,b"
834+
835+
const restored = decode(toon, { expandPaths: 'safe' })
836+
// → Matches original structure
837+
```
838+
839+
See §13.4 in the [specification](https://github.com/toon-format/spec/blob/main/SPEC.md#134-key-folding-and-path-expansion) for folding rules and safety guarantees.
840+
800841
### Arrays
801842

802843
> [!TIP]
@@ -975,6 +1016,8 @@ Converts any JSON-serializable value to TOON format.
9751016
- `indent?: number` – Number of spaces per indentation level (default: `2`)
9761017
- `delimiter?: ',' | '\t' | '|'` – Delimiter for array values and tabular rows (default: `','`)
9771018
- `lengthMarker?: '#' | false` – Optional marker to prefix array lengths (default: `false`)
1019+
- `keyFolding?: 'off' | 'safe'` – Enable key folding to collapse single-key wrapper chains into dotted paths (default: `'off'`). When `'safe'`, only valid identifier segments are folded (v1.5)
1020+
- `flattenDepth?: number` – Maximum number of segments to fold when `keyFolding` is enabled (default: `Infinity`). Values 0-1 have no practical effect (v1.5)
9781021

9791022
**Returns:**
9801023

@@ -1096,6 +1139,7 @@ Converts a TOON-formatted string back to JavaScript values.
10961139
- `options` – Optional decoding options:
10971140
- `indent?: number` – Expected number of spaces per indentation level (default: `2`)
10981141
- `strict?: boolean` – Enable strict validation (default: `true`)
1142+
- `expandPaths?: 'off' | 'safe'` – Enable path expansion to reconstruct dotted keys into nested objects (default: `'off'`). Pairs with `keyFolding: 'safe'` for lossless round-trips (v1.5)
10991143

11001144
**Returns:**
11011145

@@ -1223,7 +1267,7 @@ Task: Return only users with role "user" as TOON. Use the same header. Set [N] t
12231267
## Other Implementations
12241268

12251269
> [!NOTE]
1226-
> When implementing TOON in other languages, please follow the [specification](https://github.com/toon-format/spec/blob/main/SPEC.md) (currently v1.4) to ensure compatibility across implementations. The [conformance tests](https://github.com/toon-format/spec/tree/main/tests) provide language-agnostic test fixtures that validate implementations across any language.
1270+
> When implementing TOON in other languages, please follow the [specification](https://github.com/toon-format/spec/blob/main/SPEC.md) (currently v1.5) to ensure compatibility across implementations. The [conformance tests](https://github.com/toon-format/spec/tree/main/tests) provide language-agnostic test fixtures that validate your implementations.
12271271
12281272
### Official Implementations
12291273

‎packages/cli/README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,9 @@ cat data.toon | toon --decode
6565
| `--length-marker` | Add `#` prefix to array lengths (e.g., `items[#3]`) |
6666
| `--stats` | Show token count estimates and savings (encode only) |
6767
| `--no-strict` | Disable strict validation when decoding |
68+
| `--key-folding <mode>` | Enable key folding: `off`, `safe` (default: `off`) - v1.5 |
69+
| `--flatten-depth <number>` | Maximum folded segment count when key folding is enabled (default: `Infinity`) - v1.5 |
70+
| `--expand-paths <mode>` | Enable path expansion: `off`, `safe` (default: `off`) - v1.5 |
6871

6972
## Advanced Examples
7073

@@ -119,12 +122,81 @@ cat large-dataset.json | toon --delimiter "\t" > output.toon
119122
jq '.results' data.json | toon > filtered.toon
120123
```
121124

125+
### Key Folding (v1.5)
126+
127+
Collapse nested wrapper chains to reduce tokens:
128+
129+
#### Basic key folding
130+
131+
```bash
132+
# Encode with key folding
133+
toon input.json --key-folding safe -o output.toon
134+
```
135+
136+
For data like:
137+
```json
138+
{
139+
"data": {
140+
"metadata": {
141+
"items": ["a", "b"]
142+
}
143+
}
144+
}
145+
```
146+
147+
Output becomes:
148+
```
149+
data.metadata.items[2]: a,b
150+
```
151+
152+
Instead of:
153+
```
154+
data:
155+
metadata:
156+
items[2]: a,b
157+
```
158+
159+
#### Limit folding depth
160+
161+
```bash
162+
# Fold maximum 2 levels deep
163+
toon input.json --key-folding safe --flatten-depth 2 -o output.toon
164+
```
165+
166+
#### Path expansion on decode
167+
168+
```bash
169+
# Reconstruct nested structure from folded keys
170+
toon data.toon --expand-paths safe -o output.json
171+
```
172+
173+
#### Round-trip workflow
174+
175+
```bash
176+
# Encode with folding
177+
toon input.json --key-folding safe -o compressed.toon
178+
179+
# Decode with expansion (restores original structure)
180+
toon compressed.toon --expand-paths safe -o output.json
181+
182+
# Verify round-trip
183+
diff input.json output.json
184+
```
185+
186+
#### Combined with other options
187+
188+
```bash
189+
# Key folding + tab delimiter + stats
190+
toon data.json --key-folding safe --delimiter "\t" --stats -o output.toon
191+
```
192+
122193
## Why Use the CLI?
123194

124195
- **Quick conversions** between formats without writing code
125196
- **Token analysis** to see potential savings before sending to LLMs
126197
- **Pipeline integration** with existing JSON-based workflows
127198
- **Flexible formatting** with delimiter and indentation options
199+
- **Key folding (v1.5)** to collapse nested wrappers for additional token savings
128200

129201
## Related
130202

‎packages/cli/src/conversion.ts‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import type { DecodeOptions, Delimiter, EncodeOptions } from '../../toon/src'
1+
import type { DecodeOptions, EncodeOptions } from '../../toon/src'
22
import type { InputSource } from './types'
33
import * as fsp from 'node:fs/promises'
44
import * as path from 'node:path'
@@ -11,9 +11,11 @@ import { formatInputLabel, readInput } from './utils'
1111
export async function encodeToToon(config: {
1212
input: InputSource
1313
output?: string
14-
delimiter: Delimiter
15-
indent: number
14+
indent: NonNullable<EncodeOptions['indent']>
15+
delimiter: NonNullable<EncodeOptions['delimiter']>
1616
lengthMarker: NonNullable<EncodeOptions['lengthMarker']>
17+
keyFolding?: NonNullable<EncodeOptions['keyFolding']>
18+
flattenDepth?: number
1719
printStats: boolean
1820
}): Promise<void> {
1921
const jsonContent = await readInput(config.input)
@@ -30,6 +32,8 @@ export async function encodeToToon(config: {
3032
delimiter: config.delimiter,
3133
indent: config.indent,
3234
lengthMarker: config.lengthMarker,
35+
keyFolding: config.keyFolding,
36+
flattenDepth: config.flattenDepth,
3337
}
3438

3539
const toonOutput = encode(data, encodeOptions)
@@ -59,8 +63,9 @@ export async function encodeToToon(config: {
5963
export async function decodeToJson(config: {
6064
input: InputSource
6165
output?: string
62-
indent: number
63-
strict: boolean
66+
indent: NonNullable<DecodeOptions['indent']>
67+
strict: NonNullable<DecodeOptions['strict']>
68+
expandPaths?: NonNullable<DecodeOptions['expandPaths']>
6469
}): Promise<void> {
6570
const toonContent = await readInput(config.input)
6671

@@ -69,6 +74,7 @@ export async function decodeToJson(config: {
6974
const decodeOptions: DecodeOptions = {
7075
indent: config.indent,
7176
strict: config.strict,
77+
expandPaths: config.expandPaths,
7278
}
7379
data = decode(toonContent, decodeOptions)
7480
}

‎packages/cli/src/index.ts‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { CommandDef } from 'citty'
2-
import type { Delimiter } from '../../toon/src'
2+
import type { DecodeOptions, Delimiter, EncodeOptions } from '../../toon/src'
33
import type { InputSource } from './types'
44
import * as path from 'node:path'
55
import process from 'node:process'
@@ -51,6 +51,20 @@ export const mainCommand: CommandDef<{
5151
description: string
5252
default: true
5353
}
54+
keyFolding: {
55+
type: 'string'
56+
description: string
57+
default: string
58+
}
59+
flattenDepth: {
60+
type: 'string'
61+
description: string
62+
}
63+
expandPaths: {
64+
type: 'string'
65+
description: string
66+
default: string
67+
}
5468
stats: {
5569
type: 'boolean'
5670
description: string
@@ -103,6 +117,20 @@ export const mainCommand: CommandDef<{
103117
description: 'Enable strict mode for decoding',
104118
default: true,
105119
},
120+
keyFolding: {
121+
type: 'string',
122+
description: 'Enable key folding: off, safe (default: off)',
123+
default: 'off',
124+
},
125+
flattenDepth: {
126+
type: 'string',
127+
description: 'Maximum folded segment count when key folding is enabled (default: Infinity)',
128+
},
129+
expandPaths: {
130+
type: 'string',
131+
description: 'Enable path expansion: off, safe (default: off)',
132+
default: 'off',
133+
},
106134
stats: {
107135
type: 'boolean',
108136
description: 'Show token statistics',
@@ -129,6 +157,27 @@ export const mainCommand: CommandDef<{
129157
throw new Error(`Invalid delimiter "${delimiter}". Valid delimiters are: comma (,), tab (\\t), pipe (|)`)
130158
}
131159

160+
// Validate `keyFolding`
161+
const keyFolding = args.keyFolding || 'off'
162+
if (keyFolding !== 'off' && keyFolding !== 'safe') {
163+
throw new Error(`Invalid keyFolding value "${keyFolding}". Valid values are: off, safe`)
164+
}
165+
166+
// Parse and validate `flattenDepth`
167+
let flattenDepth: number | undefined
168+
if (args.flattenDepth !== undefined) {
169+
flattenDepth = Number.parseInt(args.flattenDepth, 10)
170+
if (Number.isNaN(flattenDepth) || flattenDepth < 0) {
171+
throw new Error(`Invalid flattenDepth value: ${args.flattenDepth}`)
172+
}
173+
}
174+
175+
// Validate `expandPaths`
176+
const expandPaths = args.expandPaths || 'off'
177+
if (expandPaths !== 'off' && expandPaths !== 'safe') {
178+
throw new Error(`Invalid expandPaths value "${expandPaths}". Valid values are: off, safe`)
179+
}
180+
132181
const mode = detectMode(inputSource, args.encode, args.decode)
133182

134183
try {
@@ -140,6 +189,8 @@ export const mainCommand: CommandDef<{
140189
indent,
141190
lengthMarker: args.lengthMarker === true ? '#' : false,
142191
printStats: args.stats === true,
192+
keyFolding: keyFolding as NonNullable<EncodeOptions['keyFolding']>,
193+
flattenDepth,
143194
})
144195
}
145196
else {
@@ -148,6 +199,7 @@ export const mainCommand: CommandDef<{
148199
output: outputPath,
149200
indent,
150201
strict: args.strict !== false,
202+
expandPaths: expandPaths as NonNullable<DecodeOptions['expandPaths']>,
151203
})
152204
}
153205
}

‎packages/toon/src/constants.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ export const COLON = ':'
1212
export const SPACE = ' '
1313
export const PIPE = '|'
1414
export const HASH = '#'
15+
export const DOT = '.'
1516

1617
// #endregion
1718

0 commit comments

Comments
 (0)