Skip to content

Commit 2e992ae

Browse files
feat: encodeLines for streaming encoding to TOON
1 parent 660ed21 commit 2e992ae

File tree

7 files changed

+269
-99
lines changed

7 files changed

+269
-99
lines changed

docs/cli/index.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,16 @@ cat data.json | toon -
102102
cat data.toon | toon --decode
103103
```
104104

105+
## Performance
106+
107+
### Streaming Encoding
108+
109+
JSON→TOON conversions use line-by-line encoding internally, which avoids holding the entire TOON document in memory. This makes the CLI efficient for large datasets without requiring additional configuration.
110+
111+
::: info Token Statistics
112+
When using the `--stats` flag, the CLI builds the full TOON string once to compute accurate token counts. For maximum memory efficiency on very large files, omit `--stats`.
113+
:::
114+
105115
## Options
106116

107117
| Option | Description |

docs/reference/api.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,68 @@ encode(data, { delimiter: '\t', keyFolding: 'safe' })
127127
```
128128
:::
129129

130+
## `encodeLines(value, options?)`
131+
132+
Converts any JSON-serializable value to TOON format as a sequence of lines, without building the full string in memory. Suitable for streaming large outputs to files, HTTP responses, or process stdout.
133+
134+
```ts
135+
import { encodeLines } from '@toon-format/toon'
136+
137+
// Stream to stdout
138+
for (const line of encodeLines(data)) {
139+
console.log(line)
140+
}
141+
142+
// Write to file line-by-line
143+
const lines = encodeLines(data, { indent: 2, delimiter: '\t' })
144+
for (const line of lines) {
145+
await writeToStream(`${line}\n`)
146+
}
147+
148+
// Collect to array
149+
const lineArray = Array.from(encodeLines(data))
150+
```
151+
152+
### Parameters
153+
154+
| Parameter | Type | Description |
155+
|-----------|------|-------------|
156+
| `value` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) |
157+
| `options` | `EncodeOptions?` | Optional encoding options (same as `encode()`) |
158+
159+
### Return Value
160+
161+
Returns an `Iterable<string>` that yields TOON lines one at a time. Each yielded string is a single line without a trailing newline character.
162+
163+
::: info Relationship to `encode()`
164+
`encode(value, options)` is equivalent to:
165+
```ts
166+
Array.from(encodeLines(value, options)).join('\n')
167+
```
168+
:::
169+
170+
### Example
171+
172+
```ts
173+
import { createWriteStream } from 'node:fs'
174+
import { encodeLines } from '@toon-format/toon'
175+
176+
const data = {
177+
items: Array.from({ length: 100000 }, (_, i) => ({
178+
id: i,
179+
name: `Item ${i}`,
180+
value: Math.random()
181+
}))
182+
}
183+
184+
// Stream large dataset to file
185+
const stream = createWriteStream('output.toon')
186+
for (const line of encodeLines(data, { delimiter: '\t' })) {
187+
stream.write(`${line}\n`)
188+
}
189+
stream.end()
190+
```
191+
130192
## `decode(input, options?)`
131193

132194
Converts a TOON-formatted string back to JavaScript values.

packages/cli/src/conversion.ts

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import * as path from 'node:path'
55
import process from 'node:process'
66
import { consola } from 'consola'
77
import { estimateTokenCount } from 'tokenx'
8-
import { decode, encode } from '../../toon/src'
8+
import { decode, encode, encodeLines } from '../../toon/src'
99
import { formatInputLabel, readInput } from './utils'
1010

1111
export async function encodeToToon(config: {
@@ -34,7 +34,17 @@ export async function encodeToToon(config: {
3434
flattenDepth: config.flattenDepth,
3535
}
3636

37-
const toonOutput = encode(data, encodeOptions)
37+
let toonOutput: string
38+
39+
// When printing stats, we need the full string for token counting
40+
if (config.printStats) {
41+
toonOutput = encode(data, encodeOptions)
42+
}
43+
else {
44+
// Use streaming encoder for non-stats path
45+
const lines = Array.from(encodeLines(data, encodeOptions))
46+
toonOutput = lines.join('\n')
47+
}
3848

3949
if (config.output) {
4050
await fsp.writeFile(config.output, toonOutput, 'utf-8')

0 commit comments

Comments
 (0)