Skip to content

Commit 2c51932

Browse files
johannschopplichjohannschopplich
committed
feat: add replacer function for encoding transformations and filtering (closes #209)
1 parent 0974a58 commit 2c51932

File tree

6 files changed

+788
-2
lines changed

6 files changed

+788
-2
lines changed

‎docs/reference/api.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,124 @@ for (const line of encodeLines(data, { delimiter: '\t' })) {
144144
stream.end()
145145
```
146146

147+
### Replacer Function
148+
149+
The `replacer` option allows you to transform or filter values during encoding. It works similarly to `JSON.stringify`'s replacer parameter, but with path tracking for more precise control.
150+
151+
#### Type Signature
152+
153+
```typescript
154+
type EncodeReplacer = (
155+
key: string,
156+
value: JsonValue,
157+
path: readonly (string | number)[]
158+
) => unknown
159+
```
160+
161+
#### Parameters
162+
163+
| Parameter | Type | Description |
164+
|-----------|------|-------------|
165+
| `key` | `string` | Property name, array index (as string), or empty string for root |
166+
| `value` | `JsonValue` | The normalized value at this location |
167+
| `path` | `readonly (string \| number)[]` | Path from root to current value |
168+
169+
#### Return Value
170+
171+
- Return the value unchanged to keep it
172+
- Return a different value to replace it (will be normalized)
173+
- Return `undefined` to omit properties/array elements
174+
- For root value, `undefined` means "no change" (root cannot be omitted)
175+
176+
#### Examples
177+
178+
**Filtering sensitive data:**
179+
180+
```typescript
181+
import { encode } from '@toon-format/toon'
182+
183+
const data = {
184+
user: { name: 'Alice', password: 'secret123', email: '[email protected]' }
185+
}
186+
187+
function replacer(key, value) {
188+
if (key === 'password')
189+
return undefined
190+
return value
191+
}
192+
193+
console.log(encode(data, { replacer }))
194+
```
195+
196+
**Output:**
197+
198+
```yaml
199+
user:
200+
name: Alice
201+
email: [email protected]
202+
```
203+
204+
**Transforming values:**
205+
206+
```typescript
207+
const data = { user: 'alice', role: 'admin' }
208+
209+
function replacer(key, value) {
210+
if (typeof value === 'string')
211+
return value.toUpperCase()
212+
return value
213+
}
214+
215+
console.log(encode(data, { replacer }))
216+
```
217+
218+
**Output:**
219+
220+
```yaml
221+
user: ALICE
222+
role: ADMIN
223+
```
224+
225+
**Path-based transformations:**
226+
227+
```typescript
228+
const data = {
229+
metadata: { created: '2025-01-01' },
230+
user: { created: '2025-01-02' }
231+
}
232+
233+
function replacer(key, value, path) {
234+
// Add timezone info only to top-level metadata
235+
if (path.length === 1 && path[0] === 'metadata' && key === 'created') {
236+
return `${value}T00:00:00Z`
237+
}
238+
return value
239+
}
240+
241+
console.log(encode(data, { replacer }))
242+
```
243+
244+
**Output:**
245+
246+
```yaml
247+
metadata:
248+
created: 2025-01-01T00:00:00Z
249+
user:
250+
created: 2025-01-02
251+
```
252+
253+
::: tip Replacer Execution Order
254+
The replacer is called in a depth-first manner:
255+
1. Root value first (key = `''`, path = `[]`)
256+
2. Then each property/element (with proper key and path)
257+
3. Values are re-normalized after replacement
258+
4. Children are processed after parent transformation
259+
:::
260+
261+
::: warning Array Indices as Strings
262+
Following `JSON.stringify` behavior, array indices are passed as strings (`'0'`, `'1'`, `'2'`, etc.) to the replacer, not as numbers.
263+
:::
264+
147265
## Decoding Functions
148266

149267
### `decode(input, options?)`
@@ -375,6 +493,7 @@ Configuration for [`encode()`](#encode-input-options) and [`encodeLines()`](#enc
375493
| `delimiter` | `','` \| `'\t'` \| `'\|'` | `','` | Delimiter for array values and tabular rows |
376494
| `keyFolding` | `'off'` \| `'safe'` | `'off'` | Enable key folding to collapse single-key wrapper chains into dotted paths |
377495
| `flattenDepth` | `number` | `Infinity` | Maximum number of segments to fold when `keyFolding` is enabled (values 0-1 have no practical effect) |
496+
| `replacer` | `EncodeReplacer` | `undefined` | Optional hook to transform or omit values before encoding (see [Replacer Function](#replacer-function)) |
378497

379498
**Delimiter options:**
380499

‎packages/toon/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -785,6 +785,32 @@ for (const line of encodeLines(largeData)) {
785785
> [!TIP]
786786
> For streaming decode APIs, see [`decodeFromLines()`](https://toonformat.dev/reference/api#decodefromlines-lines-options) and [`decodeStream()`](https://toonformat.dev/reference/api#decodestream-source-options).
787787
788+
**Transforming values with replacer:**
789+
790+
```ts
791+
import { encode } from '@toon-format/toon'
792+
793+
// Remove sensitive fields
794+
const user = { name: 'Alice', password: 'secret', email: '[email protected]' }
795+
const safe = encode(user, {
796+
replacer: (key, value) => key === 'password' ? undefined : value
797+
})
798+
// name: Alice
799+
// email: [email protected]
800+
801+
// Transform values
802+
const data = { status: 'active', count: 5 }
803+
const transformed = encode(data, {
804+
replacer: (key, value) =>
805+
typeof value === 'string' ? value.toUpperCase() : value
806+
})
807+
// status: ACTIVE
808+
// count: 5
809+
```
810+
811+
> [!TIP]
812+
> The `replacer` function provides fine-grained control over encoding, similar to `JSON.stringify`'s replacer but with path tracking. See the [API Reference](https://toonformat.dev/reference/api#replacer-function) for more examples.
813+
788814
## Playgrounds
789815

790816
Experiment with TOON format interactively using these tools for token comparison, format conversion, and validation.

‎packages/toon/src/encode/replacer.ts‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
import type { EncodeReplacer, JsonArray, JsonObject, JsonValue } from '../types'
2+
import { isJsonArray, isJsonObject, normalizeValue } from './normalize'
3+
4+
/**
5+
* Applies a replacer function to a `JsonValue` and all its descendants.
6+
*
7+
* The replacer is called for:
8+
* - The root value (with key='', path=[])
9+
* - Every object property (with the property name as key)
10+
* - Every array element (with the string index as key: '0', '1', etc.)
11+
*
12+
* @param root - The normalized `JsonValue` to transform
13+
* @param replacer - The replacer function to apply
14+
* @returns The transformed `JsonValue`
15+
*/
16+
export function applyReplacer(root: JsonValue, replacer: EncodeReplacer): JsonValue {
17+
// Call replacer on root with empty string key and empty path
18+
const replacedRoot = replacer('', root, [])
19+
20+
// For root, undefined means "no change" (don't omit the root)
21+
if (replacedRoot === undefined) {
22+
return transformChildren(root, replacer, [])
23+
}
24+
25+
// Normalize the replaced value (in case user returned non-JsonValue)
26+
const normalizedRoot = normalizeValue(replacedRoot)
27+
28+
// Recursively transform children
29+
return transformChildren(normalizedRoot, replacer, [])
30+
}
31+
32+
/**
33+
* Recursively transforms the children of a `JsonValue` using the replacer.
34+
*
35+
* @param value - The value whose children should be transformed
36+
* @param replacer - The replacer function to apply
37+
* @param path - Current path from root
38+
* @returns The value with transformed children
39+
*/
40+
function transformChildren(
41+
value: JsonValue,
42+
replacer: EncodeReplacer,
43+
path: readonly (string | number)[],
44+
): JsonValue {
45+
if (isJsonObject(value)) {
46+
return transformObject(value, replacer, path)
47+
}
48+
49+
if (isJsonArray(value)) {
50+
return transformArray(value, replacer, path)
51+
}
52+
53+
// Primitives have no children
54+
return value
55+
}
56+
57+
/**
58+
* Transforms an object by applying the replacer to each property.
59+
*
60+
* @param obj - The object to transform
61+
* @param replacer - The replacer function to apply
62+
* @param path - Current path from root
63+
* @returns A new object with transformed properties
64+
*/
65+
function transformObject(
66+
obj: JsonObject,
67+
replacer: EncodeReplacer,
68+
path: readonly (string | number)[],
69+
): JsonObject {
70+
const result: Record<string, JsonValue> = {}
71+
72+
for (const [key, value] of Object.entries(obj)) {
73+
// Call replacer with the property key and current path
74+
const childPath = [...path, key]
75+
const replacedValue = replacer(key, value, childPath)
76+
77+
// undefined means omit this property
78+
if (replacedValue === undefined) {
79+
continue
80+
}
81+
82+
// Normalize the replaced value
83+
const normalizedValue = normalizeValue(replacedValue)
84+
85+
// Recursively transform children of the replaced value
86+
result[key] = transformChildren(normalizedValue, replacer, childPath)
87+
}
88+
89+
return result
90+
}
91+
92+
/**
93+
* Transforms an array by applying the replacer to each element.
94+
*
95+
* @param arr - The array to transform
96+
* @param replacer - The replacer function to apply
97+
* @param path - Current path from root
98+
* @returns A new array with transformed elements
99+
*/
100+
function transformArray(
101+
arr: JsonArray,
102+
replacer: EncodeReplacer,
103+
path: readonly (string | number)[],
104+
): JsonArray {
105+
const result: JsonValue[] = []
106+
107+
for (let i = 0; i < arr.length; i++) {
108+
const value = arr[i]!
109+
// Call replacer with string index (`'0'`, `'1'`, etc.) to match `JSON.stringify` behavior
110+
const childPath = [...path, i]
111+
const replacedValue = replacer(String(i), value, childPath)
112+
113+
// undefined means omit this element
114+
if (replacedValue === undefined) {
115+
continue
116+
}
117+
118+
// Normalize the replaced value
119+
const normalizedValue = normalizeValue(replacedValue)
120+
121+
// Recursively transform children of the replaced value
122+
result.push(transformChildren(normalizedValue, replacer, childPath))
123+
}
124+
125+
return result
126+
}

‎packages/toon/src/index.ts‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { buildValueFromEvents } from './decode/event-builder'
55
import { expandPathsSafe } from './decode/expand'
66
import { encodeJsonValue } from './encode/encoders'
77
import { normalizeValue } from './encode/normalize'
8+
import { applyReplacer } from './encode/replacer'
89

910
export { DEFAULT_DELIMITER, DELIMITERS } from './constants'
1011
export type {
@@ -13,6 +14,7 @@ export type {
1314
Delimiter,
1415
DelimiterKey,
1516
EncodeOptions,
17+
EncodeReplacer,
1618
JsonArray,
1719
JsonObject,
1820
JsonPrimitive,
@@ -97,7 +99,13 @@ export function decode(input: string, options?: DecodeOptions): JsonValue {
9799
export function encodeLines(input: unknown, options?: EncodeOptions): Iterable<string> {
98100
const normalizedValue = normalizeValue(input)
99101
const resolvedOptions = resolveOptions(options)
100-
return encodeJsonValue(normalizedValue, resolvedOptions, 0)
102+
103+
// Apply replacer if provided
104+
const maybeReplacedValue = resolvedOptions.replacer
105+
? applyReplacer(normalizedValue, resolvedOptions.replacer)
106+
: normalizedValue
107+
108+
return encodeJsonValue(maybeReplacedValue, resolvedOptions, 0)
101109
}
102110

103111
/**
@@ -210,6 +218,7 @@ function resolveOptions(options?: EncodeOptions): ResolvedEncodeOptions {
210218
delimiter: options?.delimiter ?? DEFAULT_DELIMITER,
211219
keyFolding: options?.keyFolding ?? 'off',
212220
flattenDepth: options?.flattenDepth ?? Number.POSITIVE_INFINITY,
221+
replacer: options?.replacer,
213222
}
214223
}
215224

‎packages/toon/src/types.ts‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,42 @@ export type JsonValue = JsonPrimitive | JsonObject | JsonArray
1313

1414
export type { Delimiter, DelimiterKey }
1515

16+
/**
17+
* A function that transforms or filters values during encoding.
18+
*
19+
* Called for every value (root, object properties, array elements) during the encoding process.
20+
* Similar to `JSON.stringify`'s replacer, but with path tracking.
21+
*
22+
* @param key - The property key or array index (as string). Empty string (`''`) for root value.
23+
* @param value - The normalized `JsonValue` at this location.
24+
* @param path - Array representing the path from root to this value.
25+
*
26+
* @returns The replacement value (will be normalized again), or `undefined` to omit.
27+
* For root value, returning `undefined` means "no change" (don't omit root).
28+
*
29+
* @example
30+
* ```ts
31+
* // Remove password fields
32+
* const replacer = (key, value) => {
33+
* if (key === 'password') return undefined
34+
* return value
35+
* }
36+
*
37+
* // Add timestamps
38+
* const replacer = (key, value, path) => {
39+
* if (path.length === 0 && typeof value === 'object' && value !== null) {
40+
* return { ...value, _timestamp: Date.now() }
41+
* }
42+
* return value
43+
* }
44+
* ```
45+
*/
46+
export type EncodeReplacer = (
47+
key: string,
48+
value: JsonValue,
49+
path: readonly (string | number)[],
50+
) => unknown
51+
1652
export interface EncodeOptions {
1753
/**
1854
* Number of spaces per indentation level.
@@ -38,9 +74,16 @@ export interface EncodeOptions {
3874
* @default Infinity
3975
*/
4076
flattenDepth?: number
77+
/**
78+
* A function to transform or filter values during encoding.
79+
* Called for the root value and every nested property/element.
80+
* Return `undefined` to omit properties/elements (root cannot be omitted).
81+
* @default undefined
82+
*/
83+
replacer?: EncodeReplacer
4184
}
4285

43-
export type ResolvedEncodeOptions = Readonly<Required<EncodeOptions>>
86+
export type ResolvedEncodeOptions = Readonly<Required<Omit<EncodeOptions, 'replacer'>>> & Pick<EncodeOptions, 'replacer'>
4487

4588
// #endregion
4689

0 commit comments

Comments
 (0)