Repository navigation
Commit 15667de
perf(voice): stop token counting at the first oversized boundary
Each boundary scan in split_at_natural_boundaries tokenized every
word and clause boundary through end-of-text, skipping candidates
that already exceeded the limit instead of stopping at them. Because
token_count re-encodes the prefix text[start..end], tokenizer input
grew superlinearly in prompt length, and that work is paid before the
first chunk reaches synthesis -- taxing the time-to-first-audio this
segmentation is meant to improve. A 1.9 KB prompt at the 2000-char
MAX_TTS_TEXT_LEN cap tokenized 2.36 MB, and doubling a prompt
multiplied tokenizer input by ~5.5x.
Prepared token counts are monotonic in prefix length, so the first
candidate that overflows proves no longer candidate can fit. Break
out of the scan there.
Chunk boundaries are unchanged: verified output-identical against the
previous behavior across 5040 cases (21 corpora x max_tokens 1..=60 x
both split policies x two tokenizer shapes), while saving 1,073,280
tokenizer calls and 568 MB of tokenized input.
Co-authored-by: Tyler Longwell <[email protected]>
Signed-off-by: Tyler Longwell <[email protected]>1 parent 4e6d2df commit 15667de
1 file changed
Lines changed: 42 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
672 | 672 | | |
673 | 673 | | |
674 | 674 | | |
675 | | - | |
| 675 | + | |
676 | 676 | | |
677 | 677 | | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
678 | 686 | | |
679 | 687 | | |
680 | 688 | | |
| |||
1078 | 1086 | | |
1079 | 1087 | | |
1080 | 1088 | | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
1081 | 1122 | | |
1082 | 1123 | | |
1083 | 1124 | | |
| |||
0 commit comments