Optimize token limit documentation and minor bug fixes#130
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR optimizes token limit functionality and addresses various bug fixes across the codebase. The changes focus on improving performance for tokenization operations, updating documentation for experimental features, and fixing several minor issues.
- Updates default tokenizer model from
gpt-3.5-turbotogpt-4oacross the codebase - Adds experimental whitespace control and token/character limit parameters to text components
- Implements performance optimizations for token counting using byte-count heuristics
Reviewed Changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/poml/_tags.py | Adds experimental whitespace and token control parameters to text and inline functions |
| packages/poml/writer.ts | Optimizes token limit checking with byte-count heuristics and updates default model |
| packages/poml/tests/index.test.tsx | Adds comprehensive tests for whitespace control and token limit features |
| packages/poml/tests/base.test.tsx | Updates parameter validation tests for new experimental attributes |
| packages/poml/essentials.tsx | Updates JSDoc comments for experimental features and adds new parameters |
| packages/poml/base.tsx | Fixes whitespace trimming logic to handle edge cases properly |
| packages/poml/assets/componentDocs.json | Updates component documentation with experimental feature descriptions |
| packages/poml-vscode/panel/types.ts | Adds evaluation message types for improved debugging |
| packages/poml-vscode/lsp/server.ts | Replaces console logging with structured notification system |
| packages/poml-vscode/extension.ts | Implements output channel for evaluation results |
| packages/poml-vscode/command/testCommand.ts | Minor formatting improvement in conditional logic |
| mkdocs.yml | Restructures documentation navigation and adds new sections |
| examples/expects/301_generate_poml.txt | Removes extra blank lines for cleaner output |
| docs/* | Multiple documentation updates including new whitespace and token control guides |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment on lines
+215
to
+222
| let trimmed = child; | ||
| if (index === 0) { | ||
| trimmed = trimmed.trimStart(); | ||
| } | ||
| if (index === flattenedChildren.length - 1) { | ||
| trimmed = trimmed.trimEnd(); | ||
| } | ||
| return trimmed; |
There was a problem hiding this comment.
The variable trimmed is declared but the original child parameter is not being used for the trimming operations. This could lead to unexpected behavior where the trimming operations are applied to a copy but the modifications don't affect the actual string being processed.
Suggested change
| let trimmed = child; | |
| if (index === 0) { | |
| trimmed = trimmed.trimStart(); | |
| } | |
| if (index === flattenedChildren.length - 1) { | |
| trimmed = trimmed.trimEnd(); | |
| } | |
| return trimmed; | |
| let result = child; | |
| if (index === 0) { | |
| result = result.trimStart(); | |
| } | |
| if (index === flattenedChildren.length - 1) { | |
| result = result.trimEnd(); | |
| } | |
| return result; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.