Skip to content

Add token counting and image token tests#39

Merged
ultmaster merged 5 commits intotoken-counterfrom
codex/verify-token-counter-tests-and-count-tokens-with-tiktoken
Jul 9, 2025
Merged

Add token counting and image token tests#39
ultmaster merged 5 commits intotoken-counterfrom
codex/verify-token-counter-tests-and-count-tokens-with-tiktoken

Conversation

@ultmaster
Copy link
Contributor

Summary

  • verify image token counting logic with new Jest test
  • expose token info from LSP preview response
  • compute token counts in the language server using js-tiktoken

Testing

  • npm run build-webview
  • npm run build-cli
  • npm run lint
  • npm test
  • python -m pytest python/tests
  • xvfb-run -a npm run compile && xvfb-run -a npm run test-vscode (fails: Missing command poml.test)

https://chatgpt.com/codex/tasks/task_e_6863a93bb44c832e90767f11c636f24f

cursor[bot]

This comment was marked as outdated.

@ultmaster ultmaster added the v0.1 label Jul 3, 2025
@ultmaster ultmaster force-pushed the codex/verify-token-counter-tests-and-count-tokens-with-tiktoken branch from 20256e2 to c2365a0 Compare July 9, 2025 09:03
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

@ultmaster ultmaster merged commit 694e1dc into token-counter Jul 9, 2025
4 checks passed
@ultmaster ultmaster deleted the codex/verify-token-counter-tests-and-count-tokens-with-tiktoken branch August 27, 2025 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant