fix: truncation agent token calculations#915
Merged
kalvinnchau merged 7 commits intomainfrom Jan 30, 2025
Merged
Conversation
…ssages previously count_tokens(&msg.as_concat_text) would not count ToolResponses, update that to use count_chat_tokens on each individual message make count_tokens_for_tools a public method to use to account for tool token counts update the context_limit to take into account for the system_prompt, and tool request token counts
at this point we already know what message is being removed, the total_tokens count doesn't get used after this point and we just remove the paired message
kalvinnchau
changed the title
salman1993
reviewed
Jan 29, 2025
crates/goose/src/agents/truncate.rs
Outdated
| let context_limit = remaining_tokens; | ||
|
|
||
| // Calculate current token count of each message, use count_chat_tokens to ensure we | ||
| // capture the full content of the message |
Contributor
There was a problem hiding this comment.
lets mention tool response in the comment
wendytang
reviewed
Jan 30, 2025
| // Calculate current token count | ||
| // Take into account the system prompt, and our tools input and subtract that from the | ||
| // remaining context limit | ||
| let system_prompt_token_count = self.token_counter.count_tokens(system_prompt); |
There was a problem hiding this comment.
does we pass the system_prompt back and forth in the reply loop?
if we do, i wonder if subtracting:
count_tokens(system_prompt) * num_user_exchanges
would be accurate
Contributor
Author
There was a problem hiding this comment.
i believe we just pass it in once, and not within the messages in the loop
salman1993
approved these changes
Jan 30, 2025
salman1993
changed the title
Closed
salman1993
added a commit
that referenced
this pull request
Jan 30, 2025
* origin/main: fix: clarify linux cli install only (#927) feat: update ui for ollama host (#912) feat: add CONFIGURE=false option in install script (#920) fix: truncation agent token calculations (#915) fix: request payload for o1 models (#921) Update SupportedEnvironments.js so others don't get confused on why they can not open the macos app on x86 (#888) fix: improve configure process with error message (#919) docs: Goose on Windows via WSL (#901) fix: more graceful handling of missing usage in provider response (#907) feat: rm uv.lock cause it points to square artifactory (#917) feat: Update issue templates for bug report for goose (#913) fix: post endpoint url on sse endpoint event (#900)
michaelneale
added a commit
that referenced
this pull request
Jan 30, 2025
* main: chore: remove gpt-3.5-turbo UI suggestion, as it is deprecated (#959) chore: remove o1-mini suggestion from UI add model view (#957) fix: missing field in request (#956) docs: update provider docs, fix rate limit link (#943) fix: clarify linux cli install only (#927) feat: update ui for ollama host (#912) feat: add CONFIGURE=false option in install script (#920) fix: truncation agent token calculations (#915) fix: request payload for o1 models (#921)
michaelneale
added a commit
that referenced
this pull request
Jan 31, 2025
* main: (28 commits) ci: per semver build metadata should be after + (#971) fix: temp fix to make CI workflow pass (#970) chore: bump patch version to 1.0.3 (#967) fix: load shell automatically from env for GUI (#948) fix: update versions in release and canary workflows (#911) docs: fix typo, name (#963) docs: typo fix (#961) chore: remove gpt-3.5-turbo UI suggestion, as it is deprecated (#959) chore: remove o1-mini suggestion from UI add model view (#957) fix: missing field in request (#956) docs: update provider docs, fix rate limit link (#943) fix: clarify linux cli install only (#927) feat: update ui for ollama host (#912) feat: add CONFIGURE=false option in install script (#920) fix: truncation agent token calculations (#915) fix: request payload for o1 models (#921) Update SupportedEnvironments.js so others don't get confused on why they can not open the macos app on x86 (#888) fix: improve configure process with error message (#919) docs: Goose on Windows via WSL (#901) fix: more graceful handling of missing usage in provider response (#907) ...
ahau-square
pushed a commit
that referenced
this pull request
May 2, 2025
cbruyndoncx
pushed a commit
to cbruyndoncx/goose
that referenced
this pull request
Jul 20, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
truncation agent token calculations
truncate_messagesinstead of ignoring themself.token_counter.count_tokens(&msg.as_concat_text())was only counting messages of typeMessage::Text, ignoring anyToolResponses which tend to be a large amount of tokensself.token_counter.count_chat_tokens("", std::slice::from_ref(msg), &[])to get a full count of the message tokenscontext_limitcalculation to subtract the total amount ofsystem_promptandtoolstokens already in use