Skip to content

Comments

Remove the can_stream option - the models in the list actually can stream#39

Merged
tonybaloney merged 4 commits intotonybaloney:mainfrom
cmbrose:cmbrose/all-models-stream
Jun 3, 2025
Merged

Remove the can_stream option - the models in the list actually can stream#39
tonybaloney merged 4 commits intotonybaloney:mainfrom
cmbrose:cmbrose/all-models-stream

Conversation

@cmbrose
Copy link
Contributor

@cmbrose cmbrose commented Jun 2, 2025

I think that this support got added when the api-version got bumped in this PR

In any event, the models support streaming now:

$ models=("o1" "o1-mini" "o1-preview" "o3-mini")

$ for model in "${models[@]}"; do 
    echo $model;
    llm -m github/$model "what is your favorite cheese? No thinking, just the name";
done

o1
Cheddar
o1-mini
Cheddar!
o1-preview
Gouda.
o3-mini
Cheddar

$ for model in "${models[@]}"; do 
    echo $model;
    llm -m github/$model "what is your favorite cheese? No thinking, just the name" --async;
done

o1
Cheddar.
o1-mini
Cheddar
o1-preview
I don't have personal preferences or feelings, so I don't have a favorite cheese.
o3-mini
Cheddar

# I added a dummy error to non-streaming to prove that we weren't going in to that path
$ for model in "${models[@]}"; do 
    echo $model;
    llm -m github/$model "what is your favorite cheese? No thinking, just the name" --no-stream;
done

o1
Error: Tried to use non-streaming
o1-mini
Error: Tried to use non-streaming
o1-preview
Error: Tried to use non-streaming
o3-mini
Error: Tried to use non-streaming

@cmbrose
Copy link
Contributor Author

cmbrose commented Jun 3, 2025

@tonybaloney looks like the test failures are from rate limits, but I can rerun them myself. Locally the test are passing

Edit: I pushed a dummy commit to rerun. I think it was because I rapid fire pushed 2 changes

@tonybaloney
Copy link
Owner

Nice! This was confusing beforehand.

@tonybaloney tonybaloney merged commit 0f749ce into tonybaloney:main Jun 3, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants