Skip to content

feat: automatic batching #85

@giladgd

Description

@giladgd

Also, automatically set the right contextSize and provide other good defaults to make the usage smoother.

  • Support configuring the context swapping size for infinite text generation (by default, it'll be automatic and dynamic depending on the prompt)

Metadata

Metadata

Assignees

Labels

new featureNew feature or requestreleasedreleased on @betaroadmapPart of the roadmap for node-llama-cpp (https://github.com/orgs/withcatai/projects/1)

Type

No type

Projects

Status

Done

Relationships

None yet

Development

No branches or pull requests

Issue actions