fix: update workspace chat lastUpdatedAt - #6288

* bump to patched @mintplex-labs/mdpdf pkg

* Identify AnythingLLM Perplexity requests (#6206)

* Add Perplexity integration attribution

* Use AnythingLLM user agent for Perplexity requests

* Preserve Perplexity attribution header

* Use versioned Perplexity attribution

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Add `X-Client-Info` to You.com attribution (#6221)

* Google Vertex LLM Provider support (#6218)

* fix: stop fillSourceWindow reversing the caller's chat history array (#6231)

* fix(chat): stop fillSourceWindow reversing the caller's history array

fillSourceWindow walks history newest-first, but `history.reverse()` reverses
in place, so the array the caller passed in is left reversed once it returns.

stream.js and apiChatHandler.js both keep using that same rawHistory after
the backfill, handing it to chatPrompt and compressMessages. When the prompt
does not fit and messageArrayCompressor runs, it reverses again and walks the
result as if it were newest-first, so its "prioritize the most recent 3
message pairs" rule keeps the oldest exchanges instead. Memory reranking has
the same assumption, reading rawHistory.slice(-3) for the newest turns.

Copy before reversing. The iteration order the function wants is unchanged,
and no caller relies on the array coming back reversed.

* update test suite

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: keep workspace prompt and temperature when an embed sends no override (#6232)

fix(embed): keep workspace prompt and temperature when no override is sent

The embed stream endpoint defaults prompt and temperature to null, so a
widget that sends neither still reaches streamChatWithForEmbed with both
overrides null. The assignments were gated only on the permission flag, so
enabling an override discarded the workspace configuration:

- openAiPrompt became null, and chatPrompt's `??` fallback then served the
  stock default system prompt instead of the workspace's.
- parseFloat(null) is NaN, which is not nullish, so the openAiTemp fallback
  could not recover and the connector received NaN, which serializes as null
  in the outbound JSON.

Apply each override only when the request actually carried a value, which is
how the model override on the line above already behaves. An empty prompt and
a zero temperature stay valid overrides.

Co-authored-by: Timothy Carambat <[email protected]>

* pass include status on sessionID chat getter (#6237)

* pass include status on sessionID chat getter

* improve test coverage for module
patch reset status for MuM and SuM for how API chat handling and resets work

* tests

* fix: stop $ sequences corrupting restored Telegram code blocks (#6235)

* fix(telegram): stop $ sequences corrupting restored code blocks

markdownToTelegram extracts code blocks to placeholders, formats the rest,
then restores them with String.prototype.replace. The second argument there is
a replacement pattern, so $$, $&, $` and $' are expanded rather than inserted
literally, and code blocks are exactly where those live.

Observed on the current code:

  ```bash
  sed -E "s/foo/[$&]/" file
  ```

renders with the internal placeholder pasted into the user's own code block,
because $& expanded to the matched placeholder text. And

  ```make
  all:
  	echo $$HOME
  ```

renders as `echo $HOME`.

markdownToTelegram is on the live send path for every bot reply, through both
editMessage and sendFormattedMessage, so any shell, Makefile or sed answer can
come back mangled.

Pass a function replacer instead, which inserts the captured text verbatim.
The repo already uses function replacers for this reason in
server/utils/chats/index.js.

* expand test coverage

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Add Swedish frontend locale (#6227)

* Add Swedish frontend locale

* linting

---------

Co-authored-by: Daniel Nylander <[email protected]>
Co-authored-by: Timothy Carambat <[email protected]>

* fix: keep thread and user on the query-mode refusal row (#6236)

fix(api): keep thread and user on the query-mode refusal row

chatSync's query-mode refusal writes a chat row without threadId or user, so
the turn is stored with user_id: null and thread_id: null.

Every other WorkspaceChats.new in this file passes the scoping triple it was
invoked with, including the streaming twin of this exact branch, which saves
the same refusal text with threadId and user.

The row is then unattributed in admin Workspace Chats and in exports, and
because workspace_chats.thread_id has no foreign key while
WorkspaceThread.delete removes chats by thread_id, it is orphaned for good.
Same user action against /chat and /stream-chat persists differently.

The existing suite already states this contract: "the agent save MUST persist
the same triple it was invoked with, exactly like the non-agent saves in the
same file, or the turn is invisible everywhere." This is one of those
non-agent saves.

Covers all four real-world parameter combinations the v1 endpoints produce.

Co-authored-by: Timothy Carambat <[email protected]>

* feat: replace the Docker Model Runner provider with llmman (#6226)

* feat: replace the Docker Model Runner provider with llmman

Swaps the Docker Model Runner LLM provider for llmman
(https://github.com/llmmanorg/llmman), a local model runner that serves
the Ollama API alongside OpenAI- and Anthropic-compatible ones.

The provider is modelled on the existing Ollama-API provider rather than
on the one it replaces, since llmman serves that API directly: the same
client library, /api/tags for model discovery, and native tool calling
with the UnTooled fallback. That also brings keep-alive and auth-token
support, which the removed provider did not have.

Removed, with no references left behind:
  - AiProviders/dockerModelRunner
  - agents/aibitat/providers/dockerModelRunner.js
  - endpoints/utils/dockerModelRunnerUtils.js (its /models/create
    streaming download has no counterpart; llmman pulls by reference)
  - frontend LLMSelection/DockerModelRunnerOptions and models/utils/dmrUtils.js

Added: AiProviders/llmman, the agent provider, a Langchain chat model
mirroring the Ollama-API one so context-window preferences carry over,
LlmmanOptions in the UI, and LLMMAN_COMMON_URLS defaulting to port 17434.

Env keys move from DOCKER_MODEL_RUNNER_* to LLMMAN_*, with
LLMMAN_KEEP_ALIVE_TIMEOUT and LLMMAN_AUTH_TOKEN added to match.

Signed-off-by: Eric Curtin <[email protected]>

* leftover keys and lint issue

* update ignore

* update llmman functionality and confirm QA

---------

Signed-off-by: Eric Curtin <[email protected]>
Co-authored-by: Timothy Carambat <[email protected]>

* fix: return the API Call block's response body instead of a parse message (#6240)

* fix(agent-flows): return the API Call body instead of a parse message

executeApiCall runs an already-successful response body through
safeJsonParse with a human-readable sentence as the fallback value:

  safeJsonParse(text, "Failed to parse output from API call block")

safeJsonParse returns its fallback as the parsed VALUE, so a successful
plain-text body with no extractable JSON resolves to that sentence. The
block reports success and hands the caller a fabricated error string as
if it were the response.

A 200 returning "Successfully created item with id 42" lands in
resultVariable, or in directOutput, as "Failed to parse output from API
call block". The function's own JSDoc says it returns "Response data".

This restores the behavior the block shipped with. Before e4305d8f the
same success path read:

  try {
    return JSON.stringify(JSON.parse(data));
  } catch {
    // If not JSON, return as text
    return data;
  }

That commit refactored the call onto safeJsonParse and substituted the
message for the documented plain-text fallback. Falling back to the raw
text puts it back, and leaves the JSON path as it is today.

Every other safeJsonParse call in the repo passes null, {}, [] or a
shaped default rather than a message.

* use plain JSON.parse fallback for API call responses and expand executor test coverage

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: report a failed password update instead of always answering success (#6245)

* fix(system): report a failed password update instead of always succeeding

/system/update-password reads the result like this:

  error = await updateENV({ AuthToken, JWTSecret }, true)?.error;

await binds looser than member access, so this is
await (updateENV(...)?.error). updateENV is async, so .error is read off
the pending Promise and is always undefined. The endpoint answers
200 {"success": true} whether or not the update succeeded.

The half that bites a running server is the rejection. The Promise is
never awaited, so a rejection does not reach the handler's catch. It
becomes an unhandledRejection, and @ladjs/graceful installs a listener
for that through BackgroundWorkers, which bootHTTP and bootSSL both
start. That listener deliberately re-throws, which lands in its
uncaughtException handler, which logs and calls process.exit(1). So the
server exits instead of answering 500. updateENV calls dumpENV in
production, which writes the .env file, so an unwritable storage
directory is enough to reach it.

The other half, a truthy error, is not reachable from the shipped UI
today: both call sites pre-validate with the same character regex the
server enforces, and with force=true that check is the only thing that
sets error for these keys. It is reachable from any non-browser client,
and onboarding shipped without that regex until #5037, so the guard is
one drift away from mattering again.

This restores what the endpoint did before bfc9a96e, which replaced a
destructured result with the inline optional read.

The added test covers the reported failure, the success path, and the
rejection reaching the catch. It is written so the rejection is
pre-handled: without that, running it against the unfixed code kills the
jest worker with an unhandled rejection and reports no results at all.

* remove endpoint test

* remove comment

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: mobile reset-chat clears the whole workspace instead of the named thread (#6241)

* fix(mobile): reset only the requested thread, not the whole workspace

reset-chat resolves the thread with:

  await prisma.workspace_threads.findFirst({...})?.id

await binds looser than member access, so this is
await (findFirst({...})?.id). findFirst returns a Promise, .id on a
Promise is undefined, and await undefined is undefined. threadId is
therefore always undefined whenever threadSlug is supplied.

markThreadHistoryInvalidV2 passes the object straight to
prisma.workspace_chats.updateMany, and Prisma drops a where field whose
value is undefined. The thread_id filter disappears and every chat the
user has in that workspace is marked include: false.

Against a real sqlite database, resetting one thread also cleared the
other thread and the default thread.

Resolving the promise before reading .id is only half of it. A slug that
matches no thread must not fall through to null either, because null is
the default thread and clearing it is destructive. The handler now
resolves a supplied slug and returns 400 Thread not found when it does
not exist, leaving the absent-slug case as the only path to the default
thread. Every other error in this command handler is a 400, including
the Workspace not found beside it.

The sibling stream-chat command treats an unresolved slug as the default
thread, which is harmless there because it writes; doing the same here
would erase history the caller never named.

* unresolved threadSlug falls through to the default thread instead of 400

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: Only block explicit matches for built-in slash commands (#6243)

* fix(chats): do not match built-in slash commands inside longer commands

grepCommand matched built-in commands by bare prefix and returned early,
so any message that merely started with /reset or /img was dispatched as
that command. "/resetting my password, how?" reset the thread history,
and a preset whose name extends a built-in was shadowed and could never
run, since preset validation only rejects exact matches to a system
command.

Both spellings are creatable and both were swallowed. The preset modals
strip invalid characters, so typing "reset all" there saves /resetall,
while the server's formatCommand hyphenates and yields /reset-all, which
is also what you get by typing the hyphen or posting to the endpoint.

Reuse the boundary the preset loop already applies, (?![a-z0-9_-]), so a
built-in command does not match inside a longer command. Two near misses
this rejects: a plain \b passes between "t" and "-", so /reset-all is
still swallowed; a whitespace-only lookahead breaks /reset?, which
currently dispatches.

* block exact system command matches everywhere presets are created

---------

Co-authored-by: Timothy Carambat <[email protected]>

* 6246 pw restrictions (#6250)

* Prevent all astricks password due to masking
Fix input text to not show dummy mask on input

* add hold to reveal PW inputs
- login
- security pages
- account modal
- user pages for admin CRUD of users

* styling

* Pretty dates on tables (#6251)

* Implement LocalAI image editing via ref_images (#6222)

* Enhance editImage method for image editing support

Updated the editImage method to support image editing through the /v1/images/generations endpoint using ref_images. Added error handling for image fetching and improved logging.

* throw on ollama image gen with refs or on LocalAI non-ref support models

* remove bloated comment

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: score Lance chunks at or past orthogonal as 0, not a perfect match (#6253)

* fix(lance): score chunks at or past orthogonal as 0, not a perfect match

Both Lance search paths query with .distanceType("cosine"), so _distance
runs [0, 2]: 0 identical, 1 orthogonal, 2 opposite. distanceToSimilarity
returned 1 for anything at or above 1.0, the top of the scale, which makes
the mapping non-monotonic: 0.99 scores 0.01 while 1.5 scores 1.

#2986 already described this as the intended behaviour, saying "if distance
is greater than 1 then similarity should be 0", but its diff only changed
the distance <= 0 branch and left this one returning 1.

similarityThreshold is applied to this value in both paths, so an unrelated
chunk clears any threshold and reaches the model as context. In the standard
path the same value is also the citation score, so it is shown as a perfect
match. The reranked path normally reports rerank_score instead, so there the
effect is admission rather than the displayed score.

Flooring at 0 is the minimal monotone extension of the existing 1 - distance
mapping and leaves every score below orthogonal unchanged. Chroma and
PGVector have the same branch and are left for separate changes with their
own tests.

* fix scores everywhere

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Migrate threads on MuM migration (#6259)

* fix: stop citation cleanup mangling scraped code and numbered links (#6252)

* fix(collector): stop citation cleanup mangling scraped code and numbered links

htmlToMarkdown ran /\[[\d]+\]/g over the whole converted document to strip
citation markers. It never cleanly removed one. Turndown escapes brackets in
prose text to \[1\], which that pattern cannot match, so the two places it
did match were both places it should not have.

Inside code spans and fenced blocks Turndown leaves brackets literal, so
scraping a page documenting items[1] stored items, silently.

The other match is compactLinks, which emits an unescaped [text](href). A
link whose visible text is a number, a footnote marker or a pagination
control, lost its text and left the URL stranded:

  <p>See note<sup><a href="#fn1">1</a></sup> end.</p>
  before: See note(#fn1) end.
  after:  See note[1](#fn1) end.

Reference superscripts are removed by the DOM pass above, which drops
sup.reference and sup.noprint before conversion, so citation handling is
unaffected. The sibling \[#cite...\] and \[edit\] cleanups are left alone.

* test(collector): cover numeric alt text and bracketed link targets

The citation pattern also matched the unescaped brackets the converter emits
as syntax, not only the literal ones inside code:

  <p><img src="/a.png" alt="1"></p>   ->  !(/a.png)
  <p><a href="/docs/[1]/page">link</a></p> -> [link](/docs//page)

The second corrupts the URL itself. Both are fixed by the same deletion and
were previously unguarded.

* test(collector): full coverage of htmlToMarkdown module behavior

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: make modelPricing cache tests deterministic
test: stop aibitat emitter suite emitting the warning it explictly tests for

* fix: scope website-depth crawling by origin and path segment (#6258)

* fix(collector): scope website-depth crawling by origin and path segment

extractLinks decided whether a discovered link was in scope with a bare
string prefix:

  absoluteUrl.startsWith(origin + pathname.split("/").slice(0, -1).join("/"))

A string prefix respects neither origin nor path boundaries, and extractLinks
is the only scope filter, so links it admits can reach bulkScrapePages and be
fetched and stored.

Starting at https://example.com/docs/page, scope "https://example.com/docs"
also matched /docs-private/secret and /docs2.

Starting at a bare origin the scope is just the origin string, so
https://example.com also matched https://example.com.evil.net, and
http://127.0.0.1:5001 also matched http://127.0.0.1:50011.

Comparing the parsed origin, then whole path segments, fixes both. Origin
covers the scheme, so an http link from an https start is no longer in scope.
The scope directory itself stays in scope.

A bare origin and https://example.com/ both already produce an empty parent
path and stay site-wide. The parentPath === "/" normalisation is for
https://example.com//, whose parent path is "/"; without it that start URL
would be narrowed to almost nothing, where before it crawled the whole site.

#1800 introduced this parent-path filter, replacing an origin-only check.

* test(collector): name the double-slash case for what it is

The start URL is https://example.com//, not a root slash; https://example.com/
already yields an empty parent path and needs no normalisation.

* test: cover WebsiteDepth module-wide; skip malformed hrefs in extractLinks

Expands the PR's extractLinks scope tests with traversal, protocol-relative,
casing, and non-http scheme cases, and adds websiteScraper tests (mocked
Puppeteer) for depth, maxLinks, scope enforcement, and failure handling.

Also guards per-href URL parsing so one malformed href (e.g. href=http://)
no longer discards every other link on the page.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* patch DiD for metagenerator - resolves GHSA-rh3m-xv7m-9jhf

* docs: fix dead links in README (#6193)

* docs: fix dead links in README

* docs: sync locale READMEs with the English README

The zh-CN, ja-JP, tr-TR and fa-IR READMEs had drifted well behind
README.md. This brings all four back in line with the English source:

- Add the missing feature bullets (Dynamic Model Routing, Memories,
  Scheduled Tasks, Intelligent Skill Selection) and refresh the
  product overview / intro copy.
- Sync the LLM, embedder, transcription, TTS/STT and vector database
  lists so all four files carry the same 73 entries as README.md.
- Add Northflank to the self-hosting deploy table.
- Add the "Other outbound connections" telemetry section.
- Point Contributing at CONTRIBUTING.md and replace the retired
  VectorAdmin / Assistant Swarm entries with AnythingLLM Mobile,
  the browser extension and Embed.
- Fix relative paths that were broken from inside locales/ (./images,
  ./docker, ./cloud-deployments, ./LICENSE) and repair the language
  switchers, which linked ./locales/* from within locales/ and omitted
  Turkish and Persian.

Sponsors are not duplicated into the locales: the sponsors workflow
only writes README.md, so a copied list would immediately go stale.
Each locale links to the English section instead.

* docs: drop the Sponsors section from the locale READMEs

The sponsors workflow only writes README.md, so any sponsor content in
the locale files is never refreshed. Remove the section rather than
carry a pointer to a list these files do not maintain.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Add variable categories to agent flow variables (#6249)

* add required/optional/static variable categories to agent flows

* add tests for agent flow variable categories + legacy flow compatibility

* treat only undefined/empty-string required args as missing so falsy values like 0 pass validation

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Add generate-image agent skill (#6140)

* add generate-image agent skill + live image card in agent chat

* serve agent generated images from storage instead of inline b64

* add generate-image skill header image

* only skip cleared chats when serving agent generated files

* move image generation card payload into socket content object

* add image generation is-available endpoint + use for /img and agent skill checks

* small nits

* featch image on response

* translations

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: four server-side bugs (privatemode provider, prompt-history delete, success flag, event log) (#6165)

* fix: four server-side bugs (privatemode provider, prompt-history delete, success flag, event log)

1. getLLMProviderClass destructured 'PrivateModeLLM' but the module
   exports 'PrivatemodeLLM', so provider 'privatemode' always resolved
   to undefined (getLLMProvider in the same file uses the correct name),
   breaking context-window lookup for privatemode workspaces.

2. DELETE /workspace/prompt-history/:id used the validWorkspaceSlug
   middleware but the route has no :slug param, so the middleware looked
   up Workspace.get({slug: undefined}), resolved an arbitrary workspace,
   and the delete was scoped to that workspace's id — matching zero rows
   for any other workspace while still returning success: true (entry
   reappears on reload). Delete by id; admin/manager gate unchanged.

3. The catch block of POST suggested-messages returned 500 with
   success: true; every other handler reports success: false on error.

4. The 'document_embedded' event log read document.name, but
   workspace_documents rows have a filename column (no name), so
   documentName was always 'unknown'.

* improve IDOR validation for clearing prompt history

---------

Co-authored-by: Timothy Carambat <[email protected]>

* feat: support batch URL uploads (#6175)

* feat: support batch URL uploads

* full refactor for disambiguation and idempotent per-link processing, remove partial test

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: restrict workspace prompt-history read to admin/manager (#6137)

GET /workspace/:slug/prompt-history was guarded by ROLES.all, so any
default workspace member could read the full system-prompt history,
including previous prompt text and the username of the editor.

The DELETE routes on the same path already require admin/manager, and
the only frontend consumers live under WorkspaceSettings, which is a
manager-only route. Aligns the read guard with both.

Co-authored-by: Timothy Carambat <[email protected]>

* fix: match document search terms literally instead of as regular expressions (#6268)

* fix: match document search terms literally instead of as regular expressions

The content half of document search hands the user's term to ripgrep as a
pattern, so punctuation changes which documents match. Searching for
"price (2024)" returns nothing even when a document contains exactly that
text, because the parentheses become a capture group. Searching "[pricing]"
returns documents that do not contain the string at all, because the brackets
become a character class matching any of those letters.

An unbalanced term such as "a[b" makes ripgrep exit 2, which _rgRun maps to an
empty set, so a malformed pattern is indistinguishable from no results.

--fixed-strings matches the escaping the filename half already performs 18
lines above, and the function's own contract, which says it finds documents
whose content contains the term.

* remove test file

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: store DrupalWiki page attachments instead of skipping every one (#6266)

* fix: store DrupalWiki page attachments instead of skipping every one

sanitizeFileName strips path separators, so applying it to an already-resolved
absolute path removes every "/" in that path. A file destined for
/app/collector/hotdir/spec.pdf becomes appcollectorhotdirspec.pdf, which can
never satisfy the isWithin containment check on the next line, so the loop
skips it and no attachment is ever written.

Sanitizing the bare name and then resolving into the watch directory keeps the
containment guard doing what it was added to do. path.basename also neutralizes
a traversing name before it reaches resolve, so ../../etc/passwd.pdf lands in
the watch directory rather than escaping it.

The import still reports success while silently importing no attachments,
because the skip is a clean continue rather than an error.

* remove test file

* remove comment

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: separate PDF page boundaries instead of fusing the adjoining words (#6264)

* fix: separate PDF page boundaries instead of fusing the adjoining words

PDFLoader trims each page before returning it, so joining the pages on ""
leaves no boundary: the last word of one page and the first word of the next
become a single token. A body sentence running across a break is stored as
"grew to$4.2 million", and a page-number footer becomes "12Chapter 3".

The fused token cannot be found by a search for either word it came from, and
the citation text for that chunk reads wrong. "\n\n" also restores a preferred
split point, since it is the text splitter's highest-priority separator.

This matches the join PDFLoader already uses when it assembles pages itself.

* remove test file and redundant comment

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix(i18n): correct six Italian UI translations (#6317)

* fix(i18n): correct six Italian UI translations

* lint

---------

Co-authored-by: Timothy Carambat <[email protected]>

* feat(i18n): add Hindi (hi) language support (#6319)

feat(i18n):add Hindi (hi) language support

Co-authored-by: Timothy Carambat <[email protected]>

* fix: clear stale document_vectors rows during watched document sync (#6323)

* fix: clear stale document_vectors rows during watched document sync

Lance and PGVector never remove document_vectors rows in
deleteDocumentFromNamespace. The normal unembed path is unaffected
because Document.removeDocuments does its own deleteMany, but the
watched-document sync job calls the provider directly and then
re-adds the document, leaking one full set of rows per refresh.

Clear the rows for the docId in the sync job after the vector delete,
mirroring what Document.removeDocuments already does.

Supersedes #6302. Fixes #6301

* patch comment

* fix path resolution double slash on fastCRW (#6324)

fix path resolution double slash

* fix(collector): treat a blank published value as absent, not as the epoch (#6275)

* fix(collector): treat a blank published value as absent, not as the epoch

GET /v1/document/metadata-schema documents this key as
"epoch timestamp in ms | nullable", but Number(null), Number("") and
Number(" ") are all 0, so an explicitly null published stamps the document
1/1/1970 (12/31/1969 west of UTC).

That date is not only cosmetic. The value is plucked into chunkHeaderMeta and
stringifyHeader prepends it as literal <document_metadata> text on every chunk,
which all eight vector providers write, so the model reads the wrong date as
part of the document body.

Guard on a trimmed string so null, "", whitespace and [] are all treated as
absent. published: 0 is a real epoch value and still resolves to 1970.
undefined needs no clause: Number(undefined) is NaN, so it already fell to the
isNaN branch.

* fix(collector): harden processRawText input handling and add full test suite

Extends #6275 (null published -> epoch) to cover the remaining ways the
published normalizer could produce a wrong or garbage date, and adds
guards for the two inputs that made the module throw.

published:
- booleans are treated as absent (Number(false) is 0, Number(true) is 1)
- out-of-range values (1e20, "Infinity") fall back to now instead of
  rendering the literal string "Invalid Date" into chunk headers
- inline comments replaced with a JSDoc block describing the coverage

input validation:
- non-string textContent returns the existing empty-content failure
  instead of throwing on .split
- missing or non-string metadata.title returns a structured failure
  instead of throwing inside stripAndSlug; the collector's own
  /process-raw-text route is reachable without the server's title check

tests: processRawText suite grown from 9 to 106 cases covering input
validation, title slugging (incl. path traversal), url scheme filtering,
string metadata fallbacks, published, and the document/writer contract.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: keep chat-history attachments for TogetherAI, SambaNova, Z.ai and Cerebras (#6296)

* fix: keep chat-history attachments for TogetherAI, SambaNova, Z.ai and Cerebras

These four providers spread the raw chat history into the prompt, so an image attached on an earlier turn was never re-sent as a content part and the internal attachments field, base64 included, went out in the request body. Build the history through formatChatHistory with the provider's own content formatter, as the other providers do. For Cerebras the formatter is text-only, so the history stays plain text and only the stray field disappears.

Add a test that builds each of the four providers with fetch and the openai client stubbed, then checks that a history turn's attachments come out formatted like the current turn's and that no attachments field is left on the message.

* test: remove chatHistoryAttachments test

The test only exercised the reported happy path and is a tautology for the
text-only providers. Dropping it rather than shipping insufficient coverage.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: keep the GitLab repo scheme so http self-hosted instances can re-sync (#6294)

* fix: keep the GitLab repo scheme so http self-hosted instances can re-sync

resyncGitlab rebuilt the repo URL as "https:" plus the stored pathname, and
the GitLab chunkSource never recorded the scheme, so a watched document from
a self-hosted GitLab served over http was fetched over https on every sync
run and un-watched after five failures. Store the scheme in the chunkSource
payload and restore it on resync, defaulting to https for chunkSources
written before this change, which is the convention the GitHub and Gitea
connectors already use.

* test(collector): cover the rest of the GitLab loader module

Add tests for fetchIssues, fetchWiki, rate-limit retries, x-next-page
pagination, the real fetchGitlabFile, issueToMarkdown, and the token
validation catch. Export issueToMarkdown so it can be tested directly.
Drop the two inline comments added with the scheme fix.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: pin @qdrant/js-client-rest to 1.18.0 so lockfile-less builds keep search() (#6326)

* fix: pin @qdrant/js-client-rest to 1.18.0 so lockfile-less builds keep search()

The desktop bundler installs the merged server/collector deps without a
lockfile, so ^1.9.0 floated to 1.19.0, which removed client.search() and
broke every Qdrant retrieval with 'e.search is not a function' (#6270).
Docker and dev installs stayed on 1.9.0 via yarn.lock.

Pin to 1.18.0: the last release that still ships search(), supports Node
18, and talks to current Qdrant servers. 1.19.0 also raised the engine
floor to Node 22, which the Docker image does not meet.

1.18 added a fire-and-forget version probe on client construction; we
build a client per request, so disable it to avoid the extra GET and
the compatibility warnings against newer hosted servers.

* fix: keep the Qdrant client version warning

The compatibility probe only warns, and the warning is a useful signal
that the pinned client has drifted from the hosted server.

* fix: honor captureAs html on the scraper's fetch fallback (#6292)

* fix: honor captureAs html on the scraper's fetch fallback

getPageContent falls back to a plain fetch whenever the Puppeteer loader throws, and that branch always returned htmlToMarkdown() no matter what captureAs was. The Agent Flow Web Scraping block's Raw HTML and CSS Query Selector modes ask for html, so on the fallback they received Markdown and a selector matched nothing. Check the flag on the fallback the same way the Puppeteer branch does; text mode keeps returning Markdown.

* test: cover scrapeGenericUrl beyond the captureAs contract

Adds cases for the default captureAs, non-html values on the fallback,
custom header validation, both-methods-fail results, empty page bodies,
the saveAsDocument document shape, and delegation to processAsFile and
the YouTube transcript loader.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: preserve Unicode characters in imported skill names (#6286)

* fix: preserve Unicode characters in imported skill names

* test: move formatSkillName coverage into root Jest suite

Replace the ad-hoc node --test file and frontend test script with a Jest
test under frontend/__tests__, enable ESM test files via
--experimental-vm-modules in the root test script, and expand coverage to
mixed scripts, combining marks, separators, numbers, and non-string input.

* unsync

* remove test

* comment functionality

---------

Co-authored-by: Timothy Carambat <[email protected]>
Co-authored-by: Timothy Carambat <[email protected]>

* fix: require sql_query, not table_name (#6329)

fix: require sql_query, not table_name, in the sql-query agent tool schema

The sql-query tool's required list was copied from sql-get-table-schema and
still named table_name, a field the tool does not declare. Providers that
forward required into the tool schema (Anthropic, Bedrock) told the model to
supply a nonexistent argument while leaving sql_query optional.

Fixes #6328

* fix: update workspace chat lastUpdatedAt (#6288)

* fix: update workspace chat lastUpdatedAt

* chore: drop mock-only lastUpdatedAt test from workspace chat fix

The test only re-asserted the spread in WorkspaceChats._update through a
stubbed prisma client, so it pinned the implementation rather than the
behavior and would break on a legitimate refactor (e.g. moving the column
to @updatedAt).

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix(collector): keep upload-link metadata when the link resolves to a file (#6277)

* fix(collector): keep upload-link metadata when the link resolves to a file

POST /v1/document/upload-link accepts a metadata object (title, docAuthor,
description, docSource) and the collector forwards it into scrapeGenericUrl.
When the link's Content-Type is a file type such as application/pdf the
request is handed to processAsFile, which called processSingleFile without
the metadata. The converter then fell back to the hotdir filename for the
title and to the PDF's own info dictionary for the author and description,
and nothing was logged about the dropped values.

Thread metadata through processAsFile into processSingleFile, which already
accepts it as its third argument.

* chore: drop processAsFile forwarding test

The test only pinned the argument shape against a mocked processSingleFile
and covered a single helper in an otherwise untested module. The fix is a
straight parameter pass-through; merge it without the test.

* test: expect metadata to be forwarded to processAsFile in generic.test.js

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix(generic-openai): honor max tokens in Tooled/UnTooled agent streams (#6331)

* fix(generic-openai): honor max tokens in UnTooled agent streams

Signed-off-by: Jkand Huang <[email protected]>

* chore: drop mock-heavy regression test for max_tokens passthrough

* fix(generic-openai): send max_tokens on native tool-calling agent requests too

Add an opt-in maxTokens option to tooledStream/tooledComplete and pass it
only from the Generic OpenAI agent provider. No other provider passes the
option, so their requests are byte-for-byte unchanged.

---------

Signed-off-by: Jkand Huang <[email protected]>
Co-authored-by: Jkand Huang <[email protected]>
Co-authored-by: Timothy Carambat <[email protected]>

* Make web search a default agent skill with You.com as the default provider (#6280)

* make web search a default agent skill + default to You.com SERP with DuckDuckGo fallback

* fall back to DDG only on You.com error

* harden You.com fallback and add web search skill tests

- log a 401/403 with a configured AGENT_YOU_API_KEY as a rejected key so
  a typo'd key is distinguishable from quota exhaustion (402) in the logs
- guard results.web/news with Array.isArray so an unexpected shape cannot
  throw past the fetch catch and skip the DuckDuckGo fallback
- correct the _youSearch doc comment: only request failures fall back,
  an empty successful response is returned as no results
- add tests for _youSearch fallback/shaping and for web-browsing being
  resolved as a default skill

* keep web-browsing a configurable skill, You.com is only the default engine

Drop the promotion of web-browsing into DEFAULT_SKILLS for now. It stays
an opt-in configurable skill exactly as on master; the change in this PR
is that when it is enabled with no search provider configured, You.com's
keyless tier is used with an automatic DuckDuckGo fallback.

* remove web-browsing plugin test

* revert defaults test additions

* web enrich with additional payload items

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Enable web-browsing by default (#6335)

* cleanup copy for default skills

* Add Keenable web search provider (#6336)

* feat(agent): add Keenable web search provider (keyless by default)

Keenable is a web search API built for AI agents. Unlike the other providers,
it works without an API key by default (keyless public endpoint); setting
AGENT_KEENABLE_API_KEY uses the authenticated endpoint and lifts rate limits.

- web-browsing.js: _keenableSearch provider + switch case
- updateENV.js / systemSettings.js: AGENT_KEENABLE_API_KEY + AGENT_KEENABLE_API_URL
- agent_search_provider validation allowlist
- frontend: provider entry + options form (optional key, base URL) + icon
- .env.example (server + docker)

* fix: read Keenable result text from snippet, not description

Keenable returns both fields on every result: snippet carries the page text
and description is the page's meta description, which is empty for most
pages, so the agent received citations with an empty snippet. Keenable also
returns whole pages, so the text is collapsed and capped at 500 chars.

* Use the real Keenable mark for the provider icon

The file shipped a generic blue magnifying glass rather than our brandmark.
This is the mark from our brand guidelines, transparent so it reads on both
the light and the dark theme.

* fix logo

* path adj

* remove promotional comments

* fix URL validation on keenable

* update order and small copy edit

---------

Co-authored-by: Ilya Bogin <[email protected]>

* fix warns

* Add `TOOL_CALL_APPROVAL_TIMEOUT_MS` to admin env config (#6339)

add `TOOL_CALL_APPROVAL_TIMEOUT_MS` to admin env config

* Add Bulgarian (bg) translation (#6344)

* Add Bulgarian (bg) translation

* reorder and lint

---------

Co-authored-by: Timothy Carambat <[email protected]>

* docs: add Easypanel to self-hosting deploy table (#6360)

* docs: add Easypanel to self-hosting deploy table

* docs: use official Easypanel deploy button badge

* patch GHSA-8wh8-3v68-whwj (#6361)

* fix: accept the documented allowlist_domains array in the embed API (#6358)

* fix: accept the documented allowlist_domains array in the embed API

POST /v1/embed/new and POST /v1/embed/:embedUuid document
allowlist_domains as an array of domains, but validatedCreationData
only handled the comma-separated string the settings form sends. An
array threw on .split inside the try block and the catch returned null,
so a create answered success with no allowlist and an update overwrote
an allowlist an admin had already set. Split a string on commas, take
an array as the list, and return null for any other shape as before.
The scheme prefixing and the URL validation of each entry are
unchanged.

* nit

* fix bad ignore rule

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: drop a GitLab access token the instance rejects instead of reusing it (#6356)

The token validator sent GET /api/v4/user and threw away the response, so
only a network exception reached the catch that clears the token. A token
the instance rejected with 401 stayed set and was sent on every later
request, and GitLab answers 401 for those too, even on a public project.

Non-OK validation responses now reach the same catch, so the token is
logged as invalid and dropped, and public repositories keep importing
anonymously. The test module gains the HTTP 401 case next to the existing
thrown-request case.

Co-authored-by: Timothy Carambat <[email protected]>

* embed and model router tests

* fix: set Simple SSO temporary auth token expiry to 10 minutes (#6362)

* fix: set Simple SSO temporary auth token expiry to 10 minutes

The constant was 6 minutes under a comment claiming 1 hour. Move to a
10 minute window and drop the hardcoded duration from the comment so
the value can change without the comment going stale.

Refs #6353

* nit

* fix: paperless-ngx resync returned empty for every existing document (#6348)

* fix: paperless-ngx resync returned empty for every existing document

resyncPaperlessNgx destructured PaperlessNgxLoader as a named export
although the loader module is a CommonJS default export, so the
constructor was always undefined and every resync request failed into
success:false. It also parsed the document id with
pathname.split('//')[1] although the chunkSource paperless-ngx://<id>
URL carries the id in the host slot, leaving the id undefined.

- import the loader with its default-export contract
- read the document id from source.host
- preserve context-path base URLs on collect via resolvePaperlessBaseUrl
  so self-hosted /paperless instances round-trip correctly
- regression tests for the resync roundtrip and base-url normalization

* refactor: normalize Paperless-ngx base URL in the loader constructor

Move the context-path-preserving URL normalization into the
PaperlessNgxLoader constructor so every caller gets it, and drop the
separate resolvePaperlessBaseUrl helper. Remove the loader test file,
which destructured the default export and failed on the branch, and
keep the resync regression test without the incorrect virtual mocks.

* test: cover Paperless-ngx resync round trip and failure paths

Use a real EncryptionWorker and the real loader with only fetch stubbed
so the collect/resync chunkSource contract is exercised end to end.
Adds cases for context paths, missing chunkSource, undecryptable
payloads, 404s, network failures, and an empty document id.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* Add `deepseek-flash` to IncludeReasoning array due to deprecation/upgrade of alias (#6379)

add `deepseek-flash` to includereasoning array due to deprecation

* fix(mistral): batch embedding requests and declare the engine's limits (#6378)

* fix(mistral): batch embedding requests and declare the engine's limits

Every other embedder declares maxConcurrentChunks and embeddingMaxChunkLength
and splits a document with toChunks before calling the provider. The Mistral
engine declared neither and sent every chunk of a document in one POST, so a
document large enough to exceed Mistral's per-request input count fails as a
whole, and TextSplitter.determineMaxChunkSize fell back to its own 1000 rather
than reading a limit off the engine, which silently clamps a configured chunk
size with a warning about a limit the engine never stated.

Mistral serves an OpenAI-compatible embeddings endpoint through the same SDK,
so the batch size follows the value the OpenAI and Generic OpenAI embedders
use, and the chunk length comes from maximumChunkLength() the way it does for
every other OpenAI-compatible engine here, which keeps today's effective 1000
while making it settable through EMBEDDING_MODEL_MAX_CHUNK_LENGTH.

The failure contract from #5513 is unchanged and now covered by tests: a failed
batch still raises Mistral Failed to embed, an empty response still raises, and
a partial document is never returned. Embedding progress is reported per batch,
as the other engines do.

* refactor(mistral): align embedder error handling with OpenAI footprint

- enrich caught errors with type/message from the HTTP response and
  aggregate as [type]: message, matching the OpenAI and LocalAI embedders
- drop the redundant console.error before the throw
- annotate the intentional #5513 throw-on-empty divergence
- remove the dedicated mistral embedder test suite

* improve error handling and true concurrency cap

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: stop section and cite cleanup mangling scraped code and links (#6371)

* fix: stop section and cite cleanup mangling scraped code and links

`htmlToMarkdown` ran two more whole-document replacements after conversion:

```js
markdown = markdown.replace(/\[#cite[^\]]*\]/g, "");
markdown = markdown.replace(/\[edit\]/gi, "");
```

Both have the same shape as the numeric citation pattern removed in #6252, and
both corrupt content the same way.

Turndown leaves brackets literal inside code, so `[edit]` in a snippet was
deleted. `config[edit] = true;` stored as `config = true;`. The pattern is case
insensitive, so `[EDIT]` went too. `compactLinks` emits an unescaped
`[text](href)`, so an anchor labelled `edit` lost its label and stranded the
URL, and a bracket sequence inside a link target was eaten from the destination.

The cite pattern reaches one case the numeric one could not. `[^\]]*` spans the
backslash Turndown puts before an escaped closing bracket, so it matched across
escaped prose:

    See [#cite_note-1] for details.
      before: See \ for details.
      after:  See \[#cite\_note-1\] for details.

An anchor whose visible text is a cite marker lost its text the same way a
numbered one did:

    <a href="#cite_note-1">#cite_note-1</a>
      before: (#cite_note-1)
      after:  [#cite\_note-1](#cite_note-1)

The link damage is not hypothetical. Scraping
https://www.mediawiki.org/wiki/Help:Section produces 30 section-edit links, and
on every one of them the old cleanup removed the label and left the bare
destination behind:

    before: \[(https://www.mediawiki.org/w/index.php?title=Help:Section&action=edit&section=1)\]
    after:  \[[edit](https://www.mediawiki.org/w/index.php?title=Help:Section&action=edit&section=1)\]

Those controls sit in `.mw-editsection` spans, which are not in the DOM removal
list, so they survive that pass either way. The replacement was not removing the
control, it was breaking the link and keeping the URL.

The selector-based reference cleanup is unchanged. `stripCitations` still drops
`sup.reference`, `sup.noprint`, `.reflist`, `.references`, `.refbegin`,
`#References`, `.catlinks` and `.mw-authority-control` from the DOM before
conversion, and its existing tests still pass. What changes is that markers and
section-edit links which that pass does not classify now stay as literal text or
working links instead of being damaged by a replacement that cannot tell code,
prose and link syntax apart.

Eight cases added to the existing helper suite: the marker in fenced and inline
code, uppercase, an anchor labelled with it, a link target containing it, the
cite marker in code, a cite anchor, and escaped cite prose. The suite is 35
tests, all passing.

Against current code 8 fail. Removing only the `[edit]` line leaves 3 failing,
removing only the cite line leaves 5, and keeping both while making `[edit]`
case sensitive leaves 7, so the cases reject the partial fixes rather than
accepting any of them.

The full test suite passed with a local compatibility setup that defines the
missing `buffer.SlowBuffer` alias on this machine. The setup was used for both
the base and the branch and is not part of this change. On that setup the base
is 61 suites and 1000 tests, and this branch is 61 suites and 1008 tests.

* fix: remove MediaWiki section-edit controls in the DOM pass

The deleted [edit] text replacement was meant to drop the per-heading edit
control on MediaWiki pages, but Turndown had already turned it into
\[[edit](url)\] by the time it ran, so it broke the link instead of
removing it. Strip the .mw-editsection element before conversion alongside
the other reference chrome, where it cannot touch code, prose, or links.

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: keep the Confluence page scheme so http self-hosted instances re-sync (#6372)

* fix: keep the Confluence page scheme so http self-hosted instances re-sync

A watched Confluence page imported from a self-hosted instance served over
plain http never re-syncs. `resyncConfluence` rebuilt the target page URL as
`https:` plus the stored pathname:

```js
pageUrl: `https:${source.pathname}`, // need to add back the real protocol
```

The request itself is fine, because it goes through the stored `baseUrl`, which
keeps its scheme. Only the target selection breaks. The loader builds each
`doc.metadata.url` from that same `baseUrl`, so on an http instance the pages
load over http while the page being looked for says https, and the exact
comparison in `fetchConfluencePage` fails:

```js
const targetDocument = docs.find(
  (doc) => doc.pageContent && doc.metadata.url === pageUrl
);
```

The sync reports no content every run. A failed refresh removes the watch when
the five most recent recorded runs are failures, so the page first stops
updating and then stops being watched at all.

The scheme is already in the payload. `generateChunkSource` encrypts `baseUrl`
alongside the credentials, and `expandPayload` hands it back as a query param,
so the protocol can be read from it and put back:

```js
pageUrl: `${protocolOf(baseUrl)}${source.pathname}`,
```

This repairs payloads that already exist rather than only ones written from now
on, and it leaves the comparison alone, so two hosts or two ports are still
different targets. A payload whose `baseUrl` is missing or unparseable keeps
resolving to https, which is what every payload resolved to before.

This is the Confluence half of the same scheme loss fixed for GitLab in #6294.
That change added a `scheme` field to the GitLab payload and defaults older
payloads to https. This one reads the `baseUrl` Confluence already stores, so
the http payloads that exist today need no re-import.

Six tests in a new `collector/__tests__/extensions/resync/confluence.resync.test.js`,
following the layout of the paperless-ngx suite next to it. They run a real
`EncryptionWorker` and the real resync handler, stubbing the Confluence HTTP
responses rather than `fetchConfluencePage`, so the URL round trip is what is
under test.

Against current code two fail. Hardcoding `http:` instead fails the cloud case.
Comparing the pathname on both sides instead of the whole URL fails two, a page
whose path matches on another host and one whose path matches on another port,
each answered with a real page rather than an empty space so the comparison is
actually reached.

The full test suite passed with a local compatibility setup that defines the
missing `buffer.SlowBuffer` alias on this machine. The setup was used for both
the base and the branch and is not part of this change. On that setup the base
is 61 suites and 1000 tests, and this branch is 62 suites and 1006 tests.

* Tidy protocolOf comments and remove Confluence resync test

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: linked mbox scraping returned only the first message (#6374)

* fix: linked mbox scraping returned only the first message

`processAsFile` in the collector asks `processSingleFile` to parse a downloaded
file and, when the caller wants text rather than a stored document, returns
`documents[0].pageContent`. A file that parses into more than one document loses
everything after the first.

MBOX is one of those files. `collector/processSingleFile/convert/asMbox.js`
builds a separate document for each mail it accepts, so a mailbox parses into
one document per accepted message and the caller receives one of them.

The path is live for agents. `CollectorApi.getLinkContent` posts to the
collector's `/util/get-link`, which runs `getLinkText` with
`saveAsDocument: false`. `scrapeGenericUrl` routes `application/mbox` to
`processAsFile` because that mime is in `ACCEPTED_MIMES`. Agent Flow web
scraping and the aibitat web-scraping plugin both reach it that way, so an agent
asked to read a linked mailbox reasons over the first message and never sees the
rest.

This returns the text of every parsed document, joined with a blank line, and
skips documents that carry no text. A blank line separates adjacent message
bodies without inventing headers or metadata for them. It is a readable
separator rather than a parseable one: a message that contains a blank line of
its own is not distinguishable from a message boundary, and this change does not
try to make it so.

The document-saving path is untouched. It still returns the parser's result
unchanged, so per-message documents and the records built from them keep their
current shape.

One behavior change is worth calling out. A mailbox whose mails all lack usable
text parses successfully into zero documents, and reading `documents[0]` threw
on that. The endpoint's own catch turned the exception into a failure response.
It now returns a successful empty text result instead, because the parse did
succeed and there was simply nothing in it. Both agent callers already reject
empty content and report their own error, so neither of them changes behavior. A
mailbox that cannot be parsed at all still returns the converter's failure.

The new suite at `collector/__tests__/processLink/helpers/index.test.js` covers
the whole helpers module: content-type parsing, content-type discovery over
HTTP, the routing outcomes of `determineContentType` for html, plain text, an
accepted file mime, an unsupported mime and an unreadable one, both result
shapes, download and parse failures, metadata forwarding, and the
document-saving path returning the parser's own result.

The aggregation cases pin the separator once on its own, then walk the joined
result for five document sets and require each body verbatim, in order, with
exactly one separator between neighbours and nothing left over. Against current
code 8 of the 29 fail. Joining with no separator or a single newline fails 7
each. Dropping the empty-text filter, dropping the guard for a document with no
`pageContent`, sorting the bodies, de-duplicating them, and trimming each one
all fail 1 each, so the cases reject those implementations rather than accepting
them.

The full test suite passed with a local compatibility setup that defines the
missing `buffer.SlowBuffer` alias on this machine. The setup was used for both
the base and the branch and is not part of this change. On that setup the base
is 61 suites and 1000 tests, and this branch is 62 suites and 1029 tests.

* remove test, update comment

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: decode GitHub file content as UTF-8 when re-syncing a watched document (#6373)

* fix: decode GitHub file content as UTF-8 when re-syncing a watched document

`RepoLoader.fetchSingleFile` decoded the base64 the GitHub contents API returns
with `atob`, which produces one code unit per byte. Every character outside
ASCII came back as its individual bytes, so `café 東京 😀` refreshed into
mojibake.

Nothing downstream repairs it. `fetchGithubFile` returns the string as
`content`, the re-sync handler forwards it, and the watched document worker
compares it with the stored text and then sends it to embedding and to the
source document update. So a watched file holding any non-ASCII text refreshes
into mojibake and the sync reports success, leaving the corrupted text in place
of the file's real contents.

The initial import does not have this problem, because it goes through the
`@langchain/community` GitHub loader, which reads the file with `response.text()`
and so decodes UTF-8. This aligns re-sync with that:

```js
return new TextDecoder("utf-8").decode(Buffer.from(json.content, "base64"));
```

`TextDecoder` rather than `Buffer.toString("utf8")` because it also drops a
leading byte order mark, which is what `response.text()` does on import. Using
`toString("utf8")` instead would leave a BOM at the start of the re-synced copy
that the imported copy never had.

This is a text decoding fix and does not add binary or alternate-encoding
support. It changes the former byte-to-code-unit output: bytes that are not
valid UTF-8 now become replacement characters, so a non-UTF-8 file that `atob`
rendered as readable Latin-1 no longer reads that way. The initial import
already turns those same files into UTF-8 text through `response.text()`, so
this is the same treatment the file received when it was first imported, not a
new one.

Four cases added to the existing loader suite: accented, CJK and astral
characters, a multi-byte character positioned to straddle a base64 line break, a
leading byte order mark, and plain ASCII. The suite's contents mock now wraps
its base64 at 60 characters the way the contents API does, so the straddle case
has a real line boundary to cross.

Against current code three fail. `toString("latin1")` also fails three,
`toString("utf8")` fails the byte order mark case, and decoding each wrapped
base64 line on its own fails the straddle case, so the cases separate the
spellings rather than accepting any of them.

The full test suite passed with a local compatibility setup that defines the
missing `buffer.SlowBuffer` alias on this machine. The setup was used for both
the base and the branch and is not part of this change. On that setup the base
is 61 suites and 1000 tests, and this branch is 61 suites and 1004 tests.

* chore: trim comments in GitHub resync UTF-8 fix and tests

---------

Co-authored-by: Timothy Carambat <[email protected]>

* fix: llmman model discovery ignored the token entered in settings (#6376)

* fix: llmman model discovery ignored the token entered in settings

The llmman settings form sends the unsaved token entered in the form into model
discovery, after the field loses focus or when auto-detect is used. The saved
token is never sent back to the browser, so this draft is the only credential
that request has. `LlmmanOptions` calls `System.customModels("llmman",
authToken, basePath)`, `frontend/src/models/system.js` serializes that argument
as `apiKey`, and `POST /system/custom-models` forwards it to `getCustomModels`.
The dispatcher then called `llmmanModels(basePath)` and dropped it.

`llmmanModels` takes the token as its second argument and turns it into the
`Authorization: Bearer` header. Without it the request to `/api/tags` carries
whatever token is already in `LLMMAN_AUTH_TOKEN`, or no header at all. If the
endpoint rejects that missing or stale credential, discovery returns an empty
list and the selector stays disabled with its loading label, so the user sees no
way forward even though the token they entered is correct.

Two cases reach it. Setting up a protected server for the first time, where
nothing is stored yet and the request goes out with no header at all. And
replacing the token on a server that is already configured, where the stored
token is sent instead of the new one. A stored token that still works hides the
problem entirely.

The provider of the same shape already passes both arguments. `llmmanModels` and
`ollamaAIModels` are near identical: same base path validation, same
caller-then-environment token precedence, same `/api/tags` request, same write
back to the environment on success. llmman serves the Ollama API and its
provider is built on the `ollama` client (#6226). The `ollama` arm dispatches as
`ollamaAIModels(basePath, apiKey)`. This makes the llmman arm match.

One thing this does not change. A successful discovery writes the token it used
into `process.env` for the rest of the process, which both providers already
did. That is a process-local cache, not a saved setting. The form still has to
be saved for the token to persist.

The new suite at `server/__tests__/utils/helpers/customModels.test.js` covers
the dispatcher's public guard for an unsupported provider and the two discovery
paths that read a base path and a token, `ollama` and `llmman`. It is not
coverage of the other 49 switch arms, which each construct a provider client.
For both providers it runs base path from the argument and from the environment,
a base path that is not a URL and one that ends in a slash, no header when there
is no token anywhere, the caller's token, the caller's token winning over a
stored one, the stored token when the caller sends none, caching after a
nonempty response, leaving a stored token alone when the endpoint rejects the
new one and when the response lists nothing, and the two empty-response shapes.

Against current code 4 of the 31 fail, all of them llmman rows whose ollama
twins pass. Reading the environment first instead fails 2. Passing the arguments
in the other order fails 10. Caching the token without checking that the
response returned models fails the two controls written for it.

The full test suite passed with a local compatibility setup that defines the
missing `buffer.SlowBuffer` alias on this machine. The setup was used for both
the base and the branch and is not part of this change. On that setup the base
is 61 suites and 1000 tests, and this branch is 62 suites and 1031 tests.

* remove useless test

---------

Co-authored-by: Timothy Carambat <[email protected]>

* KoboldCPP overhaul (#6380)

* fix(koboldcpp): stop a failed stream from hanging the chat or killing the server

`KoboldCPPLLM.handleStream` is one of two provider stream handlers written by
hand rather than delegating to `handleDefaultStreamResponseV2`, and it is the
only one with no `try`/`catch` around its `for await` loop. Three consequences,
all measured:

- A stream that fails mid-flight leaves the promise unsettled. The caller awaits
  it forever, the HTTP response is never closed, and the browser spins. Every
  other provider reports `[STREAMING ERROR]`, writes an `abort` chunk and
  resolves with the text generated so far.
- `message.finish_reason` is read without optional chaining on
  `chunk?.choices?.[0]`, so a chunk carrying usage and no `choices` -- what an
  OpenAI-compatible server sends when usage reporting is on -- throws. The loop
  also never `break`s after resolving, so that chunk is reached even on a normal
  finish: the chat completes, then the throw lands in the executor of
  `new Promise(async ...)` as an unhandled rejection. Node has defaulted to
  `--unhandled-rejections=throw` since v15 and the server registers no
  `unhandledRejection` handler, so that ends the process.
- A stream that ends without a `finish_reason` of `length` or `stop` falls out
  of the loop and never resolves either.

Wrap the loop the way `openAi`'s handler does: `isAbortError` goes to
`clientAbortedHandler`, anything else is reported and resolved with the partial
text; `break` after the finish chunk; optional-chain `message`; resolve after
the loop.

Separately, `usage.prompt_tokens` was seeded from
`countTokens(stream.messages || [])`, and `stream.messages` is set nowhere --
`LLMPerformanceMonitor.measureStream` sets `start`, `duration`, `metrics` and
`endMeasurement`, and koboldCPP is the only reader of `stream.messages` in the
repo. `endMeasurement` merges the reported usage over `stream.metrics`, so that
constant replaced the prompt count the monitor had already measured: 27 tokens
of prompt were reported as 5 for every KoboldCPP chat. Reporting no
`prompt_tokens` keeps the measured one.

* koboldcpp: consolidate stream measurement, drop verbose comments and test

* remove co…