Skip to content

feat(docreader): add XMind outline parsing - #2713

Merged
lyingbug merged 5 commits into
Tencent:mainfrom
guiyangyuan:agent/xmind-docreader-support
Aug 20, 2026
Merged

lyingbug merged 5 commits into
Tencent:mainfrom
guiyangyuan:agent/xmind-docreader-support

Conversation

@guiyangyuan

@guiyangyuan guiyangyuan commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

Add native .xmind support to the built-in DocReader so mind maps can be uploaded and indexed without a manual Markdown or image export.

  • Parse modern XMind archives from content.json and classic XMind 8 archives from content.xml using only the Python standard library.
  • Preserve sheet and topic hierarchy as Markdown outlines, including plain-text node notes.
  • Prefer modern JSON when both formats are present and reject malformed, encrypted, empty, or oversized content entries with clear errors.
  • Advertise xmind through both the Python parser registry and Go engine registry so existing clients can discover support.
  • Add parser, validation, routing, and engine-registry regression tests.

User impact: teams can directly add structured XMind knowledge to WeKnora for RAG, search, and summarization workflows.

Breaking changes: none.

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation update
  • 🎨 Refactor
  • ⚡ Performance improvement
  • 🧪 Test
  • 🔧 Configuration / Build / CI

Related Issue

Closes #2705

Testing

uv run --project docreader --no-sync python -m unittest discover -s docreader/tests -p "test_*.py" -v
uv run --project docreader --no-sync python -m compileall -q docreader
go test ./internal/infrastructure/docparser -count=1

Results: the 158-test Python suite completed successfully with 12 environment-dependent fixture tests skipped; Python compilation and Go document-parser tests passed.

Checklist

  • git diff --check upstream/main...HEAD passes
  • Changed source files are formatted
  • Targeted tests for the changed packages/components pass
  • Diff-scoped lint passes where applicable (for Go: golangci-lint run --new-from-rev=origin/main ./...)
  • Full DocReader Python tests and scoped Go tests were run
  • Self-reviewed the code
  • Added/updated tests covering the change
  • Updated related documentation (README, docs/, Swagger annotations, etc.)
  • Breaking changes are clearly called out in the description above

Screenshots / Recordings

Not applicable; this change adds backend document parsing and engine discovery without UI changes.

Out of Scope

Images, labels, markers, priorities, relationships, and graphical rendering are intentionally left for follow-up work.

@guiyangyuan
guiyangyuan marked this pull request as ready for review August 18, 2026 03:02
Copilot AI lite review requested due to automatic review settings August 18, 2026 03:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds native .xmind parsing support to WeKnora’s built-in DocReader pipeline, enabling teams to upload XMind mind maps directly for indexing and downstream RAG/search workflows.

Changes:

  • Added a new Python XMindParser that reads XMind ZIP archives and renders sheet/topic hierarchies (plus plain-text notes) as Markdown outlines.
  • Registered xmind in both the Python parser registry and the Go engine’s advertised file types, plus added routing/registry regression tests.
  • Added comprehensive unit tests covering modern (content.json) vs classic (content.xml) formats and validation/error conditions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
internal/infrastructure/docparser/engines.go Advertises xmind as a supported builtin-engine file type.
internal/infrastructure/docparser/engine_registry_test.go Updates engine-registry test expectations to include xmind.
docreader/parser/xmind_parser.py Implements XMind archive parsing + Markdown outline rendering (new parser).
docreader/parser/registry.py Registers xmind → XMindParser in the builtin Python registry.
docreader/tests/test_xmind_parser.py Adds unit tests for JSON/XML parsing, rendering, and validation behaviors.
docreader/tests/test_parser_routing.py Adds a routing test ensuring xmind maps to XMindParser in the builtin registry.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docreader/parser/xmind_parser.py Outdated
Comment on lines +141 to +150
if "content.json" in archive.namelist():
content_format = "json"
info = archive.getinfo("content.json")
elif "content.xml" in archive.namelist():
content_format = "xml"
info = archive.getinfo("content.xml")
else:
raise ValueError(
"XMind archive is missing content.json or content.xml"
)
@lyingbug
lyingbug merged commit f51a139 into Tencent:main Aug 20, 2026
7 checks passed
hcy0317 added a commit to hcy0317/WeKnora that referenced this pull request Aug 30, 2026
* fix(hybrid-search): guard MatchCount=0 truncating all results

When HybridSearch is called from the HTTP endpoint without a
match_count field, params.MatchCount defaults to Go's zero value.
The guard

  if len(deduplicatedChunks) > params.MatchCount { … [:0] }

then silently truncated every result list to empty, so processSearchResults
was called with 0 chunks and returned nil — 64 RRF-fused results → 0.

Fix: treat MatchCount ≤ 0 as "no explicit limit" and fall back to the
same floor (50) used by the over-retrieval phase.  Callers that set
MatchCount explicitly (chat pipeline, knowledge.go, agent tools) are
unaffected.

Verified: vector=50 + keyword=50 → RRF=64 → processSearchResults(50) → 108
final results (50 primary + 61 enrichment neighbours/parents) for
"M1000 MP frequency bands LTE cellular" query.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(build): Enable Docker BuildKit in build_images.sh

* fix(chunk): sanitize context headers before persistence

* docs(cli): align retry guidance with retry_argv

* docs: update website screenshot tracker

* Fix/hybrid search match count normalization (#2750)

* fix(hybrid-search): normalize MatchCount once instead of at the truncation site

An omitted match_count deserializes to Go's zero value, which reached the
truncation step in HybridSearch and sliced the deduplicated chunk list to
[:0] — a successful retrieval returned an empty result set. A negative
value panicked on that same slice bound and surfaced as a 500.

Resolve the value once at the top of HybridSearch via normalizedMatchCount
so all three consumers agree, rather than patching the truncation site
alone: the over-retrieval floor, the FAQ iterative-retrieval trigger
(which never fired on this path because len(chunks) < 0 is false), and the
truncation itself.

Bound the FAQ iterative path while here, since normalization is what makes
it reachable with a large MatchCount: its TopK seeds from MatchCount*3 and
doubles five times, previously without limit. It now stops at the pool cap
instead of re-issuing an identical query.

Co-authored-by: Cursor <[email protected]>

* refactor(retrieval): share the default retrieval depth as one constant

The value 50 was written three times — the over-retrieval floor in
HybridSearch, the MatchCount fallback added alongside it, and
RetrievalConfig.GetEffectiveEmbeddingTopK — with nothing recording that the
three are meant to agree. Anyone tuning one would silently desync the
others.

Promote it to types.DefaultRetrievalTopK. No behavior change.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: drakezhang <[email protected]>
Co-authored-by: Cursor <[email protected]>

* fix(i18n): add missing tenantless copy

* fix: localize builtin agent name/description from YAML when reading DB rows

Co-authored-by: Cursor <[email protected]>

* fix(retrieval): 分段中删除的图片不再被检索到

从分段里删掉一张图片后,这张图仍会回到检索结果和模型上下文里。

三个问题串在一起:

1. 删图时 syncEditedChunkImages 会把 image_ocr / image_caption 子块置为
   禁用,但 CollectImageInfoByChunkIDs 查子块时不看 is_enabled,又把它们的
   image_info 挂回了父块。knowledge_search 工具随后把每条 image_info 无条件
   渲染成 markdown 图片写进 LLM 上下文,图片就这样回来了。这里按 is_enabled
   过滤,两级遍历都跳过禁用子块。过滤放在 collector 而不是工具输出层:纯图片
   分块的正文本身就是 OCR 文本、没有 markdown 图片语法,按正文过滤会误杀。
   chat_pipeline 里既有的 fake 仓储 fixture 同步补上 IsEnabled: true:它们不
   走 DB 默认值,零值 false 会被新的过滤当成已禁用子块。

2. UpdateImageInfo 补建 image_ocr / image_caption 子块时没有设置 IsEnabled。
   CreateChunks 用 Select("*") 插入,gorm 的 default:true 不生效,零值 false
   直接落库,这些子块一出生就是禁用的。补上 IsEnabled: true —— 否则第 1 点
   的过滤会把它们永久丢弃。其余 CreateChunks 调用方均已显式设置该字段。

3. 摘要刷新每次都新建一条 summary 分块,旧的留着且仍然启用。原因是它复用了
   ListChunksByKnowledgeID 的结果,而那个查询硬编码 chunk_type = 'text',
   "查找已有 summary 分块"的分支永远匹配不上。残留的旧摘要描述的是编辑前的
   文档,会让用户已经删掉的内容继续被检索到。新增按 chunk_type 过滤的仓储
   方法,刷新时原地更新已有 summary 分块。

* fix: navigate chat references to document folder

* fix(migrations): 补齐 Lite 模式缺失的 SQLite 版本化迁移

migrations/sqlite 落后于 migrations/versioned,导致 Lite 模式全新数据库
首次建号/建空间失败(tenants.api_principal_config、system_settings、
task_pending_ops 等缺失)。新增 000005-000011 七个 SQLite 迁移,
按 versioned 迁移主题一一补齐:

- 000005  <- 000034/000054: messages.attachments、tenant_invitations token/accepted_count
- 000006  <- 000041: task_pending_ops / task_dead_letters(与 #2433 同一 DDL)
- 000007  <- 000053: system_settings、users.is_system_admin
- 000008  <- 000055/000056: knowledge_processing_spans、knowledges.pending_subtasks_count
- 000009  <- 000060: embed_channels.allow_memory
- 000010  <- 000063: knowledge_tag_relations
- 000011  <- 000064: tenants.api_principal_config、mcp_oauth_tokens principal 列

测试:新增 internal/database 迁移测试,覆盖全新库迁移到 v11 与
v4->v11 升级保数据两条路径;go test ./internal/database 全绿。

关联 #2158;与 #2433 范围重叠(任务表部分 DDL 保持一致)。

* fix(migrations): 补齐 SQLite 迁移与 versioned 的索引与数据迁移缺口

对齐 000005/010/011 的邀请索引、tag_id 迁入与 MCP principal 唯一索引,
并扩展测试覆盖分享链接、OAuth upsert 与 v4 标签迁移路径。

* fix(deps): 升级 dingtalk-stream-sdk-go 修复 send on closed channel 崩溃

钉钉 stream SDK v0.9.1 的 processLoop 存在并发缺陷:连接断开时
processLoop 退出并 close(pongChan/closeChan),但 SetPongHandler 回调与
读消息 goroutine 仍可能向已关闭的 channel 发送,导致整个进程 panic:

  panic: send on closed channel
  goroutine [running]:
  .../[email protected]/client/client.go:161

v0.9.2-beta.1 重写了连接生命周期管理,明确修复该问题
(见 upstream issues #27/#28/#32):pong 改为带缓冲的非阻塞发送,
closeChan 改为 context 取消。

验证:go build ./internal/... 通过;go test ./internal/im/dingtalk/ 通过。

* feat: support editing document summaries (#2752)

* feat: support editing document summaries

* fix: address review feedback for manual summary editing

Reset summary edit state when the drawer closes or generation is in
flight, sync summary_status on manual description updates, and add
service tests plus API docs for partial PUT behavior.

* chore: gofmt client/initialization.go for pre-push hook

---------

Co-authored-by: wizardchen <[email protected]>

* ci: 按路径裁剪 PR checks,避免 anydoc 与无关模块 lint 误触发

migrations 或根模块测试变更不应编译 Rust 解析引擎,也不应排队 cli/client 的 golangci-lint。

* feat(frontend): 选择VLLM模型时添加supports_vision判断

由于部分文本对话模型支持视觉/多模态,例如Qwen3.6-27B等。
所以在选择VLLM模型时,额外包含 supports_vision 为 true 的模型

* fix(frontend): narrow VLLM model filter to vision-capable chat models

Restrict supports_vision inclusion to KnowledgeQA models, extract
filterModelsByType for unit tests, and re-filter when modelType changes.

* fix(session): bound the auto-generated session title to fit the column

GenerateTitle writes the model completion straight into sessions.title,
which every shipped migration declares as VARCHAR(255). When the model
ignores the "short title" instruction and returns an explanation along with
the title, the follow-up Update fails with SQLSTATE 22001 (value too long
for type character varying(255)) and the session silently ends up with no
title. The error only reaches the log and the answer itself is unaffected,
so the failure is easy to miss.

Trim the reasoning prefix and surrounding whitespace, then truncate to 100
runes - by rune rather than byte, so a multi-byte character is never cut in
half - and log a warning when truncation happens so the prompt can be
revisited.

* fix(frontend): isolate markdown preview renderers

* perf(agent): make knowledge_search MMR selection incremental

applyMMR recomputed every candidate's redundancy against every
already-selected result on each round, re-tokenizing the selected passages
each time. With C candidates and k selections that is O(k^2) calls to
getEnrichedPassage + tokenizeSimple over full chunk bodies, and the
diagnostic average-redundancy loop afterwards re-tokenizes another O(k^2)
times. Since k defaults to len(candidates), this is quadratic in the number
of results that survive dedup/rerank.

On a knowledge base where ~250 candidates reach the MMR stage this burns
tens of seconds of pure CPU inside every knowledge_search call, and no log
line attributes the time to MMR.

Cache each candidate's running max jaccard against the selected set and fold
in only the newly selected result per round, and reuse the already-computed
token sets for the average-redundancy diagnostic. Tokenization now happens
once per candidate. Selection output is unchanged, including tie-breaking;
the added test pins the new implementation against the previous one over a
deterministic corpus, and a benchmark contrasts the two.

* docs(cli): sweep stale wire vocab from AGENTS.md and close the CI gap that let it drift

The v0.9/v0.10 renames left `agent_invoke`, `session continue-stream`, and three
deleted `mcp.*` error codes documented as current, so agents integrating against
AGENTS.md branch on a surface the CLI no longer has.

Two gaps let this drift sit undetected:

- check-skill-wire-vocab.sh already banned every one of these terms, but only
  scanned skills/. The authoritative contract doc that skills are condensed from
  was never checked. Scan AGENTS.md and README.md too, and assert the scan
  targets exist so coverage cannot silently shrink again. CHANGELOG.md stays out
  of scope: recording a rename requires naming the legacy term.
- errors_doc_test.go only verified code -> doc, so a removed code could linger in
  the reference table forever. Add the reverse direction.

Also drops the matching `mcp.*` reference from the ExitCode doc comment.

* docs(cli): keep README.md out of the wire-vocab scan

README will carry upgrade notes for agents moving between CLI versions, so it
must be allowed to name legacy terms. Scan only current-truth sources: skills/
and AGENTS.md. CHANGELOG.md stays excluded for the same reason.

* fix(cli): wrap overlong t.Errorf so golangci-lint lll passes

* docs: fix URL-encoded slash in Trendshift badge alt text (#2757)

* Fix README formatting and update text

* Fix image alt text formatting in README_CN.md

* Fix badge alt text in README_JA.md

* Fix badge alt text formatting in README_KO.md

* Update README.md

* fix(wiki): stop housekeeping from force-failing documents with durable Wiki backlog

The stuck-row sweep asks the asynq inspector whether a candidate still has
work queued, and treats "nothing queued" as proof the row is orphaned. For
Wiki that question cannot be answered by asynq at all:

  - Wiki ingest work is durable per document in task_pending_ops, keyed by
    dedup_key = knowledge ID (newWikiIngestPendingOp).
  - asynq only receives a per-KB wake-up trigger, whose payload carries
    KnowledgeBaseID and no knowledge ID (enqueueWikiIngestTrigger).
  - matchesKnowledge() gates on taskTypesForKnowledgeCancel, which
    deliberately excludes TypeWikiIngest.

So HasQueuedTasksForKnowledge structurally returns false for a document
whose only outstanding work is a queued Wiki ingest. The sweep then flips a
perfectly healthy 'finalizing' row to 'failed' with "task stuck in
processing", zeroes pending_subtasks_count, and the durable op later runs
anyway — which is how a document ends up failed while present in neither the
dead-letter table nor any visible queue.

Probe task_pending_ops directly before consulting the inspector. A durable
probe error defers every candidate to the next sweep instead of failing them:
wrongly failing a live document is not user-recoverable, waiting one interval
is. The inspector's own fail-safe direction is unchanged.

A nil inspector no longer disables the whole gate, only the transient half.

TestHousekeeping_NoFalseKill_DurableWikiIngestPending reproduces the bug (it
fails on the unpatched sweep); TestHousekeeping_StillRecoversWhenNoDurableOp
guards against the gate becoming a blanket amnesty.

* feat(docreader): add XMind outline parsing (#2713)

* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <[email protected]>

* feat(system-admin): add endpoint for creating users (#2722)

* feat(types): add AdminCreateUserRequest for system-admin user creation

Password un-tagged so an empty value reaches the service and triggers
OIDC-style random-password generation.

* feat(system-admin): scaffold POST /system/admin/users/create endpoint

- Handler skeleton under the SystemAdmin()-guarded group
- Binding and validation live (400); happy path 501 until the
  service logic lands
- Response type declares the once-only generated_password contract

* test(system-admin): contract tests for user creation

- Create/audit/generated-password cases assert the final contract
  and are intentionally red against the 501 stub
- Binding and trim validation cases already pass
- Adds the system.user_created audit action as the audit contract

* feat(frontend): add createSystemUser API wrapper

Mirrors promote/revoke/reset-password conventions, including the
once-only generated_password contract on the response type.

* fix(system-admin): never rewrite password; re-check trimmed username

- Handler no longer trims the password: whitespace is part of the
  credential, hashed byte-for-byte (mirrors auth.go Register)
- Trimmed username re-checked to 2-50 runes (binding ran on the
  raw JSON)

* feat(service): implement AdminCreateUser with random-password fallback

- Empty password: OIDC-style crypto/rand generation, returned once
- Explicit password: must satisfy the 8-32 letter-and-number policy
- Delegates to Register: duplicate checks, tenant provisioning and
  Owner membership bootstrap match public registration

* feat(system-admin): implement create-user handler

- Resolves tenant mode from the shared auth.default_tenant_mode
  policy (single source for registration and admin creation)
- Emits system.user_created audit without the plaintext password
- 201 + once-only generated password; 400 for policy/duplicate,
  500 for internal failures

* test(service): whitespace-only password must be rejected

- All-whitespace password: ErrPasswordPolicy, nothing persisted
  (matches AdminResetPassword); only truly empty triggers generation
- Red against the current TrimSpace-empty check
- Handler test drops the whitespace-only 201 expectation, keeps
  byte-for-byte pass-through for spacey valid passwords

* fix(service): treat only a truly empty password as generate-request

Generation now triggers only on the empty string; whitespace-only
input is rejected by the 8-32 policy like AdminResetPassword.

* fix(service): regenerate until the generated password passes policy

A 32-char base64url draw misses digits ~0.4% of the time and could
400 a creation nobody picked a password for; regenerate until the
policy passes (expected ~1.004 draws).

* refactor(system-admin): drop TenantProvisioning from create payload

The value lived both on the request and as the service argument;
keep it only as the argument, mirroring LoginWithOIDC.

* test(system-admin): pin password_generated audit value; cover dupes

- Audit assertions check the exact password_generated boolean
  (true generated, false caller-supplied)
- Duplicate mapping table-driven over email and username errors

* feat(system-admin): explicit provide-and-validate password semantics

- Password pointer-typed: absence (or null) triggers random
  generation; any provided value, empty string included, is
  policy-checked and rejected otherwise
- Service: generation on nil only, all provided values hashed
  byte-for-byte
- Tests: empty string -> 400, absent -> generate, nil pass-through
  at the handler boundary
- Frontend: document that any provided value is policy-checked

* test(system-admin): merge duplicate rejection tests

Whitespace-only and weak-password rejection tests were structurally
identical, differing only in inputs. Merged into one table-driven
test covering weak, empty, whitespace and whitespace-only values.

* feat(system-admin): idempotent create on duplicate identity

- Register returns typed sentinels ErrUser{Email,Username}Exists
- AdminCreateUser re-queries the existing row, lookup targeted at the
  sentinel-named identity (email or username)
- Handler: duplicate sentinels -> 200 + UserInfo with idempotent=true
  audit, policy errors stay 400; drops the substring-matched branch

* fix(system-admin): reject partial identity conflicts on user create

Idempotent 200 now requires both email and username to match the existing
row; partial collisions return 409. Align handler identity sanitization and
password docs with nil-or-null generation semantics.

---------

Co-authored-by: wizardchen <[email protected]>

* feat(dsh-weknora): DeepSeek Harness plugin exposing WeKnora retrieval tools (#2759)

* feat(dsh-weknora): DeepSeek Harness plugin exposing WeKnora retrieval tools

dsh ships no retrieval, embedding or knowledge-base capability of its own:
its search tools read the workspace and web_search reads the internet. This
plugin fills that gap with the user's own documents, registering four
read-only tools against a WeKnora deployment — knowledge base listing,
hybrid retrieval, full document reading, and WeKnora's own composed answer
with citations.

The package declares no runtime dependency on any harness package. It
mirrors the slice of the tool-definition contract it uses as structural
types and hands ctx.tools.register() a plain object, so a harness release
that adds optional fields cannot break an installed copy. Configuration is
validated at plugin load, so a typo fails the load with every violation
named rather than failing inside the first tool call.

* test(dsh-weknora): unit, API contract and in-harness end-to-end coverage

Three layers, each catching a different class of break.

The unit tests drive the tools against a mock WeKnora and assert what the
model actually sees. The contract layer pins every WeKnora call the plugin
makes in a fixture, then asserts it from both sides: contract.test.mjs
checks the plugin still emits exactly those calls, and contract_test.go
checks WeKnora's real Go request and response types still accept and serve
them — so a rename on either side fails CI instead of a user's agent.

The end-to-end check installs the package into a throwaway dsh profile with
`dsh plugin add`, boots the headless profile against a deterministic
OpenAI-compatible model and a mock backend, and asserts the harness agent
loop really called the tools and answered from what they returned. It runs
both documented paths: the shipped bundle configured only by environment
variables, and a profile patch that overrides the row and renames the tool
prefix.

* docs(dsh-weknora): document the plugin and wire up its CI

The package READMEs cover installation, the configuration reference, the
permission and data-flow story, and the compatibility note naming the dsh
release the plugin is verified against. The four root READMEs gain a short
paragraph pointing at it.

CI runs the tests and a packaging check on every touch, and the end-to-end
check against a pinned harness release so a plugin change is the only thing
that can turn it red. A weekly advisory job runs the same check against
dsh@latest to notice a harness breaking change early, since dsh is in
developer preview and states that breaking changes will happen. Publishing
is gated on a tag that names the packaged version, skips a version already
on npm, and goes out with provenance.

* fix(dsh-weknora): demand the retrieval scope WeKnora actually requires

WeKnora rejects a knowledge-search that names no knowledge base,
document or tag. The search tool told the model the opposite — that the
deployment would decide the scope when a call named none — so on a
deployment with no configured default, which is what the quickstart's
optional WEKNORA_KNOWLEDGE_BASE_IDS leaves behind, the model's first
search failed with an opaque HTTP 400.

The mock backend answered unscoped retrievals where the real handler
refuses them, which is why no test caught this. Align the mock with the
handler, state the requirement in the description, and reject the call
inside the plugin so the model gets a message naming the argument to
supply instead of a transport error. Scenario C covers the resulting
discovery path end to end inside dsh.

* feat(dsh-weknora): make search resolve its own scope and match names

Two gaps made the tool set awkward to use against a real deployment.

Scope: WeKnora refuses an unscoped retrieval, so the model had to pick a
knowledge base before it could search. On a real workspace that choice is
not makeable — 25 bases named "多福多寿", "撒打算", "test 副本 2". Since
knowledge bases sharing a vector store retrieve through a single fan-out
group, searching all of them costs about what searching one costs, so an
unconfigured deployment now resolves the visible set once and searches it.

Names: read_document needs a knowledge_id that only a passage hit could
supply, so "read the X handbook" had no path at all. Search now also
matches document names through /knowledge/search, which spans every base
and needs no scope, and reports the matches it finds inside the caller's
scope. A question-shaped query matches no name and costs nothing.

Also: page 1 of a document now leads with its title and WeKnora's
generated summary, so a long document can be judged without paging; ask
is described as the broad-synthesis tool it is rather than a peer of
search; and a wide scope renders as a count instead of spelling out
every id on every result.

* feat(dsh-weknora): enhance CI workflow and documentation for WeKnora plugin

Updated the GitHub Actions workflow to include Go contract tests, ensuring compatibility with WeKnora's Go types. Enhanced README files to clarify the plugin's functionality and usage. Added error handling for incomplete responses in the client, ensuring that truncated streams are properly managed. Improved test coverage for the ask tool to validate scope resolution and agent behavior.

* ci(dsh-weknora): cache pinned dsh install to speed up e2e

The e2e job spends ~15 minutes downloading @deepseek-ai/dsh on every run
while the actual harness scenarios finish in seconds. Cache the install
directory keyed by DSH_PINNED_SPEC and enable npm dependency caching.

* ci(dsh-weknora): restore e2e timeout for cold dsh cache

First-run npm install of @deepseek-ai/dsh still takes ~15 minutes; the
10-minute job limit cancelled the run before the cache could warm up.

* fix(anydoc): rebuild truncated patched crate instead of failing cargo

A leftover .weknora-patched marker made the build skip recopying even when
Cargo.toml was missing. Require that file before reusing the tree, and build
the archive before cargo audit so CI has the gitignored patch path.

* fix(datasource): persist disabled deletion sync setting

* fix(datasource): restore sync_deletions on the in-memory object after create

GORM hydrates default:true back onto the struct during Create, so the HTTP
201 body (and a later Update on the same pointer) could disagree with the
row just written.

* feat(agent): report and attribute per-turn LLM token usage

Accumulate token usage across every LLM round of an agent turn and
expose the aggregate everywhere a turn is observable:

- AgentState.TurnUsage accumulates each round's reported usage,
  including the final-answer synthesis call; TokenUsage.Accumulate
  keeps the cache counters as prompt subsets and recomputes the
  combined cache status (unsupported survives only while every call
  was itself unsupported).
- The agent completion event carries the aggregate, and the complete
  stream event exposes it both as data.usage and as a typed
  StreamEvent.Usage that buildStreamResponse promotes into the
  previously-unused StreamResponse.Usage wire field.
- The assistant message persists it as a nullable usage column on
  both migration targets (postgres 000085 jsonb, sqlite 000012 text),
  for web and IM channels alike, so history reads still carry the
  turn's cost after the live stream is gone.
- "[LLM Usage]" log lines gain ", session_id=..., principal=type:id"
  when the context carries them; calls outside sessions stay
  byte-identical.

Usage from failed retry attempts is not accumulated: the per-call
usage log remains the ground truth, while the turn aggregate serves
as the operational signal.

* fix(i18n): add missing common locale keys

* feat(sandbox): Docker 后端支持会话级长驻容器与远程守护进程

将 Docker 沙箱后端从一次性 exec 容器改为会话级长驻容器,使其会话与文件
系统语义与 E2B / Cube 后端对齐:

- 新增 docker_engine / docker_remote_client / docker_rpc_timeout,直连
  Docker Engine API,支持 TCP+TLS 远程守护进程,并为拉镜像等慢操作单独
  设置超时;daemon 地址留空时跟随本机 docker context。
- 新增空闲清理器 docker_idle_sweeper,按 exec 刷新的活跃标记回收长驻
  容器;标记文件对沙箱非 root 账号可写。
- 会话容器补 PID1 以回收僵尸进程;标准镜像补入 curl。
- 镜像即模板:docker_template_catalog 只上报可识别的沙箱镜像,标准模板
  的拉取在后台进行并报告 building。
- 设置页与租户配置支持守护进程地址、TLS 与 CPU/内存限额,四种语言文案
  同步;沙箱健康检查覆盖 Docker 后端。
- 补充 docs/sandbox-docker-backend.md 与 POC,说明能力边界与快照语义;
  git 钩子的变更包检测跳过嵌套 module,避免拿 POC 的包去跑主 module 测试。

* fix(sandbox): shell_exec 降权到沙箱账号并修复 exec 取消时的数据竞争

shell_exec 承载的是模型自己写的命令行,此前未指定 User。Docker 适配器把空
User 解析为 root,envd 系后端解析为沙箱账号,同一个工具在不同后端拿到的权限
并不一致。改为显式指定 DefaultSandboxExecUser,并把空 User 的语义收窄为仅供
manager 自身的 bootstrap 使用——chown artifact 目录必须以 root 执行,不能以
被授权的那个账号执行。

取消 exec 时原先直接读取输出缓冲区,而 StdCopy 协程仍在向同一块内存写入。改
为先关闭流让 StdCopy 解除阻塞,等它收敛后再读;超出宽限期则丢弃部分输出,而
不是从仍在写入的协程手里读走。

Docker 守护进程地址此前只校验管理员填写的字符串,而空值会在其后由 DOCKER_HOST
或 docker context 填充,真正拨号的是填充后的值。这让 DOCKER_HOST 指向明文
tcp:// 守护进程的部署能存下一份必然失败的配置。校验改到解析之后、
RequireCompleteConfig 之后,缺镜像这类表单里看得见的问题仍然优先报告。

* fix(sandbox): Docker 文件操作改走沙箱账号,堵住符号链接越权并回收僵尸进程

archive 接口(CopyToContainer / CopyFromContainer / ContainerStatPath)由 daemon
执行,它忽略 exec user 一律以 root 操作,并在解析路径时跟随符号链接。而沙箱账号对
/workspace 有写权限,调用侧的路径守卫(isUnderRoot、cleanSessionInputPath)又都是
字符串前缀比较,于是 `ln -s /root /workspace/output/esc` 之后,
/workspace/output/esc/secret.txt 既能通过守卫,又会被 daemon 以 root 读出来。真机
验证过:非特权账号 cat 该路径是 Permission denied,docker cp 同一路径拿到明文。

WriteFile / ReadFile / Stat 改为以 DefaultSandboxExecUser 走 exec,能不能读写交给
内核判定,符号链接指向哪里都不再重要,也不存在「先校验后使用」之间被换掉链接的窗口
(这正是 realpath 预校验方案挡不住的)。Stat 用 find,它不跟随符号链接,因此链接如实
报告为 other 类型,read_sandbox_file 在尝试读取之前就会拒绝。两个 copy 端点从
dockerEngineAPI 接口里移除——留着未使用的入口等于把这一类问题继续敞着。

容器入口是 sleep,从不调用 wait(),后台进程活得比启动它的 exec 久就会变成没人回收的
僵尸,堆到 pids_limit 之后所有 exec 都会失败。补 HostConfig.Init 交给 tini。实测:
三个孤儿进程,不开 Init 留下三个 Z,开了之后为零。

同步订正文档与注释中过时的 root 表述:上一次提交已把 shell_exec 降权到沙箱账号,但
docs 仍写着「exec 默认以 root 运行」「shell_exec 默认就是 root」,并据此把活跃标记可被
无限刷新归因于「容器内 root 信任边界」——标记是 chmod 666,普通技能脚本就能刷新。

* fix(sandbox): 产物目录 bootstrap 降权,堵住 chown 跟随符号链接的容器内提权

会话级容器让「产物目录 bootstrap 以 root 执行」变成一条可复现的提权链:
沙箱账号拥有 /workspace,可以把产物目录换成指向 /etc 的符号链接;下一次
执行时 root 跑的 chown/chmod 会跟随链接,把 /etc 的属主交给沙箱账号,随后
删除重建 passwd 即可让该账号在下一次 exec 时解析成 uid 0——本 PR 刚做的
shell_exec 降权和文件操作降权会一并失效。一次性 docker run --rm 下两次执行
不共享文件系统,这条链走不通。真机验证过完整链路。

修法是让 bootstrap 和其它 exec 一样以沙箱账号运行:chown 对自己刚建的目录
是 no-op,对不属于自己的目标被内核拒绝。同时把 dockerExecUser 的空值兜底
从 root 改成 DefaultSandboxExecUser,漏传账号只会失去权限而非拿到权限,
也让三个后端对「空 User」的语义一致。

顺带订正若干过强的声明:find 只对路径最后一段不跟随符号链接,中间层的链接
仍由内核展开,因此 read_sandbox_file 的「拒绝符号链接」只覆盖路径本身指向
链接的情况;对应单测原先 mock 了一个真实 find 不会产生的返回值,改为断言
它实际覆盖的场景。文件头注释仍写着文件操作走 archive 接口,也一并更新。

* fix(handler): validate knowledge search pagination

* fix(docreader): apply PDF image SMask so embedded figures are not all-black

get_bitmap() decodes only the base image plane and ignores any /SMask
(soft transparency mask). Figures exported from plotting tools often
store their visible content in the mask while the base RGB plane is
black, so extraction produced all-black JPEGs for such images.

Decode embedded figures via get_bitmap(render=True), which renders
through pdfium's imaging pipeline and carries the mask as an alpha
channel, then composite over white (JPEG has no alpha; PDF pages render
on white). Falls back to the raw decode when rendering is unavailable.

On a production PDF with 7 SMask figures, dark-pixel ratio drops from
~100% to 0.7%-4.5%. Opaque images are unchanged.

* fix(docreader): keep pdfium bitmap alive while copying SMask pixels

to_pil() shares the BGRA buffer; dropping the PdfBitmap before convert
use-after-frees. Copy pixels first, keep opaque RGB/L modes, and log when
render=True falls back to the raw (mask-blind) decode.

* fix(im): handle DingTalk rich text messages

* fix(im): tighten DingTalk rich-text follow-up handling

Keep image/file attachments out of the empty-query guard, parse audio
recognition, and surface a hint when extra rich-text pictures are dropped.

* fix(im): wrap applyRichText signature for lll

* feat(sandbox): add tenant skill snapshot infrastructure

Give Cube/E2B a shared snapshot API and persist skill image pointers so later session sandboxes can inherit installed skills from a config snapshot instead of the base template.

Co-authored-by: Cursor <[email protected]>

* fix(sandbox): keep skill image pointer out of sandbox config PUT

The settings form rebuilds the payload without skill_image, so merge
must ignore client values or a save would drop the snapshot (and a
crafted PUT could plant one). Also reject skill names that escape
the image root and require SupportsSnapshots on the snapshot manager.

* docs: clarify Docker image release tags

* feat(chat): display message timestamps

* fix(chat): align live timestamps with persisted created_at

Echo the stored user/assistant times on the first stream event so the
UI stops showing a client-clock fallback that would jump after reload.

* fix(chat): type the timestamp test user fixture

vue-tsc failed because bindServerTurnTimestamps assigns id onto an
object that was inferred without that property.

* feat(sandbox): add tenant skill install and remove service layer

Introduce the hidden skill-installer agent and TenantSkillService so a
sandbox config can install or remove skills into snapshot images under a
per-config lock, without yet exposing HTTP or UI entry points.

Co-authored-by: Cursor <[email protected]>

* fix(sandbox): harden tenant skill install against races and credential rotation

Abort a queued install if the skill row was removed or superseded, refuse to
restamp a snapshot with a rotated provider fingerprint, and honor the workspace
sandbox kill switch before creating a maintenance session.

* fix(sandbox): keep skill install from clobbering a newer owner

A queued remove or a failed older install could stamp or delete a row a
re-upload already claimed, and an empty MCPSelectionMode handed the root
installer every tenant MCP tool. Fail closed on those races, pin the
installer surface, and refuse a missing bundle or leftover zip files.

* fix(sandbox): satisfy pre-push lint on the skill install surface

Break long lines and add the missing constructor comment so the
maintainer push hook can reach the fork.

* fix(sandbox): apply gofumpt spacing in skill-install tests

* fix: stop markdown table regex catastrophic backtracking (#2770)

* fix: stop markdown table regex catastrophic backtracking

The line_pattern in MarkdownTableUtil used overlapping character classes
([\t ]* and [^|\r\n]*) so a `|`-prefixed line missing its trailing `|`
forced the engine to explore an exponential number of backtracking paths,
stalling the single-threaded docreader gRPC event loop (100% CPU, health
check timeout).

Replace whole-text regex substitution with an O(n) line-based pass that
only treats rows beginning and ending with `|` as tables, and preserves
the previous align_pattern semantics via _is_alignment_row.

Fixes #2768

Co-Authored-By: Claude <[email protected]>

* fix: keep lone pipe rows and normalize CRLF in table formatter

A single "|" was rewritten into an empty table row and then dropped.
Unify CR/LF before the line scan so exported Markdown tables format
without mixed endings.

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: wizardchen <[email protected]>

* fix(chat): insert timestamps as conversation flow dividers

Show a centered ChatGPT-style time marker between turns instead of
under every bubble, so thinking steps no longer sit on a dangling clock.

* fix(frontend): expose history and retrieval retention to smart-reasoning agents

The Agent Editor hides the whole Conversation Settings group when the
running mode is smart-reasoning, but the agent path reads history_turns
from that group: session_agent_qa.go hands it to LoadAgentHistory, which
keeps only the newest N turns and drops the rest. An agent created through
the editor therefore runs on the server default of 5 turns, and the loss is
invisible because the UI renders the full conversation straight from the
messages table.

The gate was correct when it was added. Until 1f5970b6 the agent path
deliberately skipped the limit -- "we don't apply HistoryTurns limit here.
HistoryTurns is used in normal (KnowledgeQA) mode" -- and that refactor made
the value live for agent mode without the frontend following.

Ungate the section, the nav entry and the mode-switch redirect. Keep the
multi_turn_enabled switch hidden, though: EnsureDefaults pins it to true for
smart-reasoning, so an editable switch would silently revert on save. The
turns input carries an extra `|| isAgentMode` because a new agent defaults to
smart-reasoning with multi_turn_enabled=false in the form, which would
otherwise hide the input in exactly the case this change is about.

Raise the turns cap from 20 to 100. Twenty rounds sits two orders of
magnitude below the 200k-token context budget the engine already manages
(with LLM consolidation at half that), so the round cap always bites first
and the summariser rarely gets to run. The server does not validate this
field, so the widget was the only limit.

Surface retain_retrieval_history as well. It is read only by the agent path
(internal/agent/observe.go) and had no widget at all, leaving the API as the
only way to reach it -- builtin-wiki-fixer is the sole preset that turns it
on. It rewrites the history of the eight KB and Wiki tools only, so it is
shown when the agent has a knowledge base.

The section description advertises query rewriting, which agent mode does not
show, so it now branches per mode.

* feat(sandbox): expose tenant skill management over HTTP and in the console (#2789)

* feat(sandbox): enhance skill management in sandbox configurations

Updated the skill management system to allow listing, installing, and removing skills within sandbox configurations. The ListSkills function now accepts a sandboxConfigID to filter skills based on the selected configuration. Introduced a new SandboxSkillsDrawer component for managing skills through the UI, along with necessary updates to the API and localization files to support these features.

Co-authored-by: Cursor <[email protected]>

* fix: keep skill maintenance sessions out of the console session list

Co-authored-by: Cursor <[email protected]>

* feat: store and expose the installer transcript locators

Co-authored-by: Cursor <[email protected]>

* feat: record the skill installer agent's conversation

Co-authored-by: Cursor <[email protected]>

* feat: stream and persist the installer transcript during a skill install

Co-authored-by: Cursor <[email protected]>

* feat: expose install transcript locators on the skill type

Co-authored-by: Cursor <[email protected]>

* feat: add the floating skill install transcript window

Co-authored-by: Cursor <[email protected]>

* feat: open the install transcript from the sandbox skills drawer

Co-authored-by: Cursor <[email protected]>

* feat(sandbox): 重构技能安装体验,支持内联时间线、编辑第4步及保留镜像指针

- 替换右下角浮窗为内联安装时间线 (SkillInstallTimeline.vue)
  - 新增 GET /api/v1/sandbox-configs/:id/skills/:skillId/transcript 接口,支持回放与追尾
  - 移除 /sessions/continue-stream 依赖,避免 404 问题
  - 事件日志以 install_prompt 开头,不再将工具调用 preamble 拼入最终答案
  - 删除 SkillInstallTranscript.vue,技能条目下可展开/折叠时间线

- 编辑向导新增第 4 步「管理技能」
  - Cube/E2B 环境:连接 → 模板 → 运行配置 → 技能(第4步)
  - Docker/Local 环境:无第4步(无快照镜像)
  - 首次创建必须按 1→2→3 顺序,第3步保存成功后自动进入第4步
  - 已有配置支持步骤条数字跳转,可直接访问任意步骤
  - 卡片菜单「管理技能」打开同一抽屉并定位至第4步
  - SandboxSkillsDrawer.vue 重命名为 SandboxSkillsPanel.vue,去除独立抽屉壳

- 修复保存运行配置时冲掉 skill_image 的问题
  - 顶部展示信息(基础模板/快照ID/版本/构建时间)读取 config.skill_image
  - 编辑器保存 payload 不再包含 skill_image,合并时保留已有 SkillImage 和 VolumeMount
  - MergeSandboxConfigForUpdate 增加对 SkillImage 和 VolumeMount 的保护
  - 已冲掉的配置需重新安装技能才能恢复镜像指针

- 「查看安装过程」在安装开始时即显示
  - 技能列表中出现「正在安装」状态即展示按钮
  - 若记录尚未写入,展示等待提示并自动重试,避免误报「没有留下记录」

* 沙盒内技能目录从 UUID 改成 SKILL.md 的 name。

* fix(sandbox): heal stuck skill rows and block planted maintenance sessions

The reaper left a mid-upgrade install spinning forever when the old snapshot was still live, and restored removals to ready after the pointer had already moved. Client session create/update can no longer hide chats behind the skill_maintenance: prefix.

* style: gofumpt the skill reaper test stubs

* style: satisfy golangci on the tenant skill admin API

* feat(sandbox): rebuild skill images at turn boundaries, not mid-exec

Hold a session turn lease through AgentQA so an install cannot tear down
the live sandbox between tool calls; the next turn's first resolve picks
up the new image. Configs can instead keep existing sandboxes and boot
the new snapshot only for new sessions.

* fix(sandbox): judge stuck skill runs from the snapshot chain

InstalledSnapshotID only records the generation that installed a skill; later
installs of other skills advance the live pointer without rewriting older rows.
Comparing those IDs made the reaper fail healthy installs and delete live
skills whose removal had stalled.

---------

Co-authored-by: hailongzhao <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: wizardchen <[email protected]>

* ui(sandbox): rename settings to skill sandbox and fix skill install (#2792)

* ui(sandbox): rename settings to code sandbox and simplify cards

"Sandbox backend" leaked implementation jargon into the settings nav.
Call the page what users configure — an isolated runtime for skill
scripts — and only surface incomplete-config warnings on the cards.

* fix(sandbox): type the card overflow menu options

vue-tsc rejected pushing a themed delete item onto an untyped options
array; name the option shape so the menu still marks delete as danger.

* fix(sandbox): keep the editor wizard moving into skills

Saving on the runtime step used to close the drawer when editing, so
step 4 was only reachable from the rail. Continue after save, move
full verification off the runtime form, and stop treating "using the
base template" as the empty value for every image field.

* fix(sandbox): store skill bundles on the workspace default backend

Skill install passed a stub tenant with no default storage, so saving
the zip failed with empty provider and the 500 hid the real cause.

* ui(sandbox): regroup skill install and rename to skill sandbox

Show install vs installed groups, fold long descriptions in the list,
and call the settings surface a skill sandbox.

* ui(sandbox): show skill install transcript in a compact popup

Keep the card from growing; the run opens beside the history button
with a capped height and its own scroll.

* ui(sandbox): show installed skills on config cards

Cube/E2B tiles now list skill names so the list reads as skill sandboxes; a click opens the editor on the skills step.

* ui(sandbox): show finished skill installs as static history

A completed run already has durable messages; replaying the event log would animate the whole install again. Hide chat answer actions in this transcript.

* style(frontend): enhance compact skill timeline display with improved font sizes

Updated the CSS for the compact skill timeline to adjust font sizes for various elements, ensuring better readability in the popup interface. This includes specific adjustments for embedded agent streams and markdown content.

* ui(sandbox): refactor header structure in SandboxConfigEditorDrawer and SettingDrawer

Updated the header structure in both SandboxConfigEditorDrawer and SettingDrawer components to improve layout and organization. Added a new slot for extra header content in SettingDrawer and adjusted CSS for better alignment and spacing in both components.

* ui(sandbox): enhance skill loading feedback and improve deep check logic

Updated the SandboxConfigEditorDrawer to refine the logic for deep checks based on the current step. Enhanced the SkillInstallTimeline component to manage in-flight live loops and provide better feedback for skill loading failures. Adjusted the SandboxSettings view to display error messages for failed skill loads, ensuring users are informed of any issues during the skill installation process.

* fix(sandbox): skip identical skill installs and prune old snapshots (#2791)

* fix(sandbox): skip identical skill installs and prune old snapshots

Re-uploading the same archive of a ready skill was still booting a billed sandbox and growing a new generation. Superceded snapshots also accumulated on the provider because the ledger never deleted them.

Skip the rebuild when the digest already matches a skill still in the live image, and let the reaper delete superseded provider snapshots after 24h while leaving extras not in the ledger untouched.

* fix(sandbox): retry stale same-zip installs and prune active leftovers

A re-upload of the same archive skipped every installing row, so a dead process could not recover until the reaper. Skip only recent in-flight runs, still refresh the stored bundle, and prune non-live active snapshots after retention.

* fix(sandbox): beat install liveness and guard the snapshot prune

A fixed window on the submission time cannot tell a slow install from a
dead one: a single agent command may take installCommandTimeout and an
install runs several of them, so a two-minute window called live runs
dead, reset their progress and pushed the reaper's own deadline out.
Stamp InstallingSince while the run works and read that instead, which
also stops the reaper from failing installs that are merely long.

Answer the skip question from the image only for a ready row. The ledger
records which skill a snapshot carries, not which archive, so a row
installing a new bundle would look finished from the previous bundle's
snapshot and report a success that never happened.

Prune now refuses to delete once the credentials no longer address the
account that built the image: the delete would come back not-found, and
the ledger would record another account's still-billed snapshots as
gone. It also waits for an eligible row before building a provider
client, since most configs have nothing to prune on most sweeps.

* feat(sandbox): install skills from ClawHub, SkillHub, and git URLs

Paste a registry or GitHub link in the skill sandbox instead of only
uploading a zip. skillhub.cn pages resolve through api.skillhub.cn.

* refactor(sandbox): remove token requirement for skill installation from source

Updated the skill installation process to eliminate the need for a token when installing skills from ClawHub, SkillHub, or Git URLs. The source must now be readable anonymously, ensuring that no credentials are sent during the fetch process. Adjusted related documentation and UI components to reflect this change, enhancing the user experience by simplifying the installation workflow.

* docs(sandbox): clarify skill source requirements in installation process

Updated documentation across multiple files to specify that the `source` for skill installation must be explicitly defined as `@owner/slug` for ClawHub or a full URL for GitHub/SkillHub. Bare `owner/slug` is now rejected as ambiguous. This change enhances clarity for users regarding valid input formats and improves error handling in the installation process.

* fix(frontend): sync settings nav with a single section query

Clicking most settings sidebar items left the URL on
?section=system-global, so the route watcher snapped the panel back.
Write ?section=<navKey> for every page, including integration-claw,
and keep old ?tab= bookmarks as aliases.

* fix(frontend): align settings route query types with vue-router

* ui(chat): restyle the generated-files drawer to match settings lists

Replace the outline download pill and generic attachment icons with the
same header badge, muted file glyphs, and text icon button used by the
settings drawer and knowledge file rows.

* feat(chat): render shell_exec as a command and stream card

Give sandbox shell calls a display_type and keep stdout/stderr on the
live payload so the timeline can show the command, exit code, and
streams instead of the generic raw-output fallback.

* ui(chat): mute the generated-files count on the answer toolbar

Replace TDesign's brand-pink badge with the same secondary-container
count used by settings lists so the download control matches the
surrounding ghost icon buttons.

* fix(sandbox): advertise only image skills and unstick installs

The host skills/preloaded tree is not what execute_skill_script finds
inside the sandbox, so stop merging it into the agent's advertised set.
Seed install files as a tar, publish transcript locators before the copy,
and wait in the UI instead of 404-polling.

* fix(sandbox): hide Cube skill snapshots from the template catalog

Cube stores snapshots in the same template store as GET /templates, so
skill-image generations leaked into the settings picker. Subtract
ListSnapshots (and snap- IDs) so the step only offers base templates.

* feat(sandbox): browse installed skill files from settings

Read the stored skill bundle without booting a sandbox, and open a
resizable secondary drawer with a file tree and markdown preview.

* feat(service): enhance skill file handling with caching and concurrent downloads

Implemented caching for skill bundle archives to reduce redundant downloads and improve performance. Added tests to ensure that listing and reading skill files coalesce concurrent downloads, ensuring efficient access to skill files. Introduced new functions for managing skill zip entries and improved error handling for missing files.

* ui(agent): merge skills and sandbox into one editor tab (#2806)

* ui(agent): merge skills and sandbox into one editor tab

Skills and the sandbox they run in belong together, so the agent editor now shows one Skills section with sandbox selection driving the available list.

* fix(frontend): type selected skill names when pruning

vue-tsc could not infer the selected_skills filter callback after JSON-cloned form state.

* feat(sandbox): install skills on the Docker backend

Docker already ran session sandboxes, but skill install required snapshots
that only Cube and E2B advertised. Commit local skill images, boot sessions
from them, and show the skills step for Docker configs.

* feat(snapshot): introduce planned name for skill snapshots to identify abandoned builds

Added a new column `planned_name` to the `tenant_skill_snapshots` table to store the intended name of a snapshot before the provider call. This allows for the identification and reclamation of abandoned builds that may have left unreferenced snapshots due to process failures. Updated related functions to utilize this new field for better snapshot management and cleanup.

* feat(sandbox): pin Cube DNS on templates and freeze skill snapshot configs

Guest DNS only applies through the WeKnora template spec, and sessions
already on a skill snapshot would miss connection or rebuild changes.
Stop auto-creating templates, require explicit create/rebuild, and
refresh the settings card list when installs finish.

* fix(sandbox): drop unused ListSnapshots parameter in config tests

* fix(sandbox): rebuild templates without dropping the live ID

Deleting the WeKnora template before its replacement was ready left
sessions targeting a missing ID, and freeze was per-config while rebuild
is cluster-scoped. Rebuild in place when possible, persist the new
template ID, refuse replace across siblings with skills, and name
snapshots by tenant and config on every backend.

* fix(sandbox): delete replaced templates only after persist succeeds

A building or unpersisted replacement left sibling configs pointing at
a deleted ID. Snapshot names now include a row token, and the settings
UI freezes retargeting while a skill install or remove is in flight.

* feat(api): honor resource_urls=public on hybrid-search

Single-KB retrieval was still returning resource:// handles even when
callers asked for public URLs, unlike knowledge-search.

* feat(sandbox): remove the Local host-process backend

Local was a host-subprocess fallback that skill install, snapshots, and
session sandboxes already excluded. Drop it so Cube, E2B, and Docker are
the only backends; leftover local configs can only be deleted.

* ui(sandbox): explain who still holds a config (#2816)

* ui(sandbox): explain who still holds a config instead of occupancy metrics

The occupancy drawer was a three-stat dashboard plus truncated session
IDs, which did not say why a config could not be deleted. Show a plain
summary, named sessions, and bound agents; slim the card menu to edit /
who-is-using / delete.

* ui(sandbox): tighten occupancy chrome and rename running instances

Drop TDesign's extra divider (it stacked a second line), compact the
card menu and drawer, and replace the colloquial copy with a single
sentence plus the session list.

* ui(sandbox): match occupancy UI to settings chrome

Drop the homemade menu divider and occupancy-only drawer styling that stacked extra hairlines. Use the same dropdown as other setting cards, and label occupying chats as sessions so the drawer is readable.

* ui(agent): unify skill naming and icon across editor and mentions (#2818)

* ui(agent): unify skill naming and icon across editor and mentions

Drop the bilingual "技能 Skills" label and mixed lightbulb/bookmark icons so the editor tab, @-mention menu, and chat chips share one name and glyph. Tighten the sandbox picker so configs show name plus type instead of parenthetical labels.

* test(agent): follow the unified skills editor chrome

The section no longer ships a help-circle; assert the shared skill icon and sandbox option layout instead.

* fix(agent): keep compacted sandbox streams in persisted history

shell_exec with display_type was collapsed to a one-line omit placeholder,
so the next turn and a history reload lost stdout/stderr. Keep a capped
copy of the streams in both Output and Data.

* feat(agent): preview skill artifacts and verify installs without running them (#2823)

* feat(agent): preview skill artifacts and verify installs without running them

Generated files open in the chat drawer with overlay/Esc returning to the
list, and install verification no longer executes guessed skill entry
points. A failed install can be retried from the saved archive.

* ui(chat): use a folder icon and corner count for skill artifacts

The eye and document glyphs read as preview or copy, and TDesign's
loading spinner sat beside the icon. A folder with a top-right count
matches the generated-files drawer without covering the toolbar action.

* feat(artifact): implement inline artifact references with previews

Add support for inline artifact references in Markdown, allowing generated files to be displayed as clickable cards. Implemented a new rendering mechanism for artifacts, enabling previews directly from the chat interface. Updated localization files for new inline preview hints and missing file messages. Introduced tests for artifact reference normalization and rendering logic.

* fix(client): document AgentResponseType constants for revive

Touching the const block made golangci-lint require comments on every
exported value. Match the rest of the SDK and describe artifacts_pending.

* fix(agent): wrap artifact-reference prompt lines for lll

Pre-push golangci-lint rejected the branch on three lines over 120
characters in the sandbox: reference guidance.

* fix(agent): scope HTML scripts and tighten install verification

Keep scripted HTML preview on skill artifacts only, recreate sandbox
workspace dirs after they are deleted, and treat nested skill files as
dependencies rather than first-party imports.

* refactor(artifact): enhance artifact reference handling and metadata

Updated artifact metadata to include a stable resource handle for better identification. Adjusted artifact reference resolution to support both handle and name forms, ensuring compatibility with existing references. Enhanced tests to validate the new handling logic and ensure proper rendering of artifacts in chat messages. Improved documentation for clarity on artifact reference formats and their usage in the system.

* test(sandbox): drop TEST-NET DNS fixture from Cube tests

Use the same 8.8.8.8 nameserver already used by other Cube DNS tests instead of RFC 5737 documentation addresses.

* fix(agent): stop using the chat model to rerank knowledge search

Dedicated rerank already scores the recall set; falling back to the
conversation model mixed RRF scores with [0,1] ratings and could
resurface documents the reranker had rejected.

* fix(sandbox): pin Docker sessions so skill artifacts get collected

ArtifactCollector only drains /workspace/output from the pinned sandbox
config. Docker is session-persistent like Cube/E2B, but execution skipped
the pin, so generated HTML showed as unavailable after the turn.

* fix(i18n): 修复内置智能体名称与描述不随界面语言切换的问题 (#2828)

* fix(i18n): 修复内置智能体名称与描述不随界面语言切换的问题

问题现象:切换界面语言后,内置智能体的名称/描述仍显示旧语言,
需要强制刷新(Ctrl+F5)才能恢复正确语言。

35820ebb 已在 GetAgentByID / ListAgents 读取路径按请求语言重新本地化,
本提交补全剩余场景:

后端:
- 共享智能体读取(sharedAgentInfo / GetSharedAgentForTenant)与组织
  共享列表(ListOrgAgentShares)补 ApplyBuiltinAgentLocalization;
- IM 渠道列表(ListChannelsByTenant)改传请求 ctx,内置渠道的
  AgentName 按调用方语言从 YAML i18n 重新解析(DB 列可能为空或为
  落库时语言的残留值);
- 写入侧收敛:内置智能体配置落库时持久化 YAML default 语言的展示
  字段(GetBuiltinAgent 而非 GetBuiltinAgentWithContext,与 locale
  无关),更新/创建接口返回值再按调用方语言本地化,避免未覆盖的
  读取路径显示随机语言。

前端:
- utils/request.ts 导出 getCurrentLanguage;
- chatResources 的 agents 缓存与 orgStore 的 sharedAgents 缓存在
  新鲜度判断中加入"加载时语言 == 当前语言",切换语言后缓存立即
  失效,下次访问按新 Accept-Language 重新拉取并响应式更新。

* fix(i18n): 按请求发起时的语言给智能体缓存打戳,避免切换语言时命中旧数据

落地预取尚未返回就改界面语言时,原先会把 zh-CN 响应标成 en-US 新鲜缓存,最多冻 60s。in-flight 只复用同语言请求,写入前校验代际,并补共享/IM 本地化回归测试。

---------

Co-authored-by: asmh1989 <[email protected]>
Co-authored-by: wizardchen <[email protected]>

* feat(sandbox): inject per-user and workspace skill env vars at execution

Skills declare the environment variables they read at run time, so a
credential no longer has to be an admin-set value shared by everyone
using the same skill image.

The installer agent writes a declaration the server validates against the
bundle. Admins fill workspace-wide values from the skill drawer, members
fill their own under Settings, and a successful shell_exec captures the
declared names a run already used for the current speaker. Resolution is
keyed by Principal so IM callers do not collapse onto one synthetic
account, and values are injected only into the process of that one run.

Values are AES-GCM encrypted at rest and never returned by any endpoint.

* fix(sandbox): make skill env capture fill blanks only

Capture existed so a credential a user typed in chat survives the turn.
It also overwrote values already stored, inferred the skill from a path in
the command, and logged the command that carried the secret — so one
hallucinated `export KEY=test` could permanently replace a working key,
and the key ended up in the application log either way.

Capture now only fills a blank: a name the workspace or the caller has
already filled in is dropped before the write path sees it, the skill must
be named explicitly, and assignment values are masked in logs.

The path the error message advertised did not exist: a required variable
was refused before the call's own env was considered, so an IM user, who
has no settings page, could never get the skill to run. Values supplied by
the call now satisfy the requirement, and the message names the tool that
accepts them.

Also take envs out of the generic UpdateSkill. Every install-progress
write is a read-modify-write of the whole row, and the heartbeat reloads
it every 30 seconds, so carrying envs along could put a stale declaration
back over one just recorded. Encryption failures abort the write instead
of storing plaintext in a queryable JSONB column, matching TenantAPIKey.

Reverts an unrelated deletion of the no-volume-support warning.

* docs(api): document and expose the skill env var surface

The endpoints this PR added were reachable but undescribed: Swagger had no
/me/env-vars paths, docs/api/skill.md never mentioned the admin PATCH, and
the Go SDK could neither set a workspace value nor a personal one, so an
integration had to hand-roll the calls.

Regenerates the swag artifacts and adds the missing client methods, plus
the caveat that matters most in practice: values are stored per calling
identity, so a personal value entered in the web UI never applies to a run
driven by an API key.

* feat(chat): add a question outline for long overflowing sessions (#2839)

* feat(chat): add a question outline for long overflowing sessions

When the thread overflows, show a centered tick rail of loaded user questions on the right. Hover opens the list; click jumps to that message. Pure frontend, no API change.

* feat(chat): move question outline left with mountain hover and Q&A preview

Put the tick rail on the left inner edge. Hover grows nearby ticks like a mountain and shows a following card with that turn's question plus an answer preview.

* fix(chat): allow offsetTop on minimap tick mapping input

The type-check CI failed because tests pass measured items with offsetTop into mapQuestionTicks, which only accepted { id }.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: hailongzhao <[email protected]>
Co-authored-by: Cursor <[email protected]>

* fix(sandbox): wait until Docker containers are running before exec

ContainerStart returning success does not mean State.Running, so the first
skill-install ExecCreate can 409. Wait for running after start, and retry
once if exec still races.

* fix(sandbox): allow skills to declare WEKNORA_* credential env vars

The installer reserved the entire WEKNORA_ prefix, so names like
WEKNORA_API_KEY were dropped while install still succeeded. Reserve only
the sandbox-injected names (and WEKNORA_SKILL_*), and keep that list in
one place.

* ui(sandbox): edit skill env vars in a popup

Only show the key when the skill declared env vars, and open the editor
as a popup like the install transcript. Hide the hover tooltip while the
popup is open and keep password managers from filling the fields.

* ui(sandbox): redesign personal sandbox secrets settings

Lead with skill credentials, hide empty inputs until edit, and rename the page to 沙箱密钥 so it is not mistaken for system env vars.

* ui(chat): refine the question outline to a tighter Codex-like rail

Pack ticks closer, hide a single-question thread, and keep the hover
preview from inheriting the 20px chat type so the query stays readable.

* ui(skills): split the skill catalog out of the sandbox editor

Installing skills was buried as a wizard step. Give it a first-class settings page so admins can browse and install without opening the sandbox drawer, and keep rollout on the runtime step.

* fix(agent): keep skill-script stdout on failure for the model and UI

Failed execute_skill_script results w…
@zd52357

zd52357 commented Sep 12, 2026

Copy link
Copy Markdown

补充一个与该功能相关的问题:

我使用 WeKnora v0.8.0 Docker Compose 部署,测试 .xmind 文件上传时,发现虽然 XMind parser 已经实现,但知识库文件上传入口仍然会将 .xmind 判定为不支持的文件类型。

环境:

  • WeKnora:v0.8.0
  • Deployment:Docker Compose
  • wechatopenai/weknora-app:latest
  • app image revision:1edcd54b43606d9079bb36650efe3f68707a79ea
  • docreader 中已经存在 docreader/parser/xmind_parser.py
  • docreader/parser/registry.py 已经存在 "xmind": XMindParser

复现步骤:

  1. 创建一个新的空知识库。
  2. 上传任意 .xmind 文件。
  3. 上传失败,返回:
error code: 1007
message: unsupported file type

app 日志:

Checking file type: 0108车.xmind
Invalid file type
unsupported file type

进一步检查发现:

internal/application/service/knowledge_util.go

中的:

var supportedImportFileExtensions = map[string]struct{}{
    "pdf": {}, "txt": {}, "docx": {}, "doc": {}, "epub": {},
    "html": {}, "htm": {}, "mhtml": {}, "md": {}, "markdown": {},
    "png": {}, "jpg": {}, "jpeg": {}, "gif": {},
    "csv": {}, "xlsx": {}, "xls": {}, "pptx": {}, "ppt": {}, "json": {},
    "mp3": {}, "wav": {}, "m4a": {}, "flac": {}, "ogg": {},
}

没有包含:

"xmind": {},

而 docreader 侧已经存在:

docreader/parser/xmind_parser.py
docreader/parser/registry.py
    "xmind": XMindParser

因此目前的处理链路应该是:

XMind upload
    ↓
app: supportedImportFileExtensions
    ↓
.xmind not included
    ↓
1007 unsupported file type
    ↓
docreader/XMindParser never gets called

我在本地将 knowledge_util.go 的白名单增加:

"xmind": {},

重新构建 app 后,文件可以正常进入后续处理流程,说明问题应该就是上传入口的扩展名白名单遗漏。

另外,0.8.0 CHANGELOG 已经将 XMind Outline Parsing 标记为新功能(#2713),因此这里可能是 XMind 功能实现后遗漏了 app 侧统一扩展名白名单。

建议在 supportedImportFileExtensions 中补充 xmind,并增加相应的测试,避免后续再次出现 parser 已支持但上传入口拒绝的问题。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: 期望内置 DocReader 支持 xmind 格式解析

4 participants