Skip to content

feat(api): support self-hosted MinerU 4.x via the V1 API - #209

Merged
luoshuai990529 merged 2 commits into
Zleap-AI:mainfrom
kabishou11:feat/mineru-self-hosted-v1
Oct 3, 2026
Merged

luoshuai990529 merged 2 commits into
Zleap-AI:mainfrom
kabishou11:feat/mineru-self-hosted-v1

Conversation

@kabishou11

Copy link
Copy Markdown
Contributor

Relates to #208(维护者回复欢迎先提交自部署 MinerU 的 PR)

背景 / Why

SAG 目前只能接 302 和 mineru.net 云端 MinerU:_assert_public_host 会拒绝 loopback/内网地址,无法对接本机或局域网部署的 MinerU。MinerU 4.0 把 HTTP 服务重写为 V1 API(/v1/uploads、/v1/parse/jobs、/v1/files,移除了 /file_parse、/tasks),结果 zip 中的 Markdown 也改名为 markdown.md。

SAG can only use cloud MinerU today; self-hosted MinerU 4.x (V1 API) is blocked by the private-address check and the new protocol/zip layout.

修改 / What

  • 新增 mineru_provider="self_hosted"(parsing/mineru_v1.py),按 MinerU 4.0 官方 docs/usage/http_api.md 与 scripts/http_api_example.sh 实现 V1 流程:
    POST /v1/uploads(带 sha256,命中去重时跳过上传)→ PUT upload_url → POST /v1/uploads/{id}/complete → POST /v1/parse/jobs(output_formats=["zip"]、ocr_mode、可选 tier)→ 轮询 GET /v1/parse/jobs/{id} 至 completed/partial/failed/canceled → GET /v1/files/{zip_file_id}/content 下载 zip。
  • 复用现有解析状态:file_id/job_id 写入断点状态,暂停(should_pause)与续跑沿用原逻辑;V1 任务状态只在服务进程内,服务重启后 job 404 时自动重新上传提交一次。失败、超时、鉴权错误都映射为现有错误类型,因此继续走 MarkItDown 降级。
  • 安全边界 / Security:
    • 新增 mineru_allow_private_base_url(默认 false)。只有显式开启后,用户配置的 Base URL 才允许解析到本机/内网;未开启时给出明确的配置提示。
    • 结果下载(含重定向)必须与 Base URL 同源(scheme + host + 端口),否则拒绝。
    • 上传地址同源时才带 API Key;跨源上传地址(如对象存储签名 URL)不带 Key,并继续走原有公网地址校验。其余 SSRF 检查不变。
  • _markdown_from_zip 同时识别 markdown.md。
  • mineru_tier(可选,留空用服务端默认档位);自部署服务可匿名,mineru_configured 对 self_hosted 不再要求 API Key。
  • settings_service._normalize_overrides:只在持久化配置里有 MinerU 地址时才推断服务商。此前即使从未保存过 MinerU 配置,启动时也会把 SAG_MINERU_PROVIDER 重置为 302(official 同样受影响)。
  • compose.yaml、.env.example 透传并说明 SAG_MINERU_PROVIDER / SAG_MINERU_TIER / SAG_MINERU_ALLOW_PRIVATE_BASE_URL。
  • 仅后端,不新增依赖(纯 httpx);设置页会在后续 PR 中补充。Backend only, no new dependencies; the settings page will follow in a separate PR.

验证 / How verified

真实 MinerU 4.0.10 端到端(本机 mineru-kit api-server --tier flash --api-key ...,CPU,SAG 客户端直连 http://127.0.0.1:18000):

1) 未开启 allow_private:ConfigurationError:MinerU 自部署 Base URL 指向本机或内网地址;确认可信后请设置 SAG_MINERU_ALLOW_PRIVATE_BASE_URL=true
2) 开启后:markdown = 'Self hosted MinerU marker\n';状态依次记录 file_id → job_id → running → done
3) prepare_document:provider=mineru,缓存 e2e2.pdf.parsed.mineru-v1-flash-auto.md
4) 错误 API Key:服务端 401 → 降级 markitdown,fallback_error = "MinerU 创建上传鉴权失败,请检查 MinerU API Key"
5) 断点中的 job_id 已失效(模拟服务重启):GET job 404 → 重新上传并提交 → 成功
6) 同一文件再次解析:POST /v1/uploads 直接返回 completed(sha256 去重),跳过 PUT → 成功

服务端日志中的请求序列:POST /v1/uploads 200 → PUT /v1/uploads/{id}/content 200 → POST .../complete 200 → POST /v1/parse/jobs 202 → GET /v1/parse/jobs/{id} 200 ×N → GET /v1/files/{id}/content 200;真实 zip 内容为 markdown.md、middle_json.json、structured_content.json、model_output.json。

单元测试:新增 tests/test_mineru_self_hosted.py(httpx MockTransport,响应结构按 4.0.10 实际返回编写),覆盖:完整流程与请求体、sha256 去重、匿名访问、内网地址开关(关闭拒绝 / 开启放行 / 公网无需开关)、下载重定向同源校验、跨源上传不带 Key 且内网跨源被拒、failed/canceled/partial/缺 zip/未知状态、轮询超时保留 job、请求超时与 401、断点续跑和服务重启后重提、暂停中断、prepare_document 降级 MarkItDown、env provider 不被重置、markdown.md 优先。

cd apps/api
ruff check(CI 同款后端检查)           -> All checks passed!
python -m pytest -q                     -> 734 passed, 1 skipped
CI 中的 node 脚本测试(仓库根目录)     -> pass 28, fail 0

Add `mineru_provider="self_hosted"` for `mineru-kit api-server`
(MinerU >= 4.0). The client follows the documented V1 cycle: create
upload (with sha256 for dedup) -> PUT bytes -> complete -> submit a
parse job -> poll until a terminal state -> download the zip result via
`/v1/files/{file_id}/content`. It reuses the existing parser state for
pause/resume, resubmits when the service lost the job after a restart
(V1 job state is in-process only), and keeps the MarkItDown fallback.

Security: self-hosted services usually live on loopback or a LAN, so a
new `mineru_allow_private_base_url` (default false) is required before
the configured base URL may resolve to a private address. Result
downloads and authenticated uploads must be same-origin with the base
URL; cross-origin upload URLs never receive the API key and still pass
the existing public-host check.

Also:
- `_markdown_from_zip` recognizes `markdown.md` (MinerU 4.x zip layout)
- `mineru_tier` (optional; empty means the server default)
- only infer `mineru_provider` from a persisted base URL, so an env
  `SAG_MINERU_PROVIDER` is no longer reset to 302 at startup
- pass the new MinerU env vars through compose.yaml and document them

Relates to Zleap-AI#208
@kabishou11

Copy link
Copy Markdown
Contributor Author

CI 中「后端 · ruff + pytest」失败的是 tests/test_dsh_integration.py 里的两个用例(database is locked),看起来是已有的 SQLite 锁抖动,与本 PR 无关:

麻烦有权限的同学帮忙 re-run 一下失败的 job,谢谢!

The failing backend job hit database is locked in tests/test_dsh_integration.py, which looks like a pre-existing SQLite flake: the same failure appears in #207's CI (run 36970192660), and main (#199, run 36254889442) hit it in test_gaps.py. Locally (2 CPUs, 3 full runs each) this branch passed 3/3 while the same-base main failed once. Could a maintainer please re-run the failed job? Thanks!

@luoshuai990529 luoshuai990529 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已用最小范围修复此前的 P2,并重新复核最终提交 6800fa5。正常使用功能与 Spec 两条独立 code-review 均未发现阻塞问题,可以通过代码审批。

生产代码仅在 ModelConfigUpdate.mineru_provider 枚举中允许 self_hosted,解决环境启用自部署 MinerU 后,设置页完整表单保存或连接测试返回 422 的问题。另增加两条真实 API/数据库回归,验证保存仍保留 provider、tier、内网开关,以及连接测试使用草稿但不写入配置。原作者提交保留,未扩大解析功能范围。

@luoshuai990529
luoshuai990529 merged commit 52c9457 into Zleap-AI:main Oct 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants