Repository navigation
feat(api): support self-hosted MinerU 4.x via the V1 API - #209
Conversation
Add `mineru_provider="self_hosted"` for `mineru-kit api-server`
(MinerU >= 4.0). The client follows the documented V1 cycle: create
upload (with sha256 for dedup) -> PUT bytes -> complete -> submit a
parse job -> poll until a terminal state -> download the zip result via
`/v1/files/{file_id}/content`. It reuses the existing parser state for
pause/resume, resubmits when the service lost the job after a restart
(V1 job state is in-process only), and keeps the MarkItDown fallback.
Security: self-hosted services usually live on loopback or a LAN, so a
new `mineru_allow_private_base_url` (default false) is required before
the configured base URL may resolve to a private address. Result
downloads and authenticated uploads must be same-origin with the base
URL; cross-origin upload URLs never receive the API key and still pass
the existing public-host check.
Also:
- `_markdown_from_zip` recognizes `markdown.md` (MinerU 4.x zip layout)
- `mineru_tier` (optional; empty means the server default)
- only infer `mineru_provider` from a persisted base URL, so an env
`SAG_MINERU_PROVIDER` is no longer reset to 302 at startup
- pass the new MinerU env vars through compose.yaml and document them
Relates to Zleap-AI#208
|
CI 中「后端 · ruff + pytest」失败的是
麻烦有权限的同学帮忙 re-run 一下失败的 job,谢谢! The failing backend job hit |
There was a problem hiding this comment.
已用最小范围修复此前的 P2,并重新复核最终提交 6800fa5。正常使用功能与 Spec 两条独立 code-review 均未发现阻塞问题,可以通过代码审批。
生产代码仅在 ModelConfigUpdate.mineru_provider 枚举中允许 self_hosted,解决环境启用自部署 MinerU 后,设置页完整表单保存或连接测试返回 422 的问题。另增加两条真实 API/数据库回归,验证保存仍保留 provider、tier、内网开关,以及连接测试使用草稿但不写入配置。原作者提交保留,未扩大解析功能范围。
Relates to #208(维护者回复欢迎先提交自部署 MinerU 的 PR)
背景 / Why
SAG 目前只能接 302 和 mineru.net 云端 MinerU:
_assert_public_host会拒绝 loopback/内网地址,无法对接本机或局域网部署的 MinerU。MinerU 4.0 把 HTTP 服务重写为 V1 API(/v1/uploads、/v1/parse/jobs、/v1/files,移除了/file_parse、/tasks),结果 zip 中的 Markdown 也改名为markdown.md。SAG can only use cloud MinerU today; self-hosted MinerU 4.x (V1 API) is blocked by the private-address check and the new protocol/zip layout.
修改 / What
mineru_provider="self_hosted"(parsing/mineru_v1.py),按 MinerU 4.0 官方docs/usage/http_api.md与scripts/http_api_example.sh实现 V1 流程:POST /v1/uploads(带 sha256,命中去重时跳过上传)→PUT upload_url→POST /v1/uploads/{id}/complete→POST /v1/parse/jobs(output_formats=["zip"]、ocr_mode、可选tier)→ 轮询GET /v1/parse/jobs/{id}至completed/partial/failed/canceled→GET /v1/files/{zip_file_id}/content下载 zip。file_id/job_id写入断点状态,暂停(should_pause)与续跑沿用原逻辑;V1 任务状态只在服务进程内,服务重启后 job 404 时自动重新上传提交一次。失败、超时、鉴权错误都映射为现有错误类型,因此继续走 MarkItDown 降级。mineru_allow_private_base_url(默认false)。只有显式开启后,用户配置的 Base URL 才允许解析到本机/内网;未开启时给出明确的配置提示。_markdown_from_zip同时识别markdown.md。mineru_tier(可选,留空用服务端默认档位);自部署服务可匿名,mineru_configured对self_hosted不再要求 API Key。settings_service._normalize_overrides:只在持久化配置里有 MinerU 地址时才推断服务商。此前即使从未保存过 MinerU 配置,启动时也会把SAG_MINERU_PROVIDER重置为302(official同样受影响)。compose.yaml、.env.example透传并说明SAG_MINERU_PROVIDER/SAG_MINERU_TIER/SAG_MINERU_ALLOW_PRIVATE_BASE_URL。验证 / How verified
真实 MinerU 4.0.10 端到端(本机
mineru-kit api-server --tier flash --api-key ...,CPU,SAG 客户端直连http://127.0.0.1:18000):服务端日志中的请求序列:
POST /v1/uploads 200 → PUT /v1/uploads/{id}/content 200 → POST .../complete 200 → POST /v1/parse/jobs 202 → GET /v1/parse/jobs/{id} 200 ×N → GET /v1/files/{id}/content 200;真实 zip 内容为markdown.md、middle_json.json、structured_content.json、model_output.json。单元测试:新增
tests/test_mineru_self_hosted.py(httpxMockTransport,响应结构按 4.0.10 实际返回编写),覆盖:完整流程与请求体、sha256 去重、匿名访问、内网地址开关(关闭拒绝 / 开启放行 / 公网无需开关)、下载重定向同源校验、跨源上传不带 Key 且内网跨源被拒、failed/canceled/partial/缺 zip/未知状态、轮询超时保留 job、请求超时与 401、断点续跑和服务重启后重提、暂停中断、prepare_document降级 MarkItDown、env provider 不被重置、markdown.md优先。