Skip to content

Comments

MPH domian matcher: Support building & using cache directly (instead of building from geosite.dat when Xray starts)#5505

Merged
RPRX merged 20 commits intoXTLS:mainfrom
hossinasaadi:serialize-matcher
Jan 31, 2026
Merged

MPH domian matcher: Support building & using cache directly (instead of building from geosite.dat when Xray starts)#5505
RPRX merged 20 commits intoXTLS:mainfrom
hossinasaadi:serialize-matcher

Conversation

@hossinasaadi
Copy link
Contributor

@hossinasaadi hossinasaadi commented Jan 8, 2026 •
edited
Loading

I've implemented serialization for MphMatcherGroup and the memory usage decreased.
Same rules, This PR : 22 MB

Domain matcher cache now works across routing, DNS, and hosts.
It also useful for routers or devices with limited memory, benchmarks are included below.

#5488 (comment)
@RPRX @iambabyninja @mangustyura

Build the cache file :

1- required rules to have ruleTag :

{  "dns" : {
   "hosts" : {
     "geosite:cn" : "2.2.2.2",
     "cloudflare-dns.com" : [
       ....
   "servers" : [
     {
       "tag" : "dns1", // important
       "address" : "8.8.8.8",
       "domains" : [
         "geosite:cn"
       ],
....
   "routing" : {
     "rules" : [
       {
         "domain" : [
           "geosite:cn"
         ],
         "outboundTag" : "direct",
         "ruleTag" : "rule-1"  // important
       }

2- Sample command for pre-built domain matcher :

./xray buildMphCache -c config.json -o test.cache

then set os.Setenv("XRAY_MPH_CACHE", "./domain.cache") or export XRAY_MPH_CACHE=./domain.cache.
Note: Your DNS servers and routing rules must have tags set, otherwise the cache won’t be used.

benchmark
Without MPHCache :

           158154752  maximum resident set size
           132203072  peak memory footprint

With MPHCache :

            70991872  maximum resident set size
            51872272  peak memory footprint

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 8, 2026

ready to test.

@RPRX
Copy link
Member

RPRX commented Jan 8, 2026

2- Cached file should be alongside the geofiles and file name should be matcher.cache

配置文件内 Routing 加个配置项来指定名称吧，同时也是开启该功能的标志，不然开多个 Xray 会炸，DNS 的你也看一下

@RPRX
Copy link
Member

RPRX commented Jan 8, 2026

@yiguodev libXray 或许需要更新 BuildDomainMatcherCache()

@RPRX
Copy link
Member

RPRX commented Jan 8, 2026

本来以为 mmap 是把磁盘空间当运行内存用结果它确实是懒加载，不知道会不会像 swap，不过仍然 https://t.me/projectXtls/1384

当然 Xray 启动时也不再需要 read、unmarshal geosite、build 出完整的 DomainMatcher 从而降低了 peak memory usage

有了这项优化，前面的 123 全被解决了，重新拾起 pre-map 也没必要了

@RPRX RPRX mentioned this pull request Jan 9, 2026
Merged
@iambabyninja
Copy link
Member

iambabyninja commented Jan 13, 2026

@RPRX, could you please advise which of the three PRs (
#5499
#5488
#5505
) should be tested on iOS?

We are ready to make a test build now.

@RPRX
Copy link
Member

RPRX commented Jan 13, 2026

@iambabyninja 要测试的是这个 PR，不过由于该伊朗贡献者目前失联了所以新版本暂时不会包括这个 PR

@BrewTestBot BrewTestBot mentioned this pull request Jan 13, 2026
Merged
1 task
@RPRX RPRX mentioned this pull request Jan 14, 2026
Merged
@yiguodev yiguodev mentioned this pull request Jan 17, 2026
Closed
@RPRX RPRX mentioned this pull request Jan 17, 2026
Closed
5 tasks
@hossinasaadi hossinasaadi mentioned this pull request Jan 25, 2026
Merged
@hossinasaadi hossinasaadi force-pushed the serialize-matcher branch 2 times, most recently from a7f19fe to be0263c Compare January 27, 2026 16:17
hossinasaadi
hossinasaadi commented Jan 27, 2026
View reviewed changes
@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 27, 2026

When use this PR (with XRAY_CACHED_MATCHER to prevent build Domains) on iOS memory decrease to 20MB (main branch 23-25MB).

last changes (based on mmap) 15.5MB at start!

@RPRX
Copy link
Member

RPRX commented Jan 27, 2026

我感觉这个 PR 还是可以有的，毕竟把前面的流程都跳过了，不过 mmap 对“降低跑代理时的总体内存占用”有帮助吗

@Meo597
Copy link
Collaborator

Meo597 commented Jan 27, 2026

mmap 对“降低跑代理时的总体内存占用”有帮助吗

没有

@RPRX
Copy link
Member

RPRX commented Jan 27, 2026

mmap 对“降低跑代理时的总体内存占用”有帮助吗

没有

那感觉实际意义不大，如果本来就已经不会因为 peak memory usage 而炸掉的话

所以 #5581 (comment) 是什么原因

@Meo597
Copy link
Collaborator

Meo597 commented Jan 27, 2026

因为 core 要全部加载完所有配置，再启动核心
这就导致 dns router 各有一份反序列化好的 domain/cidr 切片

他的 pr 不是单纯的 mmap，而是那个把 geosite 下放到 core 启动时再读
内存就节省在这块了

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 28, 2026

这就导致 dns router 各有一份反序列化好的 domain/cidr 切片

In my tests, I did not use GeoIP or GeoSite in DNS, so DNS memory usage is not involved.

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 31, 2026

@hossinasaadi 这个 PR 现在咋既有环境变量又有 config，感觉乱乱的

I will remove it.

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

能 config 的话就 config，把环境变量删了吧，dns 和 routing 内加个 config 就行了，其它的好像也没用到 geosite

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 31, 2026

能 config 的话就 config，把环境变量删了吧，dns 和 routing 内加个 config 就行了，其它的好像也没用到 geosite

oh , I removed config parameter not environment, also if we use env, so we can prevent the loadSite func to unmarshall the geo that increase memory in :

Xray-core/infra/conf/router.go

Lines 206 to 208 in 2c92339

func loadSite(file, code string) ([]*router.Domain, error) {
bs, err := loadFile(file, code)
if err != nil {

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

那就先环境变量吧等稳定了再改成 config

加个简单的 ./xray command 来调用构建缓存

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 31, 2026

那就先环境变量吧等稳定了再改成 config

加个简单的 ./xray command 来调用构建缓存

./xray buildCache -c config.json -o test.cache

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 31, 2026

那就先环境变量吧等稳定了再改成 config

I agree.

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

buildCache 改名 buildMphCache

xray.mph.path 改成 xray.mph.cache

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

文件也需要改名 buildmphcache.go

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

现在的原理还是 #5488 (comment) 吧？也就是说只有非 Windows 能用，虽然 @H1JK 说 Windows 上也能 mmap

@Fangliding
Copy link
Member

Fangliding commented Jan 31, 2026

没用mmap 就是把前缀树构建结果保存下来下次不用从geosite编译了

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

那没事了，啥时候有把硬盘映射到内存且不占用内存的技术再说吧

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 31, 2026

现在的原理还是 #5488 (comment) 吧？也就是说只有非 Windows 能用，虽然 @H1JK 说 Windows 上也能 mmap

The principle is still the same, but we no longer use mmap.
Because of that, it should work on Windows as well and shouldn’t have any issues.

@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

准备合并了，更新一下 #5505 (comment) 的示例（环境变量和 command），或者你直接更新到 PR 正文吧

@RPRX RPRX changed the title Routing: Serialize matcher for iOS MPH domian matcher: Support building/using cache directly (instead of building from geosite.dat everytime) Jan 31, 2026
@RPRX RPRX changed the title MPH domian matcher: Support building/using cache directly (instead of building from geosite.dat everytime) MPH domian matcher: Support building/using cache directly (instead of building from geosite.dat when Xray starts) Jan 31, 2026
@RPRX RPRX merged commit 01a4811 into XTLS:main Jan 31, 2026
39 checks passed
@RPRX RPRX changed the title MPH domian matcher: Support building/using cache directly (instead of building from geosite.dat when Xray starts) MPH domian matcher: Support building & using cache directly (instead of building from geosite.dat when Xray starts) Jan 31, 2026
RPRX pushed a commit that referenced this pull request Jan 31, 2026
@hossinasaadi @RPRX
MPH domian matcher: Support building & using cache directly (instead …
61e1153 
…of building from geosite.dat when Xray starts) (#5505)

Like #5488 (comment)
@RPRX
Copy link
Member

RPRX commented Jan 31, 2026

https://t.me/projectXray/4631555 这个是因为没用到 geosite 的 rule 也要写 tag 吗，如果要修的话就等修好再发版

@hossinasaadi
Copy link
Contributor Author

hossinasaadi commented Jan 31, 2026

https://t.me/projectXray/4631555 这个是因为没用到 geosite 的 rule 也要写 tag 吗，如果要修的话就等修好再发版

Of course, as I mentioned before, all routing rules and DNS entries must have tags, because all domain based rules will be cached, whether they come from geosite or not.

@hossinasaadi hossinasaadi deleted the serialize-matcher branch January 31, 2026 14:56
This was referenced Jan 31, 2026
Merged
Merged
@Fangliding Fangliding mentioned this pull request Feb 2, 2026
Merged
This was referenced Feb 4, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hossinasaadi @RPRX @iambabyninja @Meo597 @Fangliding @t-e-s-tweb