Repository navigation
Commit de828c5
[fix](fe) Reserve 16 KB for a Paimon REST data token
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary: With `paimon.data-token.enabled=true` each cached Paimon table owns a
`RESTTokenFileIO`, and the vended token it holds is fetched on first data access, after the table
was weighed and admitted. The estimator reserved 4 KB for it, about twice the STS token seen in
practice (key id, secret, a 1,200-character security token and the endpoint). The token map is
whatever the REST server returns, so a server that vends a larger credential, or a refresh that
brings one, leaves the table cached below its retained size.
Paimon keeps one token per `RESTTokenFileIO` and replaces it on refresh, so the retained size is
bounded by the largest single token, but Paimon gives no hook to reweigh the table when the token
changes. Raise the fixed reserve to 16 KB, the allowance branch-4.x charges for a table's FileIO,
and test it with both a first fetch and a refresh to a security token of 8,192 characters.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PaimonCacheSizeEstimatorTest.restDataTokenFileIoIsChargedToItsTable` now checks a 1,200- and
an 8,192-character security token; with the old 4 KB reserve the second case fails (4,128
reserved, 9,232 retained). All 678 fe-connector-paimon tests pass locally.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 5b940b3 commit de828c5
2 files changed
Lines changed: 25 additions & 12 deletions
File tree
- fe/fe-connector/fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
81 | 83 | | |
82 | | - | |
| 84 | + | |
83 | 85 | | |
84 | 86 | | |
85 | 87 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
116 | 131 | | |
117 | 132 | | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
| 133 | + | |
123 | 134 | | |
124 | 135 | | |
125 | 136 | | |
| |||
0 commit comments