Repository navigation
[fix](regression) Mirror HTTP TVF fixtures in regression OSS - #66013
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
[doris-27400-static-gate] This is a P2 case-only fix. It does not touch the regression framework, shared utilities, dependencies, or build paths. Validated before requesting the CI skip:
Community buildall does not execute this P2 case. The required proof remains an exact post-merge branch-4.1 rerun of |
|
skip buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
## Proposed changes Follow-up for DORIS-27400 after #66013 replaced unstable HuggingFace URLs with immutable OSS fixtures. Post-merge exact S3 P2 build #204256 reached the mirrored `prompts.csv`, proving that the original network timeout is gone, but `qt_sql15` failed with expected 204 / actual 205. The mirrored CSV is byte-identical to the pinned dataset revision we captured and contains one header record plus 204 data records. Direct HTTP CSV TVF reads the header as data, while the previous HuggingFace path exposed the 204 dataset rows. This case-only follow-up adds `skip_lines = 1` to the three `prompts.csv` HTTP TVF calls (`qt_sql15`, `qt_sql16`, and `qt_sql17`) so the mirrored path preserves the existing 204-row semantics and schema assertion. ## Validation - #204256 terminal first error: `sql15` expected 204, actual 205; the OSS URL was reachable and the target executed - Mirrored CSV: 205 physical CSV records = 1 header + 204 data records - `git diff --check`: passed - Groovy 4.0.19 compilation: passed - Suite discover/load identity: exactly one `test_http_tvf`, group `p2`; no framework/tool/build changes A post-merge exact `external_table_p0/tvf/test_http_tvf` run is still required before closing DORIS-27400.
…loopback hosts (#70232) ### Background The `http()` table-valued function reads CSV, JSON, Parquet, and ORC files from HTTP(S) URLs; for an `hf://` URI the FE expands the HuggingFace path into download URLs. `external_table_p0/tvf/test_http_tvf` (group `p0`, so every P0 run executes it) serves most of its fixtures from an HTTP server it starts inside the regression runner, and builds their URLs from `httpTvfHost`, else `externalEnvIp`, which is `127.0.0.1` in every pipeline config. Its last seven queries read public HuggingFace datasets through `hf://`. ### What problem does this PR solve? Issue Number: None Related PR: #65877, #66013, #66027 (branch-4.1 fixes by @hello-stephen, ported here), #65299 (added the in-runner HTTP server and its host lookup), #60915 (moved the suite into P0 and relaxed sql15 / sql16 to `count > 0`) PR Type: Test Problem Summary: On master the suite still depends on huggingface.co and on HuggingFace's mutable `main` branches, and it advertises the runner as `127.0.0.1`, which a BE on another host resolves to itself. This PR ports the three branch-4.1 fixes: a loopback or wildcard `externalEnvIp` falls back to the runner's host address (an explicit `httpTvfHost` is still used as given), the seven `hf://` reads use byte-identical mirrors in the regression OSS bucket pinned to a dataset revision, and the mirrored `prompts.csv` skips its header line. With the mirror pinned, sql15 / sql16 assert the exact count 204 again. ### Details - **Failure:** None on master CI today: P0 passed 66 of 66 runs on 2026-11-04, though the HuggingFace queries took about 146 of the suite's 149 seconds (build 1092587). On branch-4.1 the suite failed with `Connection refused` on `http://127.0.0.1:<port>` with remote BEs (#65877), with `java.net.SocketTimeoutException: Connect timed out` to HuggingFace (#66013), and then with sql15 getting 205 instead of 204 on the mirror (#66027). - **Cause:** The suite assumed that every BE runs on the runner's host, true in P0 only, and that huggingface.co is reachable with fixed `main` branches: `prompts.csv` on `fka/awesome-chatgpt-prompts` `main` now has five columns and is over six times the 204-record file the golden came from. The mirrored CSV keeps a header line that `hf://` hid, hence `skip_lines = 1`. - **Stability evidence:** No query reaches huggingface.co any more. The only remote host left is `doris-regression-hk.oss-cn-hongkong.aliyuncs.com` (public reads), which P0 already uses in `query_p0/test_parquet_dict` (28 of 28 recent runs passed). Since master's P0 does not fail today, there is no before-and-after failure rate. ### Testing - `-forceGenOut` run of the suite on a local 1 FE + 1 BE cluster from master `4530f761fa1` (the FE carries local patches that don't touch the HTTP TVF), with P0's `externalEnvIp = 127.0.0.1` and no `httpTvfHost`: the runner advertised `http://192.168.1.164:<port>`, and the 22 blocks generated for sql01 to sql18 and both `replaced_*` checks are byte-identical to master's `.out`, with sql15 / sql16 now after `replaced_after`, where they run. sql19 then hit the 900 s query timeout: this machine's BE reached the 20 MB Parquet mirror through a local proxy at about 10 KB/s (a direct GET from the same machine runs at about 600 KB/s). - The suite without sql19 to sql21, in compare mode against the committed `.out`: passed. With `skip_lines` removed from sql15 it fails with `Expect cell is: 204 But real is: 205`. - sql19 to sql21 did not run locally. Each mirrored Parquet file has the same size as on HuggingFace and a byte-identical footer (schema, row groups, statistics), and each footer's minimum `text` is the first row the `.out` expects. This PR's P0 (build 1092831) then ran the whole suite against the mirror: it passed in 8.9 s, with sql19, sql20 and the two sql21 queries taking 1.3 to 1.8 s each, and the runner advertised `http://172.16.0.146:<port>`; the 39 P0 runs before it, with the HuggingFace version, took 149 to 170 s. ### Notes for Reviewers - Apart from a one-line comment on the loopback check, the suite now matches branch-4.2's except for master's own parts: the `p0` label (#60915) and the `mutable/` cases (#68847). - `hf://` loses its only end-to-end coverage (real HuggingFace URLs, tree-API glob expansion, `@revision`); `HFUtilsTest` still covers parsing, URL conversion, glob matching, and one live `*` expansion against the HuggingFace tree API. - sql16, sql20, and the second sql21 now repeat sql15 / sql19, since plain HTTP has no globbing; they stay so that the goldens and branch-4.x line up. - In P0 the runner now advertises its host address instead of `127.0.0.1`; both work there, but branch-4.x labels the suite `p2`, so P0 runs this fallback for the first time. - `ai_p0/test_ai_profile` copied the old host lookup and has the same latent loopback problem; it is not changed here. Co-authored-by: Dongyang Li <[email protected]>
What
Replace the mutable/public HuggingFace URLs in
external_table_p0/tvf/test_http_tvfwith stable public mirrors in the Doris regression OSS bucket.Why
The branch-4.1 S3 P2 validation reached the HuggingFace section and failed with
java.net.SocketTimeoutException: Connect timed out. The test data should not depend on HuggingFace network availability or mutable dataset branches.The prompts fixture is pinned to source revision
61229c8c07ea12c6e219b2b2b728f23f2842901b, which preserves the expected 204 records. The IMDb JSON and Parquet fixtures are mirrored byte-for-byte. Existing golden output is unchanged.Jira: DORIS-27400
Validation
git diff --checktest_http_tvf, tagp2This is a P2 case-only change; after merge it still requires an exact post-merge rerun of
external_table_p0/tvf/test_http_tvf.