Skip to content

[fix](regression) Mirror HTTP TVF fixtures in regression OSS - #66013

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
hello-stephen:agent/branch41-http-tvf-hf
Jul 24, 2026
Merged

yiguolei merged 1 commit into
apache:branch-4.1from
hello-stephen:agent/branch41-http-tvf-hf

Conversation

@hello-stephen

Copy link
Copy Markdown
Contributor

What

Replace the mutable/public HuggingFace URLs in external_table_p0/tvf/test_http_tvf with stable public mirrors in the Doris regression OSS bucket.

Why

The branch-4.1 S3 P2 validation reached the HuggingFace section and failed with java.net.SocketTimeoutException: Connect timed out. The test data should not depend on HuggingFace network availability or mutable dataset branches.

The prompts fixture is pinned to source revision 61229c8c07ea12c6e219b2b2b728f23f2842901b, which preserves the expected 204 records. The IMDb JSON and Parquet fixtures are mirrored byte-for-byte. Existing golden output is unchanged.

Jira: DORIS-27400

Validation

  • git diff --check
  • Groovy 4.0.19 compilation
  • Suite load/discovery: test_http_tvf, tag p2
  • Public OSS GET and byte-range GET for all four fixtures
  • Local/remote MD5 matches for all fixtures
  • Prompts CSV logical record count: 204

This is a P2 case-only change; after merge it still requires an exact post-merge rerun of external_table_p0/tvf/test_http_tvf.

@hello-stephen

Copy link
Copy Markdown
Contributor Author

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor Author

[doris-27400-static-gate]

This is a P2 case-only fix. It does not touch the regression framework, shared utilities, dependencies, or build paths.

Validated before requesting the CI skip:

  • git diff --check
  • Groovy 4.0.19 compilation
  • suite load/discovery: test_http_tvf, tag p2
  • all four mirrored objects are publicly readable and support byte-range GET (206)
  • local/remote MD5 matches for every fixture
  • pinned prompts snapshot contains exactly the expected 204 logical records
  • existing golden output is unchanged

Community buildall does not execute this P2 case. The required proof remains an exact post-merge branch-4.1 rerun of external_table_p0/tvf/test_http_tvf.

@hello-stephen

Copy link
Copy Markdown
Contributor Author

skip buildall

@hello-stephen
hello-stephen marked this pull request as ready for review July 24, 2026 10:41
@hello-stephen
hello-stephen requested a review from yiguolei as a code owner July 24, 2026 10:41
@hello-stephen

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30087052068

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@yiguolei
yiguolei merged commit 1ac74c8 into apache:branch-4.1 Jul 24, 2026
32 of 35 checks passed
yiguolei pushed a commit that referenced this pull request Jul 30, 2026
## Proposed changes

Follow-up for DORIS-27400 after #66013 replaced unstable HuggingFace
URLs with immutable OSS fixtures.

Post-merge exact S3 P2 build #204256 reached the mirrored `prompts.csv`,
proving that the original network timeout is gone, but `qt_sql15` failed
with expected 204 / actual 205. The mirrored CSV is byte-identical to
the pinned dataset revision we captured and contains one header record
plus 204 data records. Direct HTTP CSV TVF reads the header as data,
while the previous HuggingFace path exposed the 204 dataset rows.

This case-only follow-up adds `skip_lines = 1` to the three
`prompts.csv` HTTP TVF calls (`qt_sql15`, `qt_sql16`, and `qt_sql17`) so
the mirrored path preserves the existing 204-row semantics and schema
assertion.

## Validation

- #204256 terminal first error: `sql15` expected 204, actual 205; the
OSS URL was reachable and the target executed
- Mirrored CSV: 205 physical CSV records = 1 header + 204 data records
- `git diff --check`: passed
- Groovy 4.0.19 compilation: passed
- Suite discover/load identity: exactly one `test_http_tvf`, group `p2`;
no framework/tool/build changes

A post-merge exact `external_table_p0/tvf/test_http_tvf` run is still
required before closing DORIS-27400.
morningman added a commit that referenced this pull request Nov 5, 2026
…loopback hosts (#70232)

### Background

The `http()` table-valued function reads CSV, JSON, Parquet, and ORC
files from HTTP(S) URLs; for an `hf://` URI the FE expands the
HuggingFace path into download URLs.
`external_table_p0/tvf/test_http_tvf` (group `p0`, so every P0 run
executes it) serves most of its fixtures from an HTTP server it starts
inside the regression runner, and builds their URLs from `httpTvfHost`,
else `externalEnvIp`, which is `127.0.0.1` in every pipeline config. Its
last seven queries read public HuggingFace datasets through `hf://`.

### What problem does this PR solve?

Issue Number: None
Related PR: #65877, #66013, #66027 (branch-4.1 fixes by @hello-stephen,
ported here), #65299 (added the in-runner HTTP server and its host
lookup), #60915 (moved the suite into P0 and relaxed sql15 / sql16 to
`count > 0`)
PR Type: Test

Problem Summary:

On master the suite still depends on huggingface.co and on HuggingFace's
mutable `main` branches, and it advertises the runner as `127.0.0.1`,
which a BE on another host resolves to itself. This PR ports the three
branch-4.1 fixes: a loopback or wildcard `externalEnvIp` falls back to
the runner's host address (an explicit `httpTvfHost` is still used as
given), the seven `hf://` reads use byte-identical mirrors in the
regression OSS bucket pinned to a dataset revision, and the mirrored
`prompts.csv` skips its header line. With the mirror pinned, sql15 /
sql16 assert the exact count 204 again.

### Details

- **Failure:** None on master CI today: P0 passed 66 of 66 runs on
2026-11-04, though the HuggingFace queries took about 146 of the suite's
149 seconds (build 1092587). On branch-4.1 the suite failed with
`Connection refused` on `http://127.0.0.1:<port>` with remote BEs
(#65877), with `java.net.SocketTimeoutException: Connect timed out` to
HuggingFace (#66013), and then with sql15 getting 205 instead of 204 on
the mirror (#66027).
- **Cause:** The suite assumed that every BE runs on the runner's host,
true in P0 only, and that huggingface.co is reachable with fixed `main`
branches: `prompts.csv` on `fka/awesome-chatgpt-prompts` `main` now has
five columns and is over six times the 204-record file the golden came
from. The mirrored CSV keeps a header line that `hf://` hid, hence
`skip_lines = 1`.
- **Stability evidence:** No query reaches huggingface.co any more. The
only remote host left is
`doris-regression-hk.oss-cn-hongkong.aliyuncs.com` (public reads), which
P0 already uses in `query_p0/test_parquet_dict` (28 of 28 recent runs
passed). Since master's P0 does not fail today, there is no
before-and-after failure rate.

### Testing

- `-forceGenOut` run of the suite on a local 1 FE + 1 BE cluster from
master `4530f761fa1` (the FE carries local patches that don't touch the
HTTP TVF), with P0's `externalEnvIp = 127.0.0.1` and no `httpTvfHost`:
the runner advertised `http://192.168.1.164:<port>`, and the 22 blocks
generated for sql01 to sql18 and both `replaced_*` checks are
byte-identical to master's `.out`, with sql15 / sql16 now after
`replaced_after`, where they run. sql19 then hit the 900 s query
timeout: this machine's BE reached the 20 MB Parquet mirror through a
local proxy at about 10 KB/s (a direct GET from the same machine runs at
about 600 KB/s).
- The suite without sql19 to sql21, in compare mode against the
committed `.out`: passed. With `skip_lines` removed from sql15 it fails
with `Expect cell is: 204 But real is: 205`.
- sql19 to sql21 did not run locally. Each mirrored Parquet file has the
same size as on HuggingFace and a byte-identical footer (schema, row
groups, statistics), and each footer's minimum `text` is the first row
the `.out` expects. This PR's P0 (build 1092831) then ran the whole
suite against the mirror: it passed in 8.9 s, with sql19, sql20 and the
two sql21 queries taking 1.3 to 1.8 s each, and the runner advertised
`http://172.16.0.146:<port>`; the 39 P0 runs before it, with the
HuggingFace version, took 149 to 170 s.

### Notes for Reviewers

- Apart from a one-line comment on the loopback check, the suite now
matches branch-4.2's except for master's own parts: the `p0` label
(#60915) and the `mutable/` cases (#68847).
- `hf://` loses its only end-to-end coverage (real HuggingFace URLs,
tree-API glob expansion, `@revision`); `HFUtilsTest` still covers
parsing, URL conversion, glob matching, and one live `*` expansion
against the HuggingFace tree API.
- sql16, sql20, and the second sql21 now repeat sql15 / sql19, since
plain HTTP has no globbing; they stay so that the goldens and branch-4.x
line up.
- In P0 the runner now advertises its host address instead of
`127.0.0.1`; both work there, but branch-4.x labels the suite `p2`, so
P0 runs this fallback for the first time.
- `ai_p0/test_ai_profile` copied the old host lookup and has the same
latent loopback problem; it is not changed here.

Co-authored-by: Dongyang Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants