Skip to content

fix(admin): preserve HTTPS for upstream Swagger documents - #7895

Open
cloudbase-ai wants to merge 2 commits into
apache:masterfrom
cloudbase-ai:fix/swagger-upstream-protocol
Open

cloudbase-ai wants to merge 2 commits into
apache:masterfrom
cloudbase-ai:fix/swagger-upstream-protocol

Conversation

@cloudbase-ai

Copy link
Copy Markdown
Contributor

Fixes #7894.

An upstream configured as https:// with address example.com:8443 previously caused automatic Swagger loading to request http://example.com:8443/project/v3/api-docs. Preserve the transport scheme from discovery through the internal UpstreamInstance and document request, and use port 443 when HTTPS has no explicit port. Both startup scans and discovery callbacks use URI parsing instead of colon splitting. Existing callers retain HTTP by default.

Include the scheme in instance equality/hashCode and generated API document metadata. Manual Swagger import also sets the scheme on its virtual instance so imported HTTPS documents retain consistent metadata. This does not change its fetch or URL policy.

Validation on Java 17 with the Maven wrapper:

  • Before the fix, 8 of 14 request-path regression cases failed: both public loader entry points downgraded HTTPS and failed bracketed IPv6 parsing. The HTTP/default-protocol controls passed.
  • After the fix, all 33 focused tests pass. Coverage includes default HTTP, HTTPS with implicit/explicit ports, scheme values with/without ://, bracketed IPv6, document metadata, virtual import instances, equality and existing refresh/lock behavior. Request-path tests run real loader, pull service and HttpUtils code with an OkHttp application interceptor; they do not contact external hosts.
  • A separate local HttpsServer smoke test fetched and consumed the document through both startup scan and discovery UPDATE using the production HTTP client. The temporary localhost certificate was explicitly trusted in that test JVM; standard certificate and hostname verification remained enabled. No TLS policy was changed.
  • Admin and dependency reactor: 2,979 tests, 0 failures, 0 errors, 6 skipped. Checkstyle, Apache RAT and git diff --check pass.
./mvnw -B -ntp -pl shenyu-admin -am test apache-rat:check \
  -DskipRemoteResources=true -Djacoco.skip=true -Dmaven.javadoc.skip=true

PR #6456 also touches address parsing in LoadServiceDocEntryImpl as part of general IPv6 support; its changes will need reconciliation with the document-loader URI parsing here. This fix is separate from the manual-import DNS/port policy in #7868 and does not claim to reproduce #6362. Docker gateway E2E and live external HTTPS backends were not exercised for this Admin document-loading change.

  • Follow the contribution guidelines and DCO sign-off.
  • Add regression coverage and run the affected reactor checks described above.
  • Full-repository clean install (validation here is the Admin dependency reactor).

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The transport scheme was indeed lost twice — once by split(":") in the two LoadServiceDocEntryImpl conversion paths, then again by the hardcoded scheme("http") in PullSwaggerDocServiceImpl and the hardcoded "http://" protocol in DocManagerImpl#buildExtJson.

What I verified:

  • UriUtils.appendScheme only prepends when the url does not already start with http:///https://, so a discovery entry that already carries a scheme keeps it and getProtocol() is correctly ignored in that case.
  • UriUtils.getActualPort returns 443 for https/wss and 80 for http/ws when the port is absent, which is what makes https://example.com resolve to https://example.com:443 rather than https://example.com:80.
  • URI.getHost() keeps the brackets on IPv6 literals, so https://[::1]:8443 round-trips through UriComponentsBuilder.host(...) correctly. The previous split(":") produced [ as the host, so this is a fix rather than a behaviour change to preserve.
  • SwaggerUpstreamProtocolTest swaps the OkHttpClient inside the shared HttpUtils and asserts the recorded request URL, so it proves the URL that actually goes over the wire for both the startup scan and the discovery callback, across https://, bare https, http://, empty, and IPv6 forms. That is the right seam to assert at.

Non-blocking observations:

  1. buildExtJson now writes https:// into the selector's protocol ext for HTTPS upstreams. That is the point of the change, but it also means the gateway will start using TLS for upstreams that previously received plaintext requests on an https://-advertised discovery entry (a common shape behind a TLS-terminating sidecar or service mesh). Existing rows keep their old ext_json until the next scan rewrites them, so this needs a line in the release/upgrade notes rather than only a code comment.
  2. getScheme() returns the raw field. buildExtJson concatenates it unconditionally, so a null scheme would emit "protocol": "null://". The field initialiser covers the normal path, but UpstreamInstance also looks like a DTO that can be populated from ext_json, so StringUtils.defaultIfBlank(scheme, "http") in the getter would make it impossible to produce that string.
  3. URI.create(...) throws on inputs the old split(":") silently tolerated (example.com:8080:extra, stray spaces, a scheme other than http/https). Both call sites are inside a single try around the whole stream, so one malformed upstream still discards every instance for that selector. The change widens the set of inputs that can trigger that. A per-item try/catch inside the map (log and skip) would keep one bad row from hiding the healthy ones.
  4. Adding scheme to equals/hashCode changes instance identity: http://host:8443 and https://host:8443 used to collapse into one entry and now do not. That is correct, but it also means every HTTPS upstream re-registers its doc metadata once after this lands. Worth mentioning in the PR description so it is not mistaken for a regression.
  5. SwaggerUpstreamProtocolTest mutates the static PullSwaggerDocServiceImpl.HTTP_UTILS singleton and restores it in finally. That is safe today because admin tests are not parallel, but it is a global write from a test. A constructor-injected HttpUtils (or a package-private setter) would remove the landmine and make the test independent of reflection on a private static field.
  6. The @CsvSource rows duplicate the same seven cases for the scan and the event path. If a case is added later it has to be added twice; a single source feeding both entry points, or a @MethodSource, would keep them in lockstep.

@Aias00

Aias00 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

pr_build failed on 42242891, but the failure is not caused by this PR.

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 33.221 s
  <<< FAILURE! - in org.apache.shenyu.admin.discovery.ZookeeperClientDiscoveryIntegrationTest
org.awaitility.core.ConditionTimeoutException: Assertion condition defined as
  ... expected: <[127.0.0.1:19195]> but was: <[]> within 15 seconds.
  at ZookeeperClientDiscoveryIntegrationTest.assertSyncSnapshot(ZookeeperClientDiscoveryIntegrationTest.java:251)
  at ZookeeperClientDiscoveryIntegrationTest.testClientRegistrationAndInstanceChanges(...:162)

That test drives embedded ZooKeeper registration/discovery sync through AbstractSpringIntegrationTest; this PR only touches the Swagger document import path (UpstreamInstance, SwaggerImportServiceImpl, DocManagerImpl, LoadServiceDocEntryImpl, PullSwaggerDocServiceImpl) and never reaches DiscoveryDataChangedEventSyncListener or the ZK repository. The assertion never observed the first instance at all (was: <[]>), and the test burned 27 s of its 33 s before the 15 s Awaitility window expired — this is the known flakiness of that integration test on a loaded runner, not a regression here.

Could you please push an empty commit (or re-run the failed job once the run finishes) so pr_build gets a clean pass? The code review itself is approved — nothing to change on the diff. Thanks!

@Aias00

Aias00 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Follow-up on my earlier note: the failure has changed to the shared CachePluginTest case, which is still unrelated to this PR.

build (shenyu-integrated-test-http) fails in CachePluginTest, and it is not caused by this PR:

[ERROR] CachePluginTest.testMemoryModle:54 expected: <true> but was: <false>
[ERROR] CachePluginTest.testRedisModle:73  expected: <true> but was: <false>
[ERROR] Tests run: 103, Failures: 2, Errors: 0, Skipped: 5

The same two cases are red on every other PR that touches this module right now, so this is the shared e2e case rather than your change. build is only an aggregate gate, so a single shared case turns the whole check red.

Could you keep this rebased and re-run once the http e2e case is fixed upstream? Nothing else blocks my approval.

@yuluo-yx

Copy link
Copy Markdown
Member

/retest

@Aias00

Aias00 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

@cloudbase-ai This PR is approved, but 2 checks are red: build (shenyu-integrated-test-http), build (shenyu-integrated-test-combination).

I inspected these lanes today and they fail for infrastructure reasons rather than on the diff — the integrated-test jobs die while starting their containers (Elasticsearch /_cat/health returning 000 until the job is cancelled), and the http lane also hits the known CachePluginTest.testMemoryModle / testRedisModle flake. The same jobs are red on master and on unrelated PRs right now.

Please rebase onto master and push (or ask for a re-run) so the checks execute again. If a case is still failing on a fresh run, paste the test name and I'll look properly.

@Aias00

Aias00 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Hi @cloudbase-ai, this PR is already approved, but it currently cannot be merged. Here is the blocker and what is needed:

Blocker
This PR is approved but blocked by build (shenyu-integrated-test-http) -> CachePluginTest.testRedisModle / testMemoryModle (AssertionFailedError: expected: <true> but was: <false> at CachePluginTest.java:73). The identical failure reproduces on unrelated PRs (#7888, #7889), so it is a systemic/flaky cache-integration test and not caused by your HTTPS/Swagger change.

What we need
Please rebase onto current master and re-run; if CachePluginTest still fails in isolation, we will treat it as an infra issue.

Once the branch is up to date and CI is green, we will squash-merge it. Thanks!

@Aias00

Aias00 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Root cause found for the build (shenyu-integrated-test-http) failure, and it is not in your branch — no code change needed on your side.

The job fails on:

CachePluginTest.testMemoryModle:54 expected: <true> but was: <false>
CachePluginTest.testRedisModle:73  expected: <true> but was: <false>

I traced this across three unrelated PRs (#7917, #7895, #7888) and all three fail on exactly the same two assertions, so it is not specific to your change. CachePluginTest.testPass() fetches /http/test/cache twice with a 2s gap and asserts the two response bodies match; the rule handle sets timeoutSeconds=3000, so a cache hit is expected. The logs show every response carrying a fresh timestamp ("msg":"Fri Oct 23 13:07:54 UTC 2026", then 13:07:55, then 13:07:56, then 13:07:58) — the cache never serves the stored entry. The test itself has not changed since 2022, so this is a cache-plugin regression introduced by the 10-23 / 10-24 cache work (#7969 coalesce concurrent misses, then #8079 Vary replay).

It is already fixed on master. The same job passes on newer heads:

Your branch is behind master (this PR: 58 commits), which is why it still carries the broken cache code.

What to do: rebase onto current master and let CI re-run. That should clear this job without touching your diff. If it still fails after the rebase, the regression is genuinely yours and I will dig in again.


Additional note for this PR: the e2e and e2e-case (...) failures are a separate infrastructure issue, not the cache bug. The log ends with:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Automatic Swagger document loading downgrades HTTPS upstreams to HTTP

3 participants