Skip to content

fix: share MySQL/OceanBase initialization and reconcile fresh seeds - #7529

Merged
Aias00 merged 4 commits into
apache:masterfrom
yangjj-iso:codex/7521-oceanbase-shared-init
Oct 11, 2026
Merged

Aias00 merged 4 commits into
apache:masterfrom
yangjj-iso:codex/7521-oceanbase-shared-init

Conversation

@yangjj-iso

@yangjj-iso yangjj-iso commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7521

OceanBase fresh installations in MySQL mode now use the canonical db/init/mysql/schema.sql. Remove the duplicate full schema and update the native runner, Agent Gateway migration checker and packaged initialization guide.

This also changes fresh MySQL installations: remove 50 permission seeds referencing absent resources, fix loggingKafka's bootstrapServer field, and align its default-namespace configuration with the plugin configuration. All 447 effective MySQL role/resource grants are retained, and every effective grant in the former OceanBase seed is present in the shared schema. Existing databases continue to use the separate version-specific upgrade scripts.

Scope and reconciliation

  • Retain MySQL scaling tables/permissions, the MCP selector, namespace/lookup indexes and common unique constraints. Former OceanBase lookup indexes have shared equivalents with the same leading columns; differing API-key permission IDs represent the same role/resource grants.
  • Use current aiProxy fallback fields, aiPrompt handles and the supported contextPath token-limiter dictionary. Explain why obsolete aiProxy prompt handles and the unsupported default dictionary entry are not carried forward.
  • Record the H2 dictionary difference in the initialization guide: H2 still seeds DEFAULT_KEY_RESOLVER (default), and AiTokenLimiterEnum.getByName falls back to CONTEXT_PATH. H2 alignment remains a separate follow-up.
  • Preserve the OceanBase profile, JDBC driver, native CI job, immutable released upgrade-test baseline and all historical upgrade SQL.
  • Rebase onto master ad38688, preserving upstream SOFA/TARS seed cleanup. Keep the duplicate OceanBase initialization schema deleted.
  • Set the OceanBase test container's file-descriptor limit explicitly, without changing host or daemon limits.

Test fixture repairs after rebase

A separate test-only commit repairs Admin fixtures inherited from master: supply the event publisher mock, set the discovery namespace in the listener-update test, and let LoggerFactory static mocks call real methods for unstubbed loggers. This prevents JsonUtils from permanently initializing a null logger. Production Java behavior and existing assertions are unchanged; no test ordering workaround is needed.

CI diagnosis and workflow repair

The failed SDK HTTP job at e742dc5 reported every one of the 21 integrated-test reactor modules as SKIPPED under mvnd, then returned BUILD SUCCESS in 0.036 seconds. The gateway image was therefore absent locally. Compose tried to pull apache/shenyu-integrated-test-sdk-http:latest and failed with pull access denied. Health checks and tests were skipped; successful preceding build steps are not evidence that this test passed.

The workflow, integrated-test POMs and SDK HTTP Compose file at the failed head match master ad386882. An earlier master SDK HTTP job built its modules and ran four tests successfully. The daemon's internal reason for the all-skipped reactor has not been isolated; the failed CI log does not show a Maven Central or Docker Hub timeout.

The separate CI repair at 342955c0 uses the Maven wrapper for the selected integrated-test module and its reactor dependencies, skips test execution during image construction, verifies the local gateway image before Compose, and uses the wrapper with failIfNoTests=true for the actual test step. No SQL changes are made by this repair.

YAML/shell syntax checks, the CI routing regression suite in Linux, root Apache RAT and git diff --check passed. The local wrapper run compiled and packaged SDK HTTP and invoked Docker image construction, then stopped on an explicit local Docker base-image pull timeout for eclipse-temurin:17-centos7. Local Compose and the full SDK HTTP suite were not completed. The updated Linux SDK HTTP job passed at this head: integrated-image build, image inspection, Compose startup, health checks, test execution and result checking all completed successfully.

Validation of SQL and fixture changes at e742dc5

  • Main CI passed, including pr_build / Build with Maven, the aggregate build check, Agent Gateway migration consistency, license headers and Apache RAT.
  • native-sql-matrix passed: all five amd64 engines (MySQL, OceanBase, PostgreSQL, openGauss and Oracle), fresh-install/upgrade flows, runner tests, list contracts and the sql-matrix aggregate.
  • Shared MySQL/OceanBase fresh checks cover tables, indexes, unique constraints, orphan relations, duplicate natural keys, admin grants, namespace defaults and loggingKafka/aiProxy configs. All five engines also check mapper projections, binary JAR preservation, retained upgrade row IDs and index catalogs.
  • Local ./mvnw -B -ntp -pl shenyu-admin -am test -Prelease -DskipRemoteResources=true -Djacoco.skip=true passed all 34 modules: 2,586 tests, zero failures/errors and two existing skips. Admin accounts for 1,775 tests and one skip. Checkstyle passed.
  • Local Python checks passed: 23 native-matrix regression tests, 10 Agent Gateway migration regression tests, and 24 matching Agent Gateway seed rows for each of the five dialects.
  • Static reconciliation against master confirms all 50 removed MySQL permission rows are orphans, no effective MySQL/OceanBase grant is lost, and historical upgrade SQL is unchanged.
  • git diff --check passed.

The original pre-rebase full 213-module Maven reactor validation (5,122 tests, zero failures/errors, eight existing skips) remains historical evidence; current-head checks are listed separately above.

Contribution checklist

  • Read the contribution guidelines.
  • Include regression/native integration checks.
  • Document Maven validation, including the separately repaired inherited test fixture failures.

@yangjj-iso yangjj-iso changed the title fix/7521 oceanbase shared init fix: reuse MySQL initialization SQL for OceanBase Oct 10, 2026

@wy471x wy471x left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed locally at 05ae928, checked against the merge base with master (76122a4). I verified the seed reconciliation statically rather than by re-running the container matrix:

  • CI scripts: python3 .github/scripts/test-native-sql-matrix.py (23 tests OK) and .github/scripts/test-agent-gateway-migrations.py (10 tests OK) both pass on this head.
  • Permission cleanup: all 50 removed permission rows in db/init/mysql/schema.sql reference a resource_id that does not exist in the same file's resource table (the base file had exactly 50 such orphans, the head has 0). Comparing the deleted db/init/ob/schema.sql against the reconciled MySQL seed: every OceanBase-only grant either duplicates a (role, resource) pair that already exists under a different permission id, or points at a resource that is absent from both files. So pointing OceanBase at the shared seed does not drop an effective grant.
  • The two dropped plugin_handle rows (plugin 50, field prompt) are not consumed anywhere: AiProxyHandle has no prompt field, and MySQL/openGauss never shipped those rows.

Two things I would ask for before merge:

  1. Please rebase. git merge-tree --write-tree gh/master <head> reports a modify/delete conflict on db/init/ob/schema.sql: master has since removed 68 orphan permission rows from that same file (the same class of cleanup). The conflict is mechanical, but the rebase needs to keep the check scripts' expectations consistent with master's ob/og seeds.
  2. State the scope. This PR edits the canonical db/init/mysql/schema.sql, i.e. it changes what a fresh MySQL install gets (50 permission rows removed, plus the loggingKafka plugin row and a namespace_plugin_rel row). The title/description frame the change as an OceanBase fix, and that MySQL-visible part is easy to miss in a 2.5k-line deletion diff. Please call it out explicitly, or split it.

Also verified while diffing the seeds — shenyu_dict: the OceanBase-only DEFAULT_KEY_RESOLVER row (id 1679002911061737490, dict_name default) is not carried forward, and MySQL keeps CONTEXT_PATH_KEY_RESOLVER under that same id. Since AiTokenLimiterEnum has no default value and getByName falls back to CONTEXT_PATH, behaviour is unchanged and the guide's "unsupported default dictionary entry" note is accurate. But the H2 schema still ships DEFAULT_KEY_RESOLVER and no CONTEXT_PATH_KEY_RESOLVER, so all three schema families now disagree; a follow-up issue (or one line in the initialization guide) would keep that from being rediscovered later.

The added native-matrix checks (column/nullability, PK/index/uniqueness, orphan relations, duplicate natural keys, invalid JSON, grants and namespace defaults) are a real improvement — that is the right place to catch this class of seed regression.

@github-actions
github-actions Bot force-pushed the codex/7521-oceanbase-shared-init branch from 05ae928 to 5e876de Compare October 11, 2026 02:13
@yangjj-iso yangjj-iso changed the title fix: reuse MySQL initialization SQL for OceanBase fix: share MySQL/OceanBase initialization and reconcile fresh seeds Oct 11, 2026

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: share MySQL/OceanBase initialization and reconcile fresh seeds

This is a well-scoped, thoroughly validated consolidation. The design is sound and the CI evidence is comprehensive.

What is correct

  • Shared fresh-install source, immutable upgrade baseline. Fresh OceanBase now uses db/init/mysql/schema.sql (native-sql-matrix.py:43), while the upgrade flow still reads the historical db/init/ob/schema.sql via git show {BASELINE}:... (schema_source() at native-sql-matrix.py:51-56). Deleting the duplicate full OB schema from the tree is therefore safe — the released baseline is pinned by commit and only consumed for upgrade compatibility. This is a clean way to avoid drift between the two copies.
  • Fresh-seed contract is now machine-checked. check_fresh_mysql_schema() (native-sql-matrix.py:77-140) validates tables, secondary indexes, unique constraints, referential integrity (orphan permission/resource/user_role/namespace_*_rel), duplicate natural keys, admin resource permission, default namespace plugin, and the loggingKafka bootstrapServer value. test-native-sql-matrix.py adds positive/negative coverage for both the shared-source routing and the validator. Good defense-in-depth.
  • mysql/schema.sql edits match the stated reconciliation. 50 permission rows referencing absent resource_ids are removed (verified as orphans by the new validator's permission.resource_id → resource check); loggingKafka (plugin id 33) is corrected from the non-existent namesrvAddr to bootstrapServer, and its namespace_plugin_rel (id 1801816010882822171) now carries the full config aligned with the plugin row — both asserted by check_fresh_mysql_schema. All 447 effective grants are retained.
  • CI repair is targeted and correct. integrated-test.yml drops the mvnd path (which silently skipped the entire subreactor and reported BUILD SUCCESS in 0.036s) in favor of the Maven wrapper with -Dmaven.test.skip=true for image construction, a docker image inspect gate before compose, and -DfailIfNoTests=true for the actual run. native-sql-matrix.py:66-69 sets --ulimit nofile=65536:65536 only for the disposable OB container, without touching host/daemon limits.
  • Test-only fixture repairs are clean. The four mockStatic(LoggerFactory.class, CALLS_REAL_METHODS) changes (ExceptionHandlers/WebsocketCollector/MetaDataService/UpstreamCheckService tests), the discoveryDO.setNamespaceId addition, and the ApplicationEventPublisher mock in ProxySelectorServiceImplTest align the inherited master fixtures without altering production behavior or existing assertions.

Evidence

  • pr_build, native-sql-matrix (mysql/ob/pg/og/oracle, fresh+upgrade), agent_gateway_migrations, rat_check, check-license-header, and all 21 shenyu-integrated-test-* jobs are green at head 342955c0.
  • The shared-source routing and the validator are covered by new Python unit tests.

Non-blocking suggestions

  • The H2 dictionary DEFAULT_KEY_RESOLVER vs CONTEXT_PATH_KEY_RESOLVER divergence is correctly documented as a separate follow-up; please keep that tracked so the H2 seed does not silently diverge further.
  • Consider adding a short release note that fresh MySQL installs also change (orphan permission removal + loggingKafka field rename), since the impact is not OB-only — existing deployments must use the version-specific db/upgrade scripts, which is already called out in init-guide.md.

Approving. Nice work on the reconciliation rigor.

@Aias00
Aias00 merged commit a98da24 into apache:master Oct 11, 2026
51 checks passed
@Aias00 Aias00 added this to the 2.7.2 milestone Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task] Reuse MySQL initialization SQL for OceanBase and remove the duplicate script

3 participants