Issue Number: close apache#67916

Related PR: apache#67917

Problem Summary:
The max_ngram_diff creation limit incorrectly changed analyzer identity, allowing equivalent custom analyzers to bypass duplicate-index detection. FE also accepted non-ASCII digits that the BE integer parser rejects.

Exclude max_ngram_diff from ngram tokenizer identity and require its value to use ASCII integer syntax.

Release note:
None

Validation:
- ./run-fe-ut.sh --run org.apache.doris.analysis.invertedindex.AnalyzerIdentityBuilderTest,org.apache.doris.indexpolicy.PolicyValidatorTests (27 tests passed)
- FE Checkstyle passed as part of the targeted test run

Behavior changed:
Equivalent ngram analyzers now share an identity regardless of max_ngram_diff, and FE rejects non-ASCII max_ngram_diff values.

Documentation impact:
None

### What problem does this PR solve?

Issue Number: close apache#67916

Related PR: apache#67917

Problem Summary:

An arbitrarily large max_ngram_diff could multiply token output without a hard fan-out bound, and the deterministic regression assertion was not stored as a runner-generated golden.

Cap max_ngram_diff at 255 consistently in FE and BE, cover the accepted and rejected boundaries, and replace the manual token-list assertion with a named golden query generated by the regression runner.

### Release note

The max_ngram_diff tokenizer setting accepts values from 0 through 255.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [x] Manual test
- Behavior changed:
    - [ ] No.
    - [x] Yes. max_ngram_diff values above 255 are rejected to bound per-position token fan-out.
- Does this need documentation?
    - [ ] No.
    - [x] Yes. Document the supported max_ngram_diff range.

Validation:

- ./build.sh --be -j8 (ASAN, Java extensions, build hygiene, and glibc compatibility)
- ./build.sh --fe -j8
- NGramTokenizerTest: 16/16 passed
- AnalyzerIdentityBuilderTest and PolicyValidatorTests: 29/29 passed
- test_ngram_max_diff_custom_analyzer: runner-generated golden and clean comparison passed on an isolated local FE/BE
- clang-format 16, clang-tidy, and Checkstyle passed

…judge regressions against the merge base

Review of apache#12 raised two gaps in the regression rule:

1. doc-templates.md scoped the Major floor to a list of categories while
   verify-review-docs.py floored every `Regression: yes` regardless, so a
   reviewer following the documented exception for an observability change
   produced a document the verifier rejected.
2. The regression evidence was `git show $BASE_SHA:<path>`, the target-branch
   tip. When the PR branch is behind that tip, a fix that landed on the target
   branch after the PR branched off is absent at HEAD without the PR having
   removed anything, and the rule would have turned it into a Major regression.

Category-aware floor:

- Every finding carries a mandatory `Category` / `类别` line from a closed
  vocabulary: functional-bug, functional-loss, data-error, resource-leak,
  performance, observability, test-coverage, wording, maintainability. The
  verifier rejects an unknown value (a typo must not escape the floor),
  accepts one parenthetical domain note after the class, resolves ZH aliases
  (功能性bug, 功能性缺失, 资源泄漏, 性能), and requires EN and ZH to agree.
- A regression in the first five categories is at least Major. A regression
  in the other four may stay Minor/Nit only with a `Severity rationale`
  paragraph, which the verifier now checks for.
- The JSON result adds `floored_regressions`. post-pass-comment.sh refuses on
  a floored regression (it can only appear if the two scripts disagree) and
  refuses to post when a non-floored regression would go undisclosed: the
  receipt needs at least one note, and SKILL.md / pr-comment-format.md say
  every such finding is named there first.

Merge base:

- Regression evidence is `git show $MERGE_BASE:<path>` in SKILL.md (premise
  checks, step 6), prompts.md (preamble, D1, E), the ledger skeletons and the
  templates; `{MERGE_BASE}` joins the prompt placeholders. `BASE_SHA` keeps
  its two jobs: the diff range and the commit the receipt binds to.
- prepare-review-context.sh records `TARGET_AHEAD` (target-branch commits
  since the merge base) in meta.env and prints it when non-zero.
- Two new common traps: filing a real behaviour change under a soft category,
  and comparing against the target-branch tip.

Tests: verify/repo-review covers each floored category, the rationale
requirement and acceptance for each non-floored category, the parenthetical
note, spelling variants, missing / unknown / disagreeing categories, and the
poster's disclosure gate. Suite: 63 PASS (was 47). Also replayed against the
real apache/doris#67916 documents: they verify after relabelling with the
closed vocabulary, a test-coverage regression is rejected without a rationale
and accepted with one, and a functional-bug regression rated Minor is
rejected; that review's own header already had BASE_SHA != MERGE_BASE
(TARGET_AHEAD = 5).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01H2in7A8yA4qQ3LwAghueTR

…fference #67917 (#68774)

### What problem does this PR solve?

Issue Number: close #67916

Related PR: #67917

Problem Summary:

Cherry-pick of #67917 (commit 6a79f95, a squash merge) to
branch-4.2. Custom ngram tokenizers can now set `max_ngram_diff` (0 to
255, default 1) instead of the hard-coded difference of 1. FE and BE
validate the same range, and newly created policies cap absolute gram
sizes at 1024. A persisted `max_ngram_diff` marker keeps marker-less
policies from before the change valid. `max_ngram_diff` does not change
the analyzer identity.

branch-4.2 also has the gram `mode` tokenizer, which the source change
does not know about. Adaptations:
- `max_ngram_diff` and the size limits apply to the legacy (no `mode`)
path only.
- `max_ngram_diff` together with `mode` is rejected.
- The compatibility marker is added only to new legacy ngram policies.

### Release note

Allow custom ngram tokenizers to configure the maximum difference
between max_gram and min_gram with max_ngram_diff values from 0 through
255.

### Check List (For Author)

- Test
- [ ] Regression test (the case is included; it was not run locally and
is left to CI)
    - [x] Unit Test
    - [ ] Manual test

- Behavior changed:
    - [ ] No.
- [x] Yes. New custom ngram tokenizers can opt into a wider gram-size
range (difference up to 255, absolute size up to 1024).

- Does this need documentation?
    - [ ] No.
    - [x] Yes. Same documentation as the source PR.

### Validation

- FE: `run-fe-ut.sh --run
PolicyValidatorTests,AnalyzerIdentityBuilderTest`: 117 tests, 0
failures; Checkstyle passed.
- BE: `NGramTokenizerTest.*`: 21/21 passed (includes the 7 new tests),
ASAN.
- clang-format 16 and Checkstyle merge-tree preflight: PASS.
- Local-environment notes (not part of this PR): `origin/branch-4.2`
currently declares thrift field id 229 twice in
`PaloInternalService.thrift` (`enable_paimon_rust_reader`,
`enable_prune_nested_column`), so a clean branch-4.2 build fails; I
bumped one id locally only to run the tests. For the BE UT I also
stubbed the unrelated paimon_rust sources locally and linked with
`--unresolved-symbols=ignore-all`. None of this is in the commit.

### Hunk audit (against #67917)

| File | Hunk | Result |
|---|---|---|
| ngram_tokenizer.cpp | `@@ NGramTokenizer::init` buffer size |
Ported(verbatim) |
| ngram_tokenizer.h | trailing newline | Ported (4.2 keeps
`compile_check_end.h`) |
| ngram_tokenizer_factory.cpp | `initialize` limits and diff check,
newline | Ported(verbatim); the 4.2 gram branch returns earlier, so
these apply to the legacy path only |
| ngram_tokenizer_factory.h | `MAX_NGRAM_DIFF`/`MAX_NGRAM_SIZE`
constants | Ported(verbatim) |
| ngram_tokenizer_test.cpp | 7 new tests | Ported; 4.2's
`RejectsMalformedUtf8AndCanBeReset` kept |
| NGramTokenizerValidator.java | `ALLOWED_PROPS`, constants,
`isValidPolicy`, size/diff checks | Adapted: 4.2 `mode` props kept,
`max_ngram_diff` rejected with `mode` |
| IndexPolicy.java | `@@ isInvalid` | Ported (uses `isValidPolicy`; same
behavior as 4.2's try/catch) |
| IndexPolicyMgr.java | `@@ validateReferencedComponentsUsableLocked` |
N-A: 4.2 already has it (tokenizer `isInvalid` check present) |
| IndexPolicyMgr.java | `@@ createIndexPolicy` marker | Adapted: only
for non-`mode` ngram |
| IndexPolicyMgr.java | `@@ validateReferencedPolicy` isInvalid check |
N-A: already present in 4.2 |
| IndexPolicyMgr.java | `@@ warnIfUnsupported` message | Ported |
| AnalyzerIdentityBuilder.java | `PROP_MAX_NGRAM_DIFF`, `isInvalid`
identity, `remove(max_ngram_diff)` | N-A: all three already present in
4.2 (verified with `git grep` on `origin/branch-4.2`) |
| AnalyzerIdentityBuilderTest.java | 3 tests | Ported; 4.2's
`testNgramDefaultBoundsDoNotChangeAnalyzerIdentity` kept |
| PolicyValidatorTests.java | difference/size/marker tests | Ported; 4.2
gram-mode tests kept; 1 new test for `max_ngram_diff` + `mode` |
| regression-test .groovy/.out | new files | Ported(verbatim) |

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 5.5 <[email protected]>

…69171)

## Problem and behavior

`branch-4.2` is missing several external type mapping fixes already
merged into master. This adapted backport aligns these paths while
retaining the branch's existing VARBINARY and TIMESTAMPTZ support:

| Path | Result | Upstream |
| --- | --- | --- |
| Parquet/ORC file TVFs | Unannotated binary fields infer VARBINARY with
the legacy option absent, false, or true, including nested fields |
#68786 |
| Oracle JDBC | BINARY_FLOAT maps to FLOAT; BINARY_DOUBLE maps to DOUBLE
| #68200 |
| SQL Server JDBC | Resolve user-defined aliases by JDBC type code;
strip IDENTITY decoration only when the base type agrees with that code
| #67916 |
| Iceberg metadata scanner | Preserve signed epoch microseconds and
timestamp-without-zone wall-clock values through nested structs, arrays,
and maps | #68985 |

Logical UUID annotations retain their existing file-format semantics
instead of being overridden by the raw-binary default. Parquet schema
discovery retains canonical STRING values and ORC retains native UUID
values. Modern and legacy Parquet ENUM annotations remain STRING. Full
external native UUID alignment and nanosecond timestamp support in
#69163 remain separate. Golden results in this PR target its own
branch-4.2 build and do not assume those reader changes have landed. The
Iceberg scanner keeps this branch's full-schema source indexes rather
than copying master's projected-row indexing. The master JDBC scanner
architecture remains outside this backport. Oracle connections enable
binary IEEE floating-point bindings so valid BINARY_DOUBLE values
outside NUMBER’s range can be written without overflow.

Binary TVF columns display hexadecimal values. Direct CTAS into an
internal VARBINARY column is rejected, consistently with master. The
CTAS partitioning/distribution suite explicitly decodes its UTF-8 binary
fixture into STRING; the dedicated binary suite checks rejection of
direct VARBINARY CTAS and byte-preserving HEX materialization. Paimon
raw-file TVF expectations now preserve binary bytes even without the
obsolete mapping option. Variant shredding assertions likewise require
binary metadata and residual values while preserving all physical-layout
and logical round-trip checks.

## Validation

- Reproduced the Oracle failure against a real Oracle database: the old
executor throws `IllegalArgumentException: Overflow` from
`setDouble(1e200)`; the fixed executor preserves large/small doubles,
floats, and NULLs through INSERT/readback.
- Added shared-schema coverage for modern/legacy ENUM annotations and
UUID precedence, plus an ENUM-only CTAS regression with both legacy
option values.
- Updated scalar/nested UUID regression coverage so a raw-binary default
cannot override logical UUID annotations.

- Built FE and the full ASAN BE from this PR worktree. FE Checkstyle and
clang-format 16 passed.
- Five Oracle Java tests passed against a real Oracle database,
including the optional INSERT/readback integration test; the new Doris
JDBC regression block also passed.
- Two focused production-library-linked Parquet schema tests passed; the
new logical-annotation test failed against the previous implementation
as expected. All four modified BE test translation units passed
compilation checks.
- All seven UUID regression suites and the complete HDFS Parquet group 6
suite passed actual generation followed by independent comparison runs,
with zero skipped suites. Updated golden blocks were taken from those
executions.

The previous coverage build stopped while downloading Maven (`wget` exit
4), before project compilation. This is an infrastructure failure and is
retriggered with the updated source. Muted unrelated compaction/cache/S3
timeout cases are not changed here.