Repository navigation
[doc](variant) Rework the VARIANT reference for write, CAST, NULL, ALTER and ordering semantics - #4170
Conversation
…TER and ordering semantics Rewrite the dev (5.0.0) VARIANT reference in English and Chinese around the behavior of master with apache/doris#67675: - how each write path turns input into a VARIANT value (INSERT keeps strings, PARSE_TO_VARIANT and load jobs parse JSON, invalid JSON stays a string); - what storage normalizes (null members, empty containers, root null, dates, dotted keys, mixed booleans and numbers); - CAST in both directions, failure to NULL regardless of strict mode, and the implicit conversion rules for sub-path comparisons, function and JSON function arguments, with no Schema Template auto cast; - SQL NULL versus VARIANT null, for computed and for stored values; - equality, hashing and the total order used by GROUP BY, DISTINCT, joins, ORDER BY and window keys, with the supported-operation matrix and caveats; - Schema Template write conversion, reading and pattern matching; - storage type inference and conflicts; - ALTER TABLE support (no NOT NULL ADD COLUMN, no template or property change except variant_doc_materialization_min_rows); - column properties, session variables and BE configurations. Also fix PARSE_TO_VARIANT, TRY_PARSE_TO_VARIANT, VARIANT_TYPE and ELEMENT_AT, the workload guide and import examples, and the VARIANT feature page, which still inserted JSON strings or described 4.x behavior. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Fix what two independent reviews and further tests on a PR #67675 cluster found in the first revision: - make every cross-page link a file link (.md/.mdx); with trailingSlash enabled, extension-less relative links resolve under the current page and already break on the published site; - document the whole-document fallback to a string when JSON contains an integer outside [-2^63, 2^64 - 1] or a number outside the DOUBLE range, plus the 128-level depth limit, invalid UTF-8, and the duplicate-key and invalid-JSON BE configurations; - document the implicit conversion pitfalls: DECIMAL(38, 9) overflow and rounding, no index or zone-map use, VARIANT mixed with other types in IF/CASE/COALESCE/IFNULL/UNION, and `v IN (literal list)`; - INSERT ... SELECT from table functions and http_stream keep strings; load jobs parse strings of any format; top-level JSON booleans load as 1/0; - storage normalization applies to object members outside arrays, all-null arrays can be dropped, and dotted-key conflicts fail the whole write; - ALTER on a column with a Schema Template only allows DROP, RENAME and a comment change; DOC mode template types; TIMESTAMP_NS text; LARGEINT and UUID in CAST; VARIANT_TYPE timestamp names; - use explicit CAST in index examples, fix FAQ 5 and the TRY_PARSE_TO_VARIANT description, regenerate the import page DESC output, and clean up the old sections (version notes, terminology, table pipe escaping). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…review A second independent review of the reworked pages, checked on a PR #67675 cluster, found statements that still did not match the engine: - CAST from VARIANT returns NULL only when the value's type has a conversion to the target; a number to TIMESTAMPTZ/IPV4/IPV6 or a date to BOOLEAN fails the statement, and TIMESTAMPTZ does not accept numbers; - a JSON number on an IPV4/IPV6/TIMESTAMPTZ template path fails the whole write, and mixing strings and numbers on date, time or IP template paths can drop valid values (caution added); - group commit INSERT through JDBC server-side prepared statements parses JSON like a load job; the Arrow format cannot load VARIANT; filtered NOT NULL rows count toward max_filter_ratio; - an alias of a subpath is a whole value for implicit conversion; TIMESTAMPTZ compares in the session time zone; only comparisons whose implicit type differs from the stored type lose index use; - the JSON workaround turns 39-digit integers into strings; deeply nested documents rejected by the parser are kept as strings; the duplicate-path configuration also covers dotted-key conflicts; - fix FAQ 6, the zh rounding wording and link text, VARIANT_TYPE for LARGEINT, the workload guide ALTER note, and the zh import description. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
|
||
| ## 导入时值如何转换 | ||
|
|
||
| - **CSV 格式:** `VARIANT` 字段的文本按 JSON 解析。`\N` 导入为 SQL `NULL`。非法 JSON 文本导入为 VARIANT 字符串。 |
There was a problem hiding this comment.
CSV 格式、JSON 格式、超范围的数值 不用提及
| - **超范围的数值:** 包含超出 [-2^63, 2^64 - 1] 的整数或超出 `DOUBLE` 范围的数值的文档无法解析,整个文档会作为一个 VARIANT 字符串导入。 | ||
| - **`NOT NULL` 列:** VARIANT 值为 SQL `NULL` 的行会被过滤,包括解析失败而得到 SQL `NULL` 的值。被过滤的行计入 `max_filter_ratio`(默认 `0`),因此默认情况下导入作业会失败。 | ||
| - **INSERT 的行为不同:** `INSERT` 不解析字符串。`INSERT INTO t VALUES (1, '{"a": 1}')` 写入的是 VARIANT 字符串 `{"a": 1}`,从 `s3()`、`hdfs()`、`local()` 执行 `INSERT INTO ... SELECT` 也是如此。需要写入对象时请使用 `PARSE_TO_VARIANT`。 | ||
| - **存储会规范化值:** 值为 `null` 的对象成员,以及值为空对象或空数组的成员,在写入时可能被移除。 |
|
|
||
| - `INSERT` 会把字符串作为 VARIANT 字符串写入,不再按 JSON 解析,从 `s3()`、`hdfs()` 等表函数执行 `INSERT INTO ... SELECT` 也是如此。用 `INSERT` 写入 JSON 文本时请使用 `PARSE_TO_VARIANT`;Stream Load 等导入作业仍会解析 JSON。 | ||
| - 整个 VARIANT 值支持 `GROUP BY`、`DISTINCT` 和集合运算;两个 VARIANT 值之间可以用 `=`、`!=`、`<=>` 比较,也可以作为 Join 键、`ORDER BY` 键和窗口键。 | ||
| - 即使路径在 Schema Template 中声明了类型,`v['path']` 仍是 `VARIANT` 类型,需要显式 CAST。会话变量 `enable_variant_schema_auto_cast` 已不再生效。 |
There was a problem hiding this comment.
“即使路径在 Schema Template 中声明了类型” 这段不用描述
|
|
||
| ## 写入数据 {#write-data} | ||
|
|
||
| ### 输入如何变成 VARIANT 值 {#how-input-becomes-a-variant-value} |
There was a problem hiding this comment.
“输入如何变成 VARIANT 值 ”, 这段不保留删了,前面重复了
|
|
||
| - 如果要将字符串或 JSON/JSONB 表达式解析为结构化 VARIANT 值,请使用 [PARSE_TO_VARIANT](../../../sql-functions/scalar-functions/variant-functions/parse-to-variant)。 | ||
| - 如果要将受支持的 SQL 值转换为 VARIANT,请使用 `CAST(expression AS VARIANT)`。字符串会保留为 VARIANT 字符串值,该 CAST 不解析 JSON。 | ||
| ### 解析错误 {#parse-errors} |
There was a problem hiding this comment.
这里你再理一下正确行为应该是怎么样的,这里的行为比较奇怪, 这些 BE 的 config 看下是不是可以淘汰了
| | 数组内的值,如 `{"arr": [{"a": null}, {}, [], null]}` | 原样保留;`v['arr'][1]['a']` 是 VARIANT `null`。 | | ||
| | 根值为 JSON `null`,如 `PARSE_TO_VARIANT('null')` | 空对象 `{}` | | ||
| | 根值为空数组 `[]` 或空对象 `{}` | 保留 | | ||
| | 不在 Schema Template 路径上的 `DATE`、`DATETIME` 值,如 `CAST(date_col AS VARIANT)` | 以文本形式存储,读回时是字符串 | |
There was a problem hiding this comment.
为什么读回时是字符串, parquet 编码没有这个类型?
| | `BIGINT` 范围内的整数 | `tinyint`、`smallint`、`int` 或 `bigint` 中能容纳该值的最小类型 | | ||
| | 超出 `BIGINT` 范围、不超过 18446744073709551615 的整数 | `decimal` | | ||
| | 带小数部分或指数的数值 | `double` | | ||
| | 超出 [-2^63, 2^64 - 1] 的整数,或超出 `DOUBLE` 范围的数值 | 不作为数值处理:整段文本都是非法 JSON(参见[解析错误](#parse-errors)) | |
There was a problem hiding this comment.
这里如果要支持?应该怎么做?或者加上引号比如"12345"?
| ``` | ||
|
|
||
| ### Decimal 与日期时间转换限制 | ||
| 读取整个 VARIANT 值会返回 JSON 文本。对象 key 按字节序输出,且不含空白,因此与输入文本并非按字节完全一致。字符串根值输出时不带引号。`CAST(v AS STRING)` 则按对应 SQL 类型的格式输出标量根值,例如布尔根值变为 `1` 或 `0`。 |
There was a problem hiding this comment.
"CAST(v AS STRING) 则按对应 SQL 类型的格式输出标量根值,例如布尔根值变为 1 或 0" 这句话不用写
| | `CHAR`、`VARCHAR`、`STRING` | VARIANT 字符串,不解析 JSON 文本。字符串必须是合法的 UTF-8,否则 CAST 报错。 | | ||
| | `BOOLEAN` | 布尔值。 | | ||
| | `TINYINT`、`SMALLINT`、`INT`、`BIGINT` | 整数。 | | ||
| | `LARGEINT` | 定点数。绝对值超过 10^38 - 1 的值会变成字符串。 | |
There was a problem hiding this comment.
“绝对值超过 10^38 - 1 的值会变成字符串” 这是正确的行为吗?应该怎么处理
| | `BOOLEAN` | 布尔值保持不变;数值非零即为 `true`;字符串按 `CAST(string AS BOOLEAN)` 的规则转换。 | | ||
| | `TINYINT`、`SMALLINT`、`INT`、`BIGINT`、`LARGEINT` | 整数。定点数和浮点数的小数部分被截断(`1.5` 变为 `1`);布尔值变为 `1` 或 `0`;`"123"` 这样的数字字符串会被转换。 | | ||
| | `FLOAT`、`DOUBLE`、`DECIMAL(p, s)` | 数值和数字字符串。`DECIMAL` 按 scale `s` 舍入。超出 `FLOAT` 范围的值变为 `Infinity`。 | | ||
| | `DATE`、`DATETIME(p)`、`TIMESTAMP_NS` | 日期时间值、日期时间格式的字符串,以及 `20240102` 这样的数值。 | |
| | `DATE` | 不含时间和时区的日历日期。 | | ||
| | `DATETIME(p)` | `0 <= p <= 6`,不做时区调整。 | | ||
| | `TIMESTAMP_NS` | 纳秒精度,不做时区调整;值必须在 TIMESTAMP_NS 取值范围内。 | | ||
| | `TIMESTAMPTZ(p)` | 不能 CAST 为 VARIANT。可以在 Schema Template 中把路径声明为 `TIMESTAMPTZ`。 | |
There was a problem hiding this comment.
Schema Template 禁止一下TIMESTAMPTZ,直接不支持
| ## NULL 语义 {#null-semantics} | ||
|
|
||
| 这些规则适用于 `GROUP BY`、`DISTINCT`、`COUNT(DISTINCT ...)`、`INTERSECT`、`EXCEPT` 和 `UNION DISTINCT` 等已支持的操作,但不会开放根 VARIANT 比较谓词:直接执行 `VARIANT = VARIANT` 或排序比较仍不支持。 | ||
| ### SQL NULL 与 VARIANT null {#sql-null-and-variant-null} |
|
|
||
| 如果 `COALESCE` 或 `IFNULL` 中的 `x` 不是 VARIANT,VARIANT 一侧会先被 CAST,参见[隐式转换](#implicit-conversion)。 | ||
|
|
||
| ### 计算侧的值与存储侧的值 {#computed-values-and-stored-values} |
There was a problem hiding this comment.
"计算侧的值与存储侧的值" 这一节删掉, 前面提过了
| | `TIMESTAMPTZ(p)` | 不能 CAST 为 VARIANT。可以在 Schema Template 中把路径声明为 `TIMESTAMPTZ`。 | | ||
| | `TIME` | 不支持。 | | ||
|
|
||
| ### 隐式转换 {#implicit-conversion} |
| | {"datetime_val":"datetimev2","decimal_val":"decimal128i","ip_val":"ipv4","large_int_val":"largeint","string_val":"string","tz_val":"timestamptz"} | | ||
| +---------------------------------------------------------------------------------------------------------------------------------------------------+ | ||
| ``` | ||
| `CHAR`、`VARCHAR`、`DECIMALV2`、`TIME`、`JSON`、`MAP`、`STRUCT` 和嵌套数组不能用于 Schema Template。DOC mode 下,模板字段进一步限制为字符串、整数、`FLOAT`、`DOUBLE`、`BOOLEAN` 以及由这些类型组成的数组。 |
| 列创建后不能修改 Schema Template,参见 [ALTER TABLE](#alter-table)。 | ||
|
|
||
| 当同一路径出现不兼容类型(如同一字段既出现整数又出现字符串)时,将提升为 JSONB 类型以避免信息丢失: | ||
| ## ALTER TABLE {#alter-table} |
| - 输入为 JSON 字面量 `null` 时返回 Variant/JSON `null`,它与 SQL `NULL` 不同。 | ||
| - JSON 非法、对象 key 重复且当前校验设置不允许、嵌套深度超限或发生其他校验错误时,查询失败。 | ||
| - 输入为 JSON 字面量 `null` 时返回 VARIANT `null`,它与 SQL `NULL` 不同。 | ||
| - 非法 JSON 文本会作为 VARIANT 字符串返回;包含超出 [-2^63, 2^64 - 1] 的整数或超出 `DOUBLE` 范围的数值的 JSON 也是如此,整段文本变为一个字符串。该行为由 BE 配置 `variant_throw_exeception_on_invalid_json`(默认 `false`)控制;设置为 `true` 后,这类输入会使查询失败。 |
There was a problem hiding this comment.
variant_throw_exeception_on_invalid_json 这个配置是否不应该保留, 有 try_parse_to_variant/parse_to_variant 本身控制, 比如 try 的不报错,否则报错?
- VARIANT reference: drop the Schema Template auto-cast note, the input-to-VARIANT table, the implicit conversion subsection, the computed-vs-stored NULL subsection and the unsupported template types line; shorten NULL semantics and ALTER TABLE; rename the CAST section. - Import page: drop the CSV, JSON and out-of-range number bullets and add a demo of how storage removes null members and empty containers. - Restore the key-features page, ELEMENT_AT and TRY_PARSE_TO_VARIANT changes that 587127d reverted by mistake. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Import page: separate the normalization paragraph from the list so it renders as its own paragraph, rename the section to match what is left, and point to Parse errors instead of the removed write rules. - NULL semantics and FAQ: the null-member removal applies only outside arrays. - Mixing VARIANT with an integer converts to DECIMAL(38, 9); say so where the page mentions the conversion, which FAQ 6 relies on. - ALTER TABLE: only MODIFY COLUMN is restricted on a column that has a Schema Template; restore the rule for omitted properties; name the only property that can change. - ELEMENT_AT: move the null note below the index example it interrupted. - TRY_PARSE_TO_VARIANT: add the duplicate-key configuration and the empty string case, as on the PARSE_TO_VARIANT page. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…plates ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: A Schema Template path declared as TIMESTAMPTZ is only half supported: CAST(TIMESTAMPTZ AS VARIANT) is rejected, DOC mode rejects the path, and a JSON number written to the path fails the whole write. Remove TIMESTAMP_TZ from the VARIANT sub-type whitelist. CREATE TABLE and ALTER TABLE ... ADD COLUMN now reject VARIANT<'p': TIMESTAMPTZ> and VARIANT<'p': ARRAY<TIMESTAMPTZ>> with "VARIANT unsupported sub-type". The whitelist is only checked when a type is defined, so tables that already declare such a path (branch-4.0 and branch-4.1 accept it) keep working. test_variant_timestamptz_sparse and the TIMESTAMPTZ case of test_variant_json_functions can no longer create their tables and are removed. The sparse binary round trip they covered is still tested by DataTypeTimeStampTzSerDeTest.binary_roundtrip. ### Release note A VARIANT Schema Template can no longer declare TIMESTAMPTZ paths. Existing tables are not affected. ### Check List (For Author) - Test: Regression test (test_predefine_ddl, test_variant_json_functions) and Unit Test (TypeTest) - Behavior changed: Yes (CREATE/ALTER with a TIMESTAMPTZ template path fails) - Does this need documentation: Yes (apache/doris-website#4170) Co-Authored-By: Claude Opus 5.5 <[email protected]>
…d as text - apache/doris#67675 now rejects TIMESTAMPTZ in a Schema Template; remove it from the accepted types, the example and the write caution, and say so in the date/time conversion table. - Dates become text only in the root value and on paths that also hold other types. An object path that holds only dates keeps them as dates in subcolumns, sparse columns and DOC mode, also after compaction. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
|
||
| ## Using VARIANT | ||
| :::info Version | ||
| This page describes VARIANT in Doris 5.0.0 and later. The differences from Doris 4.x that are most likely to affect existing SQL: |
| ### 排序规则 {#ordering} | ||
|
|
||
| SQL `NULL` 与 Variant/JSON `null` 是不同的值: | ||
| 对 VARIANT 值执行 `ORDER BY` 时使用统一的全序。不同种类的值先按种类排序: |
There was a problem hiding this comment.
这里再补充一下number的比较规则再细化一下
| - Variant/JSON `null` 是一个 VARIANT 值,例如 `PARSE_TO_VARIANT('null')` 的返回值。 | ||
| - `TRY_PARSE_TO_VARIANT` 遇到非法输入时返回 SQL `NULL`,这与成功解析 JSON 字面量 `null` 不同。 | ||
| ```text | ||
| null < boolean < number < string < binary < date < timestamp with time zone |
There was a problem hiding this comment.
把number 里float,int,decimal 具体怎么比较的再详细说一下吧
Explain integral normalization, exact mixed numeric comparison, and decimal versus floating-point tie-breaking in the English and Chinese VARIANT ordering sections. Include 1/1.0/1.00, fractional values, decimal scales, negative zero, floating-point precision, and a runnable ordering example. Validation: execute the new sorting example and eight equality checks on existing PR-compatible binaries; verify comparison code against #67675; pass diff, markdown structure, frontmatter and changed-link checks.
What changed
Reworks the dev (Doris 5.0.0) VARIANT documentation, English and Chinese, so that it describes the behavior of master together with apache/doris#67675:
VARIANT.md), still one page for the type:INSERTandCASTkeep strings, whilePARSE_TO_VARIANTand load jobs parse JSON. Parse errors: out-of-range numbers turn the whole document into a string, plus the depth limit, invalid UTF-8, duplicate keys, and the BE configurations that change these rules.nullmembers and empty containers outside arrays are dropped, while values inside arrays are kept; a rootnullreads back as{}; dates outside a Schema Template become strings; dotted keys become nested paths; mixed booleans and numbers can become1/0.NULLregardless ofenable_strict_cast, while a type pair without any conversion fails the statement.NULLversus VARIANTnullfor the common operators, and how storage removesnullmembers.1 = 1.0butDECIMAL 1.5 != DOUBLE 1.5;ADD COLUMN ... VARIANT NOT NULLis impossible;variant_doc_materialization_min_rowscan change;PARSE_TO_VARIANT/TRY_PARSE_TO_VARIANT: which inputs error or returnNULL, and which are kept as strings.VARIANT_TYPE: returns one type name for the value itself.ELEMENT_AT: theNULLcases and null semantics.INSERTs a JSON string into VARIANT any more;DESCoutput regenerated from its own sample;.md/.mdx). The site setstrailingSlash: true, so extension-less relative links resolve under the current page. For example, the published VARIANT page currently links to.../semi-structured/VARIANT/variant-workload-guide/.Why
The dev pages still described several 4.x behaviors that no longer hold on master, and some pages contradicted each other:
INSERT ... VALUES ('{...}'), which now stores a VARIANT string.nulland SQLNULLare equivalent.TRY_PARSE_TO_VARIANTclaimed that invalid JSON returnsNULL.VARIANT_TYPEshowed a per-path JSON map.TIMESTAMPTZas a source.CAST behavior, ALTER support, NULL handling and GROUP BY/ORDER BY semantics were not documented at all.
Version wording: "Doris 4.2 and later" is replaced by "Doris 5.0.0 and later", because branch-4.2 keeps
enable_variant_v2 = false(VARIANT V1 by default), and other dev pages already refer to master features as 5.0.0. The 4.x pages are intentionally unchanged.Validation
050b83b148c), with default session variables and BE configuration. Coverage:INSERT, group commit,http_stream, and Stream Load in CSV and JSON format;TypeCoercionUtils,CheckCast,Column.checkSchemaChangeAllowed,PropertyAnalyzer,NereidsLoadUtils,LogicalPlanBuilder;variant_canonical.cpp,variant_string_parse.cpp,variant_path_builder.cpp,new_json_reader.cpp.ELEMENT_AT,TRY_PARSE_TO_VARIANT) that the third commit had reverted to master by mistake. Another independent review of that commit found a list-rendering bug and a few imprecise sentences, fixed in the fifth commit.TIMESTAMPTZSchema Template paths, and narrows the note on dates stored as text: that happens to root values and to paths that also hold other types. Object paths that hold only dates keep their type in subcolumns, sparse columns and DOC mode; verified on the cluster, including after a full compaction.git diff --checkpassed.docs:links:changed,docs:sql-functions:changed,docs:features:changed, and the frontmatter and markdown-structure lints report no findings.docs:i18n-sync:changedreports onlyi18n-sync-version-counterpartwarnings. This is an intentional exception, because the 4.x docs describe VARIANT V1.docs:seo:changedreports only the existing cross-version duplicate-title warnings.Behaviors that look like bugs are documented as they are today and tracked separately:
DECIMAL(38, 9)comparisons;NULL, for type pairs without a conversion, such as a number toTIMESTAMPTZorIPV4;The pages will need updating if these are fixed.
🤖 Generated with Claude Code