Skip to content

[fix](fe) Preserve inferred JSON column name casing - #67411

Merged
hello-stephen merged 2 commits into
apache:masterfrom
0AyanamiRei:fix/doris-25132-json-key-case-matching
Oct 21, 2026
Merged

[fix](fe) Preserve inferred JSON column name casing#67411
hello-stephen merged 2 commits into
apache:masterfrom
0AyanamiRei:fix/doris-25132-json-key-case-matching

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: JSON schema inference lowercased field names before creating the table-valued function schema, while JSON readers match object keys case-sensitively. Uppercase keys therefore appeared as queryable lowercase columns but were silently materialized as NULL. Preserve the inferred key casing for JSON only, while retaining case-insensitive duplicate detection and the existing lowercase behavior for other file formats.

Release note

JSON table-valued functions now read inferred mixed-case and uppercase keys without silently producing NULL values.

Check List (For Author)

  • Test: Regression test and manual tests
    • Regression test: test_json_uppercase_keys (raw and jsonpaths control)
    • Manual test: File Scanner V2, legacy scanner, and http_stream returned the two expected rows
    • Manual compatibility test: CSV inferred column names remained lowercase
  • Behavior changed: Yes. Inferred JSON field names preserve source casing.
  • Does this need documentation: No

@0AyanamiRei
[fix](fe) Preserve inferred JSON column name casing
61d6f6f 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: JSON schema inference lowercased field names before creating the table-valued function schema, while JSON readers match object keys case-sensitively. Uppercase keys therefore appeared as queryable lowercase columns but were silently materialized as NULL. Preserve the inferred key casing for JSON only, while retaining case-insensitive duplicate detection and the existing lowercase behavior for other file formats.

### Release note

JSON table-valued functions now read inferred mixed-case and uppercase keys without silently producing NULL values.

### Check List (For Author)

- Test: Regression test and manual tests
    - Regression test: test_json_uppercase_keys (raw and jsonpaths control)
    - Manual test: File Scanner V2, legacy scanner, and http_stream returned the two expected rows
    - Manual compatibility test: CSV inferred column names remained lowercase
- Behavior changed: Yes. Inferred JSON field names preserve source casing.
- Does this need documentation: No
@hello-stephen

hello-stephen commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei

0AyanamiRei commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 61d6f6f45927bdf11667e5f7578565b9d04dab05, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17570	3057	3029	3029
q2	2083	254	217	217
q3	10267	892	519	519
q4	4689	247	197	197
q5	7670	562	385	385
q6	138	111	92	92
q7	518	502	379	379
q8	9240	908	889	889
q9	3431	2378	2370	2370
q10	6505	850	697	697
q11	414	202	178	178
q12	633	263	195	195
q13	18102	1537	1166	1166
q14	159	152	137	137
q15	q16	440	392	373	373
q17	1367	913	820	820
q18	3060	2260	2239	2239
q19	1240	900	824	824
q20	385	280	204	204
q21	5620	1810	1857	1810
q22	335	266	231	231
Total cold run time: 93866 ms
Total hot run time: 16951 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3401	3317	3312	3312
q2	510	416	390	390
q3	2181	2277	2141	2141
q4	1194	1159	893	893
q5	2152	2108	2098	2098
q6	170	115	88	88
q7	1003	901	869	869
q8	1597	1414	1405	1405
q9	3117	3092	3094	3092
q10	1840	1813	1624	1624
q11	357	267	248	248
q12	451	424	356	356
q13	1466	1532	1152	1152
q14	181	179	163	163
q15	q16	388	395	345	345
q17	3510	3333	3231	3231
q18	4781	4373	4692	4373
q19	893	792	941	792
q20	993	947	866	866
q21	3806	3168	3250	3168
q22	388	350	337	337
Total cold run time: 34379 ms
Total hot run time: 30943 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 61d6f6f45927bdf11667e5f7578565b9d04dab05, data reload: false

query5	4257	403	335	335
query6	391	136	119	119
query7	4961	395	237	237
query8	291	127	123	123
query9	8709	2898	2883	2883
query10	408	226	191	191
query11	5378	1040	914	914
query12	117	72	70	70
query13	1223	460	325	325
query14	6083	2177	2072	2072
query14_1	1955	1960	1944	1944
query15	178	121	113	113
query16	930	365	348	348
query17	810	464	367	367
query18	2347	323	237	237
query19	168	143	113	113
query20	71	67	75	67
query21	209	101	88	88
query22	5203	5356	5215	5215
query23	6571	6191	6068	6068
query23_1	6092	5876	6026	5876
query24	7299	1092	795	795
query24_1	777	798	765	765
query25	439	304	259	259
query26	1221	243	131	131
query27	2777	406	251	251
query28	4753	1507	1512	1507
query29	944	458	362	362
query30	256	159	132	132
query31	851	400	327	327
query32	143	76	77	76
query33	467	213	189	189
query34	1001	851	497	497
query35	399	405	359	359
query36	571	575	527	527
query37	122	81	70	70
query38	1005	832	804	804
query39	501	510	478	478
query39_1	479	448	472	448
query40	206	90	80	80
query41	59	60	55	55
query42	77	71	75	71
query43	275	239	213	213
query44	1014	540	561	540
query45	105	99	105	99
query46	809	836	524	524
query47	777	745	715	715
query48	307	310	232	232
query49	545	225	180	180
query50	715	264	191	191
query51	8198	8188	8149	8149
query52	67	66	58	58
query53	192	193	161	161
query54	226	177	170	170
query55	74	64	53	53
query56	206	268	167	167
query57	705	642	599	599
query58	204	167	165	165
query59	1202	1230	1113	1113
query60	259	197	182	182
query61	142	114	129	114
query62	359	201	178	178
query63	178	137	139	137
query64	2885	679	607	607
query65	1653	1642	1609	1609
query66	1810	263	204	204
query67	9871	9626	9569	9569
query68	3038	1192	746	746
query69	347	212	188	188
query70	676	601	626	601
query71	259	175	169	169
query72	2362	1718	1766	1718
query73	680	644	338	338
query74	1984	1196	1126	1126
query75	1140	1103	959	959
query76	2360	720	552	552
query77	252	253	213	213
query78	3855	3610	3201	3201
query79	2822	811	554	554
query80	1633	319	276	276
query81	522	153	130	130
query82	1164	123	95	95
query83	273	216	189	189
query84	299	117	91	91
query85	843	346	298	298
query86	475	180	168	168
query87	992	974	886	886
query88	2963	2124	2114	2114
query89	294	198	172	172
query90	2020	130	129	129
query91	131	117	101	101
query92	95	71	66	66
query93	2211	1122	716	716
query94	662	280	217	217
query95	535	250	220	220
query96	786	606	264	264
query97	1077	1032	1010	1010
query98	174	138	136	136
query99	432	348	308	308
Total cold run time: 180307 ms
Total hot run time: 81900 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 61d6f6f45927bdf11667e5f7578565b9d04dab05, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.04	0.04
query3	0.24	0.11	0.11
query4	1.60	0.10	0.09
query5	0.17	0.15	0.15
query6	1.27	0.73	0.68
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.29	0.21	0.21
query10	0.35	0.33	0.34
query11	0.17	0.12	0.11
query12	0.15	0.11	0.11
query13	0.30	0.31	0.30
query14	0.45	0.45	0.45
query15	0.36	0.37	0.35
query16	0.24	0.23	0.23
query17	0.72	0.70	0.68
query18	0.18	0.16	0.17
query19	1.17	1.21	1.18
query20	0.01	0.00	0.00
query21	15.47	0.15	0.13
query22	5.05	0.05	0.04
query23	16.20	0.26	0.10
query24	2.93	0.33	0.24
query25	0.12	0.03	0.03
query26	0.74	0.16	0.12
query27	0.04	0.04	0.02
query28	3.66	0.58	0.29
query29	12.41	3.23	2.54
query30	0.26	0.10	0.12
query31	2.76	0.36	0.17
query32	3.53	0.31	0.24
query33	1.35	1.49	1.40
query34	15.38	2.15	1.74
query35	1.75	1.74	1.72
query36	0.46	0.30	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.13	0.08	0.07
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.36 s
Total hot run time: 14.61 s

sollhui
sollhui approved these changes Oct 21, 2026
View reviewed changes

@sollhui sollhui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen

hello-stephen commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

skip buildall

@hello-stephen
hello-stephen merged commit 7a84ab6 into apache:master Oct 21, 2026
36 checks passed
@Gabriel39 Gabriel39 mentioned this pull request Oct 22, 2026
Merged
4 tasks done
@sollhui sollhui mentioned this pull request Oct 22, 2026
Merged
yiguolei pushed a commit that referenced this pull request Oct 23, 2026
@sollhui
branch-4.2: [fix](fe) Preserve inferred JSON column name casing (#69451)
56043de 
### What problem does this PR solve?

Issue Number: None

Related PR: #67411

Problem Summary: Backport #67411 to branch-4.2. JSON schema inference
lowercased field names while JSON readers match object keys
case-sensitively, causing mixed-case and uppercase keys to return NULL.
Preserve inferred JSON key casing, retain case-insensitive duplicate
detection, and keep existing lowercase behavior for other file formats.

The conflict resolution also preserves branch-4.2's existing Lance
column name casing behavior.

### Release note

JSON table-valued functions now read inferred mixed-case and uppercase
keys without silently producing NULL values.

### Check List (For Author)

- Test: Compilation and tests skipped as requested. The original PR's
regression test and fixture are included in this backport. CI requested
with `run buildall`.
- Behavior changed: Yes. Inferred JSON field names preserve source
casing.
- Does this need documentation: No.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sollhui sollhui sollhui approved these changes

Assignees

No one assigned

Labels

dev/4.2.0-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@0AyanamiRei @hello-stephen @sollhui @yiguolei