Skip to content

feat(json): add type_json_skip_invalid_typed_paths setting#89886

Merged
Avogar merged 1 commit intoClickHouse:masterfrom
maxjustus:type-json-add-skip-invalid-typed-paths-setting
Dec 4, 2025
Merged

feat(json): add type_json_skip_invalid_typed_paths setting#89886
Avogar merged 1 commit intoClickHouse:masterfrom
maxjustus:type-json-add-skip-invalid-typed-paths-setting

Conversation

@maxjustus
Copy link
Copy Markdown
Contributor

@maxjustus maxjustus commented Nov 11, 2025

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Add setting type_json_skip_invalid_typed_paths to disable exceptions for inserts/type casts to JSON type when input JSON cannot be cast to explicit typed paths in JSON type. Falls back to null/zero value of typed path. Closes #86917

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Details

Adds type_json_skip_invalid_typed_paths setting so you can gracefully handle unknown input data when you want to always have a specific type for a given path:

select '{"some_field": "asdf", "foo": 1}'::JSON(some_field UInt64) SETTINGS type_json_skip_invalid_typed_paths=1
-- result: {"some_field: 0, "foo": 1}

@nihalzp nihalzp added the can be tested Allows running workflows for external contributors label Nov 12, 2025
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Nov 12, 2025

Workflow [PR], commit [f6c7d3d]

Summary:

job_name test_name status info comment
Stateless tests (amd_msan, parallel) failure
00816_long_concurrent_alter_column FAIL cidb, issue
Integration tests (amd_asan, db disk, old analyzer, 3/6) failure
test_restore_replica/test.py::test_fix_metadata_version_on_attach_part_after_restore FAIL cidb, issue
AST fuzzer (amd_debug) failure
Logical error: 'Function dictGetUInt64(__table1.color_id IN (SELECT __table1.id AS id FROM dictionary(_CAST('colors', 'LowCardinality(String)')) AS __table1 WHERE _CAST('red', 'FixedString(3)') = _CAST(__table1.name, 'LowCardinality(Nullable(String))')), 'colors', 'n', __table1.color_id) expects argument 1 to have LowCardinality(Nullable(UInt8)) type but receives Nullable(UInt8) after running InverseDictionaryLookupPass pass' FAIL cidb
BuzzHouse (amd_debug) failure
Logical error: 'Inconsistent AST formatting: the query: FAIL cidb, issue
BuzzHouse (amd_tsan) failure
ThreadSanitizer (STID: 3438-62e3) FAIL cidb
Finish Workflow failure
python3 ./ci/jobs/scripts/workflow_hooks/feature_docs.py failure

@clickhouse-gh clickhouse-gh bot added the pr-feature Pull request with new product feature label Nov 12, 2025
@maxjustus maxjustus force-pushed the type-json-add-skip-invalid-typed-paths-setting branch 2 times, most recently from 9b6ee8e to 7a2041a Compare November 18, 2025 22:21
@maxjustus maxjustus force-pushed the type-json-add-skip-invalid-typed-paths-setting branch 3 times, most recently from 75573c7 to d953314 Compare November 24, 2025 22:24
@maxjustus
Copy link
Copy Markdown
Contributor Author

@nihalzp I think this is good to go, the test failures look unrelated and transient

@Avogar Avogar self-assigned this Nov 25, 2025
@Avogar
Copy link
Copy Markdown
Member

Avogar commented Nov 25, 2025

I will look at it this week

@maxjustus maxjustus force-pushed the type-json-add-skip-invalid-typed-paths-setting branch from d953314 to 63d5157 Compare December 3, 2025 01:18
@maxjustus maxjustus force-pushed the type-json-add-skip-invalid-typed-paths-setting branch from 63d5157 to f6c7d3d Compare December 3, 2025 01:19
@maxjustus
Copy link
Copy Markdown
Contributor Author

ok @Avogar - I pushed an update that should address your feedback!

@Avogar Avogar mentioned this pull request Dec 4, 2025
56 tasks
@Avogar
Copy link
Copy Markdown
Member

Avogar commented Dec 4, 2025

@Avogar Avogar added this pull request to the merge queue Dec 4, 2025
Merged via the queue into ClickHouse:master with commit bf45f13 Dec 4, 2025
123 of 130 checks passed
@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Dec 4, 2025
@telperions
Copy link
Copy Markdown

Hi @maxjustus thanks for implementing this. Is there a way to track when typed paths are skipped with this setting enabled, maybe via warnings?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-feature Pull request with new product feature pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setting json_type_use_default_on_incompatible_cast to allow adding type-hints to non-perfect JSON data

5 participants