Skip to content

Escape non-ASCII characters in tojson like Python's json.dumps - #16908

Open
yashaggarwal85d wants to merge 2 commits into
dbt-labs:mainfrom
yashaggarwal85d:fix-tojson-ensure-ascii
Open

yashaggarwal85d wants to merge 2 commits into
dbt-labs:mainfrom
yashaggarwal85d:fix-tojson-ensure-ascii

Conversation

@yashaggarwal85d

Copy link
Copy Markdown

Resolves #16830

Problem

tojson writes non-ASCII characters as raw UTF-8, but dbt 1.x uses json.dumps with its default ensure_ascii=True, which writes \uXXXX. So local_md5(tojson(...)) gives a different hash on v2 for anything with non-ASCII text, which is what makes Elementary re-insert its artifacts every time you switch engines.

Solution

Added write_string_fragment to PythonStyleFormatter so it escapes everything outside printable ASCII the way Python does: lowercase \uXXXX, surrogate pairs above U+FFFF, and DEL (0x7f) too, since Python escapes that one as well. Same idea as PythonJsonFormatter in dbt-state. Added a test with あ, é, an emoji, U+2028 and DEL, plus the local_md5 hash from the issue (f3dd52a0cdda6cf470609dcfc4759191), all checked against json.dumps in Python.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes ... (tojson output changes for non-ASCII input, but only back to what 1.x does)

This fix was AI-assisted and reviewed by me. Thanks

@yashaggarwal85d
yashaggarwal85d requested a review from a team as a code owner October 27, 2026 12:58
@cla-bot cla-bot Bot added the cla:yes label Oct 27, 2026
codescene-delta-analysis[bot]

This comment was marked as outdated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v2 Bug] tojson writes non-ASCII characters raw instead of escaping them like dbt 1.x, so tojson-based hashes differ between engines

1 participant