Add \xHH Unicode escape code to basic strings by eksortso · Pull Request #796 · toml-lang/toml

eksortso · 2020-12-19T08:32:41Z

Per a conversation in #622, this PR adds a two-character \x escape code sequence for the first 256 Unicode code points. This provides a short explicit sequence for, among other code points: ASCII characters, certain control characters, and whitespace characters such as tab (\x09) and space (\x20) that need visible representation without resorting to longer \u and \U codes. Such sequences are commonly found in languages that also support the \u and \U codes for Unicode.

This PR replaces #709 and is intended for a post-v1.0.0 version of the TOML standard.

marzer · 2021-11-05T10:00:05Z

@pradyunsg Where did we land with this? It seems like a friction-free addition to the language (and one that would bring TOML in line with many other languages which also support \x). It's also practically free for implementers since the code paths that would already be handling \u and \U can be trivially extended to support \x.

pradyunsg · 2021-11-05T13:11:49Z

Yea, let's do this.

marzer · 2021-11-16T17:25:48Z

@eksortso Do you have time to update this PR against the current master? If not I'm happy to fork your changes and look after it.

eksortso · 2021-11-17T16:48:44Z

I'll update it later this evening.

marzer · 2021-11-17T17:29:27Z

Awesome, thanks @eksortso

eksortso · 2021-11-18T09:36:51Z

@marzer I think that's got it.

eksortso · 2021-11-18T09:49:04Z

The second change to CHANGELOG.md dresses up the bare URL for toml.io, just to appease the linter. I'll undo it if necessary.

eksortso · 2022-05-03T07:30:07Z

For some reason I'd thought this had already been taken care of. It's up to date again.

@marzer @pradyunsg Care to take look at it?

ChristianSi

Looks good otherwise!

eksortso · 2022-06-06T21:11:00Z

@pradyunsg Any updates on this? Is the issue with pre-commit.ci impeding merges? Or are you planning a larger roadmap for the next release of TOML?

pradyunsg · 2022-07-30T15:53:10Z

Thanks @eksortso! ^>^

toml-lang/toml#796

toml-lang/toml#796 This commit fixes all the remaining TOML 1.1.0 tests, namely: - [string] hex escape - [spec-1.1.0] common 12

* Update the code generator to match the formatting of ./Scripts/format.sh * Reformat the code This commit is a result of `./Scripts/format.sh` * Fix file name mistake in the header of Generated swift files The file is called Scripts/generate-tests.py not Scripts/sync_compliance_tests.py * Prefactoring: get rid of magic constant 8 This prefactoring will be important in later commits when I add support for optional seconds (TOML 1.1.0) cause it won't longer be constant 8 but either 8 or 5 (depending on wheter the seconds are presented or not) The magic constant 8 comes from scanTime where in case of success (not nil value is returned) the total increment to the index variable equals to 8 * Bump toml-test None of the tests are failing after the bump * Bump toml-test spec version 1.0.0 -> 1.1.0 The following tests are failing now: 􀢄 Test "[inline-table] newline" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 3) Syntax error: newline not allowed in inline table. 􀢄 Test "[spec-1.1.0] common 29" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 1) Syntax error: extra chars after value. 􀢄 Test "[string] hex escape" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 Illegal escape character 'x'. 􀢄 Test "[spec-1.1.0] common 12" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 Illegal escape character 'x'. 􀢄 Test "[spec-1.1.0] common 47" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 4) Syntax error: newline not allowed in inline table. 􀢄 Test "[string] escape esc" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 Illegal escape character 'e'. 􀢄 Test "[datetime] no seconds" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 2) Syntax error: extra chars after value. 􀢄 Test "[spec-1.1.0] common 34" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 1) Syntax error: extra chars after value. 􀢄 Test "[spec-1.1.0] common 31" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 1) Syntax error: extra chars after value. 􀢄 Test "[inline-table] newline" failed after 0.014 seconds with 1 issue. 􀢄 Test "[spec-1.1.0] common 29" failed after 0.016 seconds with 1 issue. 􀢄 Test "[string] hex escape" failed after 0.017 seconds with 1 issue. 􀢄 Test "[spec-1.1.0] common 12" failed after 0.017 seconds with 1 issue. 􀢄 Test "[inline-table] newline comment" recorded an issue at TOMLComplianceSupport.swift:34:25: Issue recorded 􀄵 (Line 4) Syntax error: newline not allowed in inline table. 􀢄 Test "[spec-1.1.0] common 47" failed after 0.019 seconds with 1 issue. 􀢄 Test "[string] escape esc" failed after 0.019 seconds with 1 issue. 􀢄 Test "[datetime] no seconds" failed after 0.021 seconds with 1 issue. 􀢄 Test "[spec-1.1.0] common 34" failed after 0.025 seconds with 1 issue. 􀢄 Test "[spec-1.1.0] common 31" failed after 0.026 seconds with 1 issue. 􀢄 Test "[inline-table] newline comment" failed after 0.031 seconds with 1 issue. 􀢄 Suite TOMLValidationTests failed after 0.041 seconds with 10 issues. 􀢄 Test run with 743 tests in 8 suites failed after 0.550 seconds with 10 issues. I will fix the tests in the next couple of commits * TOML 1.1.0: Allow newlines and trailing commas in inline tables toml-lang/toml#904 This commit fixes the following tests: - [inline-table] newline - [inline-table] newline comment * TOML 1.1.0: Seconds in datetime and time values are now optional toml-lang/toml#894 This commit fixes the following tests: - [spec-1.1.0] common 34 - [spec-1.1.0] common 29 - [spec-1.1.0] common 31 - [datetime] no seconds * TOML 1.1.0: Support \e escape for the escape character toml-lang/toml#790 This commit fixes the following test: - [string] escape esc * TOML 1.1.0: Support \xHH notation to basic strings for codepoints <255 toml-lang/toml#796 This commit fixes all the remaining TOML 1.1.0 tests, namely: - [string] hex escape - [spec-1.1.0] common 12 * Docs: Now, we fully implement TOML 1.1.0

Add \x00 notation to basic strings; update ABNF

46b08e9

eksortso mentioned this pull request Dec 19, 2020

Include an \xHH escape code sequence #709

Closed

pradyunsg added the post-1.0 label Dec 19, 2020

pradyunsg removed the post-1.0 label Jul 16, 2021

Merge branch 'master' into unicode_escapes

f3f8f0b

pradyunsg mentioned this pull request Mar 6, 2022

Is it good to add short escaping alias \s for space (\u0020)? #622

Closed

Merge branch 'master' into unicode_escapes

4658a40

marzer approved these changes May 3, 2022

View reviewed changes

ChristianSi requested changes May 4, 2022

View reviewed changes

Comment thread CHANGELOG.md Outdated

Revert unrelated change in CHANGELOG.md

5deb952

ChristianSi approved these changes May 5, 2022

View reviewed changes

marzer mentioned this pull request Jun 20, 2022

Add a pre-commit configuration file and adopt Prettier #910

Merged

pradyunsg approved these changes Jul 30, 2022

View reviewed changes

pradyunsg merged commit 61ad52a into toml-lang:main Jul 30, 2022

eksortso deleted the unicode_escapes branch July 30, 2022 15:42

awvwgk mentioned this pull request Jul 30, 2022

Support \x escape code in strings toml-f/toml-f#97

Closed

arp242 mentioned this pull request Jul 30, 2022

Add test for \xHH escape codes toml-lang/toml-test#121

Closed

hukkin mentioned this pull request Aug 5, 2022

Add \xHH Unicode escape code to basic strings hukkin/tomli#202

Merged

cyyynthia added a commit to squirrelchat/smol-toml that referenced this pull request Dec 21, 2025

feat: support \xHH unicode escape in strings

2f64496

toml-lang/toml#796

DecimalTurn mentioned this pull request Jan 20, 2026

Update to support TOML v1.1 DecimalTurn/toml-patch#91

Closed

10 tasks

jcmfernandes mentioned this pull request Feb 12, 2026

feat: support for TOML v1.1.0 pelletier/go-toml#1031

Open

roberth mentioned this pull request Feb 13, 2026

Update Request: builtins.fromTOML parser NixOS/nix#15129

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add \xHH Unicode escape code to basic strings#796

Add \xHH Unicode escape code to basic strings#796
pradyunsg merged 4 commits intotoml-lang:mainfrom
eksortso:unicode_escapes

eksortso commented Dec 19, 2020

Uh oh!

marzer commented Nov 5, 2021

Uh oh!

pradyunsg commented Nov 5, 2021

Uh oh!

marzer commented Nov 16, 2021

Uh oh!

eksortso commented Nov 17, 2021

Uh oh!

marzer commented Nov 17, 2021

Uh oh!

eksortso commented Nov 18, 2021

Uh oh!

eksortso commented Nov 18, 2021

Uh oh!

eksortso commented May 3, 2022

Uh oh!

ChristianSi left a comment

Uh oh!

Uh oh!

eksortso commented Jun 6, 2022

Uh oh!

pradyunsg commented Jul 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

eksortso commented Dec 19, 2020

Uh oh!

marzer commented Nov 5, 2021

Uh oh!

pradyunsg commented Nov 5, 2021

Uh oh!

marzer commented Nov 16, 2021

Uh oh!

eksortso commented Nov 17, 2021

Uh oh!

marzer commented Nov 17, 2021

Uh oh!

eksortso commented Nov 18, 2021

Uh oh!

eksortso commented Nov 18, 2021

Uh oh!

eksortso commented May 3, 2022

Uh oh!

ChristianSi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eksortso commented Jun 6, 2022

Uh oh!

pradyunsg commented Jul 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants