Skip to content

codegen: CRLF and lone CR line endings (#230, line-ending part) - #252

Merged
luisleo526 merged 7 commits into
mainfrom
cx/iss-230
Oct 25, 2026
Merged

luisleo526 merged 7 commits into
mainfrom
cx/iss-230

Conversation

@luisleo526

Copy link
Copy Markdown
Contributor

Fixes #230 (CRLF / CR part). The lexer folds \r\n to \n (also inside strings) and treats a lone \r as a line ending in code and comments, with the same output and line numbers as the LF form. A lone \r inside a string literal stays data. The normalized source feeds pragma extraction and the parser too; direct Lexer callers share the rule (docs in AGENTS.md / CLAUDE.md contract table + CHANGELOG).

Tests: tests/test_line_endings.py (expectations computed from the LF form of the same script) + glue/version-directive updates. Reviewed (REV-CX GO, four rounds). Part 1 of #230 (:= on a UDT field in a single-line function body) is a separate PR.

🤖 Generated with Claude Code

luisleo526 and others added 5 commits October 25, 2026 20:12
Normalize CRLF and lone CR code/comment line endings in Lexer initialization after checking the original source size. Construct that lexer at the front of _parse_bounded and pass lexer.source to pragma extraction and Parser, so transpile, transpile_full, the request inventory API, and JSON/Pyodide glue use the same text and locations. Keeping the rule in Lexer also covers direct token users and expression reparsing.

Preserve quoted string contents, including raw CR characters, escaped delimiters and triple-quoted strings. LF source passes through unchanged. Add an Unreleased changelog entry and update the matching project memory.

Regression tests are in the preceding tests-only commit. No tests or builds were run on this Mac, as instructed; validation here was static Python syntax parsing and git diff checks only.

Signed-off-by: luisleo526 <[email protected]>
A CRLF pair is a line terminator wherever it appears, including inside a
multiline string, a wrapped one-quote string and right after a backslash, so
a script saved with CRLF now yields the same string values, C++ and
diagnostics as the LF form. A lone CR inside a string stays data (the
committed regression tests pin it); lone-CR-only files with a multiline or
wrapped string are a documented limit.

Also build the pragma pre-pass lexer inside its try, so a whole-source size
error keeps the recovery it had before the size check moved into the lexer.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…#230)

A global replace of CRLF turned "\r\r\n" (a lone CR followed by a CRLF, the
artifact of a file converted twice) into CR+LF, which the scanner then read
as one break where the LF reading has two: a blank line was lost, later
diagnostic line numbers were one short and a comment line merged with the
next. The CRLF pair is now folded only inside the scanner's string states
(and right after a backslash), and outside strings the existing branch takes
CRLF and a lone CR as one break each, in order. A CRLF inside a one-quote
string applies the same unwrapped-line recovery as the LF form, and that
recovery also treats a following CRLF blank line as blank.

Tests compare tokens, C++ and diagnostics with the LF form of the same
script for "\r\r\n" after code, after a comment and inside both string kinds.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings the branch up to date with codegen main (CHANGELOG backfill, #248). The
merge is clean: CHANGELOG.md auto-merged and keeps both main's entries and the
Unreleased note for CRLF and CR line endings (#230).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 and others added 2 commits October 25, 2026 23:04
The two double-CRLF translate tests used scripts without //@Version=6, so the
LF form was refused for the missing directive before it could be compared. They
now start with the directive like the other tests in the file, and the
line-three diagnostics test does too, so its syntax error is the one compared
(not the missing-directive refusal).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526
luisleo526 merged commit 063c4e3 into main Oct 25, 2026
16 checks passed
luisleo526 added a commit that referenced this pull request Oct 26, 2026
## What

Replaces the three-bullet `## Unreleased` block of `CHANGELOG.md` with the full planned-1.8.0 entry and its migration notes. `CHANGELOG.md` is the only file changed (+54/-10). Documentation only.

## Why

The release workflow creates the GitHub release with a fixed install line and no migration notes; the notes users read are this file. They must be on main before the release dispatch. The heading stays `## Unreleased` until the `v1.8.0` tag exists, as the file's own release note policy says; a separate docs-only commit dates it after the tag.

## Content

- Compatibility and migration: exact pairing with engine `v1.8.0`; regenerate strategy C++ and relink; the new `color` input type with its unsigned packed `0xAARRGGBB` default that hosts must preserve; the `strategy.account_currency` emitted-C++ change (intended to change only for scripts that read it; not a byte comparison against 1.7.0); CRLF and lone CR line endings accepted.
- The four pull requests merged after `v1.7.0`: #251 (#231), #250 (#227), #252 (#230), #248 (CHANGELOG backfill, no behaviour change).
- One sentence about pineforge-hpo accepting color inputs from its 1.8.0 release; true of hpo main today, re-checked against hpo's 1.8.0 entry before the release.

## Checks

- Read as a committed diff by an independent reviewer: GO. The four commits since `v1.7.0` are the four bullets.
- The `color.red` value `4294063685` (`0xFFF23645`) was recomputed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Supersedes #256 (same change; the commit message no longer carries an internal tracking id).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Front end: := on a UDT field in a single-line function body, and CRLF line endings

1 participant