Skip to content

gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin#94386

Merged
pablogsal merged 2 commits intopython:mainfrom
pablogsal:gh-94360
Jul 5, 2022
Merged

gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin#94386
pablogsal merged 2 commits intopython:mainfrom
pablogsal:gh-94360

Conversation

@pablogsal
Copy link
Member

@pablogsal pablogsal commented Jun 28, 2022 •
edited
Loading

@pablogsal pablogsal requested a review from lysnikolaou as a code owner June 28, 2022 13:21
@pablogsal pablogsal added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Jun 28, 2022
@pablogsal
Copy link
Member Author

pablogsal commented Jun 28, 2022 •
edited
Loading

I'm not adding a test (for now) because this doesn't reproduce nicely on the test suite.

@pablogsal pablogsal marked this pull request as draft June 28, 2022 13:46
@pablogsal
pythongh-94360: Fix a tokenizer crash when reading encoded files with…
a7cf92c 
… syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>
@pablogsal pablogsal marked this pull request as ready for review June 28, 2022 15:23
@zware
Copy link
Member

zware commented Jun 28, 2022

Confirmed that this resolves the crash as produced by my reduced reproducer in main and 3.11, and without the change to Parser/pegen_errors.c (which doesn't exist in 3.10) it also resolves both reproducers for 3.10.

@pablogsal
Copy link
Member Author

pablogsal commented Jun 28, 2022

Thanks for checking @zware ! ♥️

ambv
ambv reviewed Jul 5, 2022
View reviewed changes
Parser/pegen_errors.c
const char* buf_end = p->tok->fp_interactive ? p->tok->interactive_src_end : p->tok->inp;

for (int i = 0; i < relative_lineno - 1; i++) {
char *new_line = strchr(cur_line, '\n') + 1;
Copy link
Contributor

@ambv ambv Jul 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NULL + 1 is beautifully strongly typed, huh? 😎

Copy link
Member Author

@pablogsal pablogsal Jul 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6lujn3.jpg

ambv
ambv reviewed Jul 5, 2022
View reviewed changes
@pablogsal pablogsal merged commit 36fcde6 into python:main Jul 5, 2022
@pablogsal pablogsal deleted the gh-94360 branch July 5, 2022 16:39
@miss-islington
Copy link
Contributor

miss-islington commented Jul 5, 2022

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

@bedevere-bot
Copy link

bedevere-bot commented Jul 5, 2022

GH-94573 is a backport of this pull request to the 3.11 branch.

@miss-islington
Copy link
Contributor

miss-islington commented Jul 5, 2022

Sorry, @pablogsal, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 36fcde61ba48c4e918830691ecf4092e4e3b9b99 3.10

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 5, 2022
@pablogsal @miss-islington
pythongh-94360: Fix a tokenizer crash when reading encoded files with…
2012ce5 
… syntax errors from stdin (pythonGH-94386)

* pythongh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>

* nitty nit

Co-authored-by: Łukasz Langa <[email protected]>
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
miss-islington added a commit that referenced this pull request Jul 5, 2022
@miss-islington @pablogsal
gh-94360: Fix a tokenizer crash when reading encoded files with synta…
d49c99f 
…x errors from stdin (GH-94386)

* gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>

* nitty nit

Co-authored-by: Łukasz Langa <[email protected]>
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
@bedevere-bot
Copy link

bedevere-bot commented Jul 5, 2022

GH-94574 is a backport of this pull request to the 3.10 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label Jul 5, 2022
pablogsal added a commit to pablogsal/cpython that referenced this pull request Jul 5, 2022
@pablogsal
[3.10] pythongh-94360: Fix a tokenizer crash when reading encoded fil…
005fc4b 
…es with syntax errors from stdin (pythonGH-94386)

* pythongh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>

* nitty nit

Co-authored-by: Łukasz Langa <[email protected]>.
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
ambv pushed a commit that referenced this pull request Jul 5, 2022
@pablogsal
[3.10] gh-94360: Fix a tokenizer crash when reading encoded files wit…
697e78c 
…h syntax errors from stdin (GH-94386) (GH-94574)

Signed-off-by: Pablo Galindo <[email protected]>
Co-authored-by: Pablo Galindo Salgado <[email protected]>
Co-authored-by: Łukasz Langa <[email protected]>

(cherry picked from commit 36fcde6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ambv ambv ambv left review comments

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Assignees

@pablogsal pablogsal

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Tokenizer crash when redirecting input to stdin

5 participants

@pablogsal @zware @miss-islington @bedevere-bot @ambv