Repository navigation
[fix] Handle CR record endings in CSV readers - #10657
Merged
sannya-singal merged 2 commits intoSep 30, 2026
Merged
Conversation
Contributor
PR TriageMissing issue link: Please link the issue this PR addresses using |
sannya-singal
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix CSV ingestion silently returning no documents for CR-delimited records in
CSVReaderandFieldLabeledCSVReader.Both readers already open synchronous path inputs with
newline="", but their in-memory wrappers use the defaultStringIOnewline mode. With CR-only record separators,csv.readerraisesnew-line character seen in unquoted field; the readers catch it and return[]. This affects binary/text uploads in both modes and path inputs in async mode.Use
newline=""for the sixStringIOwrappers, matching the existing path handling. This recognizes LF, CRLF, and CR record boundaries while preserving quoted multiline cells for the existing cell normalization.Minimal reproduction (no network, model, or database needed):
Before: both calls log a CSV error and return
[]. After: each returns a document containing all three rows.Type of change
Checklist
./scripts/format.shand./scripts/validate.sh)Duplicate and AI-Generated PR Check
Additional Notes
Generated by Codex with the account holder's authorization. Codex performed the diff review, reproduction, and checks described here. No claim of independent human code review is made.
Regression coverage includes both readers, sync/async calls, path/bytes/text inputs, LF/CRLF/CR separators, quoted multiline cells, small and paginated inputs, row order, names, and caller-owned stream lifetime. On upstream
476f61b000ef15a77ab3a6851b2c6cefbc7571c9, the new test matrix reports 20 failed, 52 passed. After the fix, it reports 72 passed; combined with both existing CSV reader suites, 171 passed.Validation on macOS ARM64, Python 3.12.14, pytest 9.1.1, Ruff 0.15.20, mypy 2.1.0, SQLAlchemy 2.0.54:
The initial development dependency resolution installed SQLAlchemy 2.1.1, under which full validation reported four type errors in unchanged database/public-limit modules. Validation passes with SQLAlchemy 2.0.54, within the project's declared dependency range. This PR does not alter dependency constraints or those modules. The entire repository test suite and live integrations were not run.
No new public API or cookbook is needed; the regression test documents the expected newline behavior.