Skip to content

[fix] Handle CR record endings in CSV readers - #10657

Merged
sannya-singal merged 2 commits into
agno-agi:mainfrom
lakers-abing:codex/fix-csv-universal-newlines
Sep 30, 2026
Merged

sannya-singal merged 2 commits into
agno-agi:mainfrom
lakers-abing:codex/fix-csv-universal-newlines

Conversation

@lakers-abing

Copy link
Copy Markdown
Contributor

Summary

Fix CSV ingestion silently returning no documents for CR-delimited records in CSVReader and FieldLabeledCSVReader.

Both readers already open synchronous path inputs with newline="", but their in-memory wrappers use the default StringIO newline mode. With CR-only record separators, csv.reader raises new-line character seen in unquoted field; the readers catch it and return []. This affects binary/text uploads in both modes and path inputs in async mode.

Use newline="" for the six StringIO wrappers, matching the existing path handling. This recognizes LF, CRLF, and CR record boundaries while preserving quoted multiline cells for the existing cell normalization.

Minimal reproduction (no network, model, or database needed):

import asyncio
from io import BytesIO
from agno.knowledge.reader.csv_reader import CSVReader

payload = b"name,city\rAlice,Paris\rBob,Rome\r"
reader = CSVReader(chunk=False)
print(reader.read(BytesIO(payload)))
print(asyncio.run(reader.async_read(BytesIO(payload))))

Before: both calls log a CSV error and return []. After: each returns a document containing all three rows.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Improvement
  • Model update
  • Other:

Checklist

  • Code complies with style guidelines
  • Ran format/validation scripts (./scripts/format.sh and ./scripts/validate.sh)
  • Self-review completed
  • Documentation updated (comments, docstrings)
  • Examples and guides: Relevant cookbook examples have been included or updated (if applicable)
  • Tested in clean environment
  • Tests added/updated (if applicable)

Duplicate and AI-Generated PR Check

  • I have searched existing open pull requests and confirmed that no other PR already addresses this issue
  • If a similar PR exists, I have explained below why this PR is a better approach
  • Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.)

Additional Notes

Generated by Codex with the account holder's authorization. Codex performed the diff review, reproduction, and checks described here. No claim of independent human code review is made.

Regression coverage includes both readers, sync/async calls, path/bytes/text inputs, LF/CRLF/CR separators, quoted multiline cells, small and paginated inputs, row order, names, and caller-owned stream lifetime. On upstream 476f61b000ef15a77ab3a6851b2c6cefbc7571c9, the new test matrix reports 20 failed, 52 passed. After the fix, it reports 72 passed; combined with both existing CSV reader suites, 171 passed.

Validation on macOS ARM64, Python 3.12.14, pytest 9.1.1, Ruff 0.15.20, mypy 2.1.0, SQLAlchemy 2.0.54:

pytest libs/agno/tests/unit/reader/test_csv_newlines.py libs/agno/tests/unit/reader/test_csv_reader.py libs/agno/tests/unit/reader/test_csv_field_label_reader.py -q
# 171 passed
./scripts/format.sh
# passed; no unrelated formatting changes
./scripts/validate.sh
# passed: Agno (1062 source files), agnoctl (21), cookbook lint and pattern checks
git diff --check
# passed

The initial development dependency resolution installed SQLAlchemy 2.1.1, under which full validation reported four type errors in unchanged database/public-limit modules. Validation passes with SQLAlchemy 2.0.54, within the project's declared dependency range. This PR does not alter dependency constraints or those modules. The entire repository test suite and live integrations were not run.

No new public API or cookbook is needed; the regression test documents the expected newline behavior.

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Missing issue link: Please link the issue this PR addresses using fixes #<issue_number>, closes #<issue_number>, or resolves #<issue_number> in the PR description. If there is no existing issue, please create one first.

@sannya-singal
sannya-singal merged commit 4db4180 into agno-agi:main Sep 30, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants