Hitting _csv.Error: field larger than field limit (131072)
#229
Labels
bug
Something isn't working
Comments
|
Same issue as #227. |
|
I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving |
|
I'm going to use this pattern from https://stackoverflow.com/a/15063941 import sys
import csv
maxInt = sys.maxsize
while True:
# decrease the maxInt value by factor 10
# as long as the OverflowError occurs.
try:
csv.field_size_limit(maxInt)
break
except OverflowError:
maxInt = int(maxInt/10) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a csv file where one of the fields is so large it is throwing an exception with this error and stops loading:
_csv.Error: field larger than field limit (131072)The stack trace occurs here: https://github.com/simonw/sqlite-utils/blob/3.1/sqlite_utils/cli.py#L633
There is a way to handle this that helps:
https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072
One issue I had with this problem was sqlite-utils only provides limited context as to where the problem line is.
There is the progress bar, but that is by percent rather than by line number. It would have been helpful if it could have provided a line number.
Also, it would have been useful if it had allowed the loading to continue with later lines.
The text was updated successfully, but these errors were encountered: