Skip to content

Make pprint/pformat tolerant of non-UTF-8 bytes data - #6117

Merged
bsipocz merged 3 commits into
astropy:masterfrom
taldcroft:table-decode-replace
May 27, 2017
Merged

Make pprint/pformat tolerant of non-UTF-8 bytes data#6117
bsipocz merged 3 commits into
astropy:masterfrom
taldcroft:table-decode-replace

Conversation

@taldcroft

@taldcroft taldcroft commented May 25, 2017

Copy link
Copy Markdown
Member

Currently if a bytestring column has content that is not UTF-8 compliant you cannot get the repr of that column:

In [3]: t = Table([[b'Z\xf0']])

In [4]: t
Out[4]: ---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
...
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf0 in position 1: unexpected end of data

This fixes that by decoding with errors='replace'.

@taldcroft taldcroft mentioned this pull request May 25, 2017
Merged
@taldcroft taldcroft added Bug and removed Enhancement labels May 25, 2017
@taldcroft taldcroft added this to the v1.3.3 milestone May 25, 2017
bsipocz
bsipocz reviewed May 25, 2017
View reviewed changes
Comment thread CHANGES.rst Outdated

- ``astropy.tests``

- Fix problem with Table pprint/pformat raising an exception

@bsipocz bsipocz May 25, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it astropy.table rather than astropy.tests?

@pllim pllim May 25, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, should be table.

@taldcroft taldcroft May 25, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do'oh! Fixed.

@mhvk

mhvk commented May 26, 2017

Copy link
Copy Markdown
Contributor

Looks OK in general, though should it be namereplace_errors for Python >=3.5? (we'll be there soon enough...). Also, how does this interact with #5700 - do we still get bytes there? With that in place, should any bytes just be typeset as b'...'?

I guess my bottom line: definitely for 1.3.3, but let's be sure we get it right for 2.0.

@taldcroft

taldcroft commented May 27, 2017

Copy link
Copy Markdown
Member Author

It looks to me like namereplace_errors() is only for encoding.

@bsipocz
bsipocz merged commit 0eb551e into astropy:master May 27, 2017
@bsipocz

bsipocz commented May 27, 2017

Copy link
Copy Markdown
Member

Based on the discussion above, this seemed to be agreed on to be good to go into 1.3.3. I'm cutting the release now, thus merging.

bsipocz added a commit that referenced this pull request May 29, 2017
@bsipocz
Merge pull request #6117 from taldcroft/table-decode-replace
c2fd364 
Make pprint/pformat tolerant of non-UTF-8 bytes data
@taldcroft taldcroft mentioned this pull request May 31, 2017
Merged
@taldcroft
taldcroft deleted the table-decode-replace branch October 1, 2019 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pllim pllim pllim left review comments
@bsipocz bsipocz bsipocz left review comments

Assignees

No one assigned

Labels

Bug Ready-for-final-review table

Projects

None yet

Milestone

v1.3.3

Development

Successfully merging this pull request may close these issues.

4 participants

@taldcroft @mhvk @bsipocz @pllim