Fix UnicodeEncodeError when string comparison with unicode has failed. by AiOO · Pull Request #1870 · pytest-dev/pytest

AiOO · 2016-08-25T15:59:15Z

Thanks for submitting a PR, your contribution is really appreciated!

Here's a quick checklist that should be present in PRs:

[v] Target: for bug or doc fixes, target master; for new features, target features;
[v] Make sure to include one or more tests for your change;
[v] Add yourself to AUTHORS;
[v] Add a new entry to CHANGELOG.rst
- Choose any open position to avoid merge conflicts with other PRs.
- Add a link to the issue you are fixing (if any) using RST syntax.
- The pytest team likes to have people to acknowledged in the CHANGELOG, so please add a thank note to yourself ("Thanks @user for the PR") and a link to your GitHub profile. It may sound weird thanking yourself, but otherwise a maintainer would have to do it manually before or after merging instead of just using GitHub's merge button. This makes it easier on the maintainers to merge PRs.

UnicodeEncodeError is occurred when comparing two strings with any unicode character. This PR fixes this error.

Relative issue: #1864.

Error reproduction:

$ cat test.py
# -*- coding: utf-8 -*-
def test_with_unicode_literal():
    assert u'테스트' == u'Test'

$ py.test test.py
==================================================== test session starts =====================================================
platform darwin -- Python 2.7.12, pytest-3.0.0, py-1.4.31, pluggy-0.3.1
rootdir: /Users/bright/Playground/test_test, inifile:
collected 1 items
test.py
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/main.py", line 96, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/main.py", line 131, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/main.py", line 152, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 253, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/runner.py", line 79, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/runner.py", line 133, in call_and_report
INTERNALERROR>     call = call_runtest_hook(item, when, **kwds)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/runner.py", line 151, in call_runtest_hook
INTERNALERROR>     return CallInfo(lambda: ihook(item=item, **kwds), when=when)
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/runner.py", line 168, in __init__
INTERNALERROR>     self.excinfo = ExceptionInfo()
INTERNALERROR>   File "/REDUCED/lib/python2.7/site-packages/_pytest/_code/code.py", line 357, in __init__
INTERNALERROR>     exprinfo = str(tup[1])
INTERNALERROR> UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-10: ordinal not in range(128)
================================================ no tests ran in 0.01 seconds ================================================

coveralls · 2016-08-25T16:15:38Z

Coverage decreased (-0.02%) to 93.006% when pulling 6cbcc35 on AiOO:bugfix/assertion-with-unicode into 9c45d6c on pytest-dev:master.

The-Compiler · 2016-08-25T16:16:39Z

+                    try:
+                        exprinfo = str(tup[1])
+                    except UnicodeEncodeError:
+                        exprinfo = unicode(tup[1])


Looking at tup[1], it always seems to contain an unicode object, even when there are byte strings in the assertion:

-> exprinfo = str(tup[1]) (Pdb) tup[1] AssertionError(u"assert 'a' == 'b'\n - a\n + b",)

So I think the proper fix would be to always use unicode() and do the check with u'assert ' below.

Actually, tup[1] is an AssertionError instance containing a unicode string. I think the proper fix is actually using py._builtin._totext:

diff --git a/_pytest/_code/code.py b/_pytest/_code/code.py index 040e4fc..9a6d326 100644 --- a/_pytest/_code/code.py +++ b/_pytest/_code/code.py @@ -354,7 +354,7 @@ class ExceptionInfo(object): if exprinfo is None and isinstance(tup[1], AssertionError): exprinfo = getattr(tup[1], 'msg', None) if exprinfo is None: - exprinfo = str(tup[1]) + exprinfo = py._builtin._totext(tup[1]) if exprinfo and exprinfo.startswith('assert '): self._striptext = 'AssertionError: ' self._excinfo = tup

py._builtin._totext is just unicode in Python 2 and str in Python 3. With the above change the test passes to me.

coveralls · 2016-08-25T16:25:13Z

Coverage decreased (-0.02%) to 93.006% when pulling 6cbcc35 on AiOO:bugfix/assertion-with-unicode into 9c45d6c on pytest-dev:master.

nicoddemus · 2016-08-26T00:09:24Z

 * Add ``buffer`` attribute to stdin stub class ``pytest.capture.DontReadFromInput``
  Thanks `@joguSD`_ for the PR.

+* Fix UnicodeEncodeError when string comparison with unicode is failed. (`#1864`_)


Nitpick: "has failed" or "fails" 😁

Also: Use double backticks around UnicodeEncodeError so it appears in a monospaced font (see buffer on the item above this one).

AiOO · 2016-08-26T00:51:42Z

Thanks for your kind review! I've updated this PR.

Use proper method py._builtin._totext instead of str or unicode.
Fix typo error and use backticks.

And,

do the check with u'assert ' below.

I think this fix is unnecessary because below code runs well:

Python 2.7.12 (default, Jun 29 2016, 14:05:02)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> u'assert: 유니코드'.startswith('assert: ')
True

coveralls · 2016-08-26T01:01:11Z

Coverage remained the same at 93.027% when pulling 86b8801 on AiOO:bugfix/assertion-with-unicode into 9c45d6c on pytest-dev:master.

nicoddemus · 2016-08-26T01:01:50Z

do the check with u'assert ' below.
I think this fix is unnecessary because below code runs well:

I agree. 😁

Thanks for the update, I will merge as soon as CI passes.

nicoddemus · 2016-08-26T01:04:06Z

Oh, just a tip: on GitHub if you add a string like "Fixes #XXX" or "Resolves #XXX" to the commit message it will automatically close the issue #XXX when merged. See here. 😉

nicoddemus · 2016-08-26T01:29:39Z

Thanks again for the PR!

AiOO force-pushed the bugfix/assertion-with-unicode branch from 5e37be7 to 6cbcc35 Compare August 25, 2016 16:05

The-Compiler reviewed Aug 25, 2016
View reviewed changes

nicoddemus reviewed Aug 26, 2016
View reviewed changes

AiOO added 2 commits August 26, 2016 09:41

Fix UnicodeEncodeError when string comparison with unicode has failed.

856ad71

Update AUTHORS and CHANGELOG

86b8801

AiOO force-pushed the bugfix/assertion-with-unicode branch from 6cbcc35 to 86b8801 Compare August 26, 2016 00:51

AiOO changed the title ~~Fix UnicodeEncodeError when string comparison with unicode is failed.~~ Fix UnicodeEncodeError when string comparison with unicode has failed. Aug 26, 2016

nicoddemus merged commit a947e83 into pytest-dev:master Aug 26, 2016

nicoddemus mentioned this pull request Aug 26, 2016

Failure when traceback includes unicode strings #1864

Closed

The-Compiler mentioned this pull request Aug 27, 2016

UnicodeEecodeError while running hypothesis #1880

Closed

This was referenced Aug 31, 2016

pytest 3.0.+ and unicode #1894

Closed

Tests, failed on comparing unicoded values, cause an internal error (instead of Assertion) #1899

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix UnicodeEncodeError when string comparison with unicode has failed.#1870

Fix UnicodeEncodeError when string comparison with unicode has failed.#1870
nicoddemus merged 2 commits intopytest-dev:masterfrom
AiOO:bugfix/assertion-with-unicode

AiOO commented Aug 25, 2016

Uh oh!

coveralls commented Aug 25, 2016

Uh oh!

The-Compiler Aug 25, 2016

Uh oh!

nicoddemus Aug 26, 2016

Uh oh!

coveralls commented Aug 25, 2016

Uh oh!

nicoddemus Aug 26, 2016

Uh oh!

AiOO commented Aug 26, 2016

Uh oh!

coveralls commented Aug 26, 2016

Uh oh!

nicoddemus commented Aug 26, 2016

Uh oh!

nicoddemus commented Aug 26, 2016

Uh oh!

nicoddemus commented Aug 26, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

AiOO commented Aug 25, 2016

Uh oh!

coveralls commented Aug 25, 2016

Uh oh!

The-Compiler Aug 25, 2016

Choose a reason for hiding this comment

Uh oh!

nicoddemus Aug 26, 2016

Choose a reason for hiding this comment

Uh oh!

coveralls commented Aug 25, 2016

Uh oh!

nicoddemus Aug 26, 2016

Choose a reason for hiding this comment

Uh oh!

AiOO commented Aug 26, 2016

Uh oh!

coveralls commented Aug 26, 2016

Uh oh!

nicoddemus commented Aug 26, 2016

Uh oh!

nicoddemus commented Aug 26, 2016

Uh oh!

nicoddemus commented Aug 26, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants