Issue with stdio_encoding being set to None

@embray

@embray - my Jenkins instance is showing:

+ /Users/Shared/Jenkins/virtualenvs/astropy/python2.7-numpy1.6.2/bin/python setup.py build
Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    import ah_bootstrap
  File "/Users/Shared/Jenkins/Home/workspace/astropy-master-osx-10.8-multiconfig/NV/1.6.2/PV/2.7/ah_bootstrap.py", line 716, in <module>
    _main()
  File "/Users/Shared/Jenkins/Home/workspace/astropy-master-osx-10.8-multiconfig/NV/1.6.2/PV/2.7/ah_bootstrap.py", line 713, in _main
    use_astropy_helpers(**kwargs)
  File "/Users/Shared/Jenkins/Home/workspace/astropy-master-osx-10.8-multiconfig/NV/1.6.2/PV/2.7/ah_bootstrap.py", line 202, in use_astropy_helpers
    is_submodule = _check_submodule(path, use_git=use_git)
  File "/Users/Shared/Jenkins/Home/workspace/astropy-master-osx-10.8-multiconfig/NV/1.6.2/PV/2.7/ah_bootstrap.py", line 392, in _check_submodule
    return _check_submodule_using_git(path)
  File "/Users/Shared/Jenkins/Home/workspace/astropy-master-osx-10.8-multiconfig/NV/1.6.2/PV/2.7/ah_bootstrap.py", line 443, in _check_submodule_using_git
    stdout = stdout.decode(stdio_encoding)
TypeError: decode() argument 1 must be string, not None

It seems we should have some kind of fallback for this case? We can still fix this for astropy 0.4 and not have to release a new astropy-helpers, but we should fix it in the ah_bootstrap.py there too of course.

That's weird. Does it only happen in Jenkins? Really this should be reported against astropy-helpers. If need be 0.4.1 can be released with a fix. It's not a big deal.

@embray

@embray - right, I reported it here to mark it as a potential 0.4 blocker if we decided it was. I can close and open there if we decide it's not 0.4 crucial. It only happens in Jenkins, maybe because some environment variable isn't set, or because it's not running in an interactive terminal.

I don't think it is. In Jenkins try setting LC_ALL=en_US.UTF-8.

Reverse testing - if I don't set LC_ALL on my Mac, I get the issue.

I guess the issue is that I explicitly set that because of previous issues, but I'm worried about people who might get that error if they don't have it set. At the very least we should make the error more explicit and tell people to set it, right? But if possible, we should maybe assume some sensible default.

On a clean mac (default environment variables), I get:

Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    import ah_bootstrap
  File "/Users/tom/Dropbox/Code/development/Astropy/astropy/ah_bootstrap.py", line 716, in <module>
    _main()
  File "/Users/tom/Dropbox/Code/development/Astropy/astropy/ah_bootstrap.py", line 713, in _main
    use_astropy_helpers(**kwargs)
  File "/Users/tom/Dropbox/Code/development/Astropy/astropy/ah_bootstrap.py", line 202, in use_astropy_helpers
    is_submodule = _check_submodule(path, use_git=use_git)
  File "/Users/tom/Dropbox/Code/development/Astropy/astropy/ah_bootstrap.py", line 392, in _check_submodule
    return _check_submodule_using_git(path)
  File "/Users/tom/Dropbox/Code/development/Astropy/astropy/ah_bootstrap.py", line 432, in _check_submodule_using_git
    stdio_encoding = locale.getdefaultlocale()[1]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
    return _parse_localename(localename)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8

I know this is MacOS X's fault, but it might be worth setting a sensible default to avoid things blowing up for too many users, right?

Try LC_CTYPE=en_US.UTF8 (in addition)

I agree though.

See for reference http://bugs.python.org/issue18378

Sorry, to be clear setting those environment variables on my mac does work, which is why I hadn't seen the issue before. I tried making sure my .bash_profile didn't run and then it crashed in the same way as Jenkins. I'll try setting the env variables on Jenkins now, but I'm sure it'll work.

In any case, I'll fix this to have a fallback at least.

Just to confirm, setting LC_CTYPE=en_US.UTF8 in Jenkins first works. Note that there are two places where the locale seems to be needed. On line 432 of ah_bootstrap.py, but then also when importing the setup_helpers it seems that sphinx import docutils which tries to get the locale and crashes on Mac... Not sure that this one can be fixed :-( On a plain Mac:

In [1]: import docutils.io
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-577063d5ea4d> in <module>()
----> 1 import docutils.io

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/docutils/io.py in <module>()
     16 from docutils import TransformSpec
     17 from docutils._compat import b
---> 18 from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput
     19 
     20 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/docutils/utils/__init__.py in <module>()
     19 from docutils import nodes
     20 import docutils.io
---> 21 from docutils.utils.error_reporting import ErrorOutput, SafeString
     22 
     23 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/docutils/utils/error_reporting.py in <module>()
     45     locale_encoding = None
     46 else:
---> 47     locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
     48     # locale.getpreferredencoding([do_setlocale=True|False])
     49     # has side-effects | might return a wrong guess.

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.pyc in getdefaultlocale(envvars)
    541     else:
    542         localename = 'C'
--> 543     return _parse_localename(localename)
    544 
    545 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.pyc in _parse_localename(localename)
    473     elif code == 'C':
    474         return None, None
--> 475     raise ValueError, 'unknown locale: %s' % localename
    476 
    477 def _build_localename(localetuple):

ValueError: unknown locale: UTF-8

So there are two things going on. On the clean Jenkins instance, it seems the encoding gets set to None which we can fix, but on a Mac with no env variables customized, we may not be able to fix the issue because docutils doesn't import. Ugh.

It seems that I have the same problem when trying to build the 0.4 release for Debian...

@olebole

@olebole - what is the output of locale on Debian?

The solution remains to set the appropriate env variables.

In the build process, locale gives

LANG=C
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

Setting LC_ALL=C.UTF-8 helps. With en_EN-UTF-8, the debian build helpers (perl) complain about missing locale support.

@olebole

@olebole If it makes things easier you could also just include this patch for ah_bootstrap.py:

diff --git a/ah_bootstrap.py b/ah_bootstrap.py
index 8c0b11b..b191056 100644
--- a/ah_bootstrap.py
+++ b/ah_bootstrap.py
@@ -429,32 +429,50 @@ def _check_submodule_using_git(path):
                 '`git submodule status` command:\n{0}'.format(str(e)))


-    stdio_encoding = locale.getdefaultlocale()[1]
+    # Can fail of the default locale is not configured properly.  See
+    # https://github.com/astropy/astropy/issues/2749.  For the purposes under
+    # consideration 'latin1' is an acceptable fallback.
+    try:
+        stdio_encoding = locale.getdefaultlocale()[1] or 'latin1'
+    except ValueError:
+        # Due to an OSX oddity locale.getdefaultlocale() can also crash
+        # depending on the user's locale/language settings.  See:
+        # http://bugs.python.org/issue18378
+        stdio_encoding = 'latin1'

Sorry for making this difficult =_=

Wait, sorry, wrong patch. Hold on--

Updated the patch in this comment to the correct one: #2749 (comment)

I'm seeing the same thing when building the RPM for Fedora. I will test the patch

With #2825 and #2827 merged, along with the "known issues" entry this can be considered fixed.

Merge pull request astropy#58 from embray/issue-2749

Navigation Menu

Repository navigation

Issue with `stdio_encoding` being set to `None` #2749

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with stdio_encoding being set to None #2749

Description

Activity

embray commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

embray commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

embray commented on Jul 15, 2014

embray commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

embray commented on Jul 15, 2014

astrofrog commented on Jul 15, 2014

14 remaining items

olebole commented on Jul 16, 2014

astrofrog commented on Jul 16, 2014

embray commented on Jul 16, 2014

olebole commented on Jul 17, 2014

embray commented on Jul 17, 2014

embray commented on Jul 17, 2014

embray commented on Jul 17, 2014

sergiopasra commented on Jul 29, 2014

embray commented on Aug 6, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Issue with `stdio_encoding` being set to `None` #2749