Failures on Windows with Python 3 (possible fix) #539

mdboom · 2012-12-06T01:52:49Z

This is a shot in the dark to fix #538.

astrofrog · 2012-12-06T07:50:33Z

@mdboom - this seemed to help. One of the Python 3 builds that was failing before now reports

______________________________ test_find_by_hash _______________________________



    @remote_data

    def test_find_by_hash():



        from ..data import get_readable_fileobj, get_pkg_data_filename, clear_download_cache



        import hashlib



        with get_readable_fileobj(TESTURL, cache=True) as googlepage:

>           hash = hashlib.md5(googlepage.read())

E           TypeError: Unicode-objects must be encoded before hashing



astropy\utils\tests\test_data.py:54: TypeError

------------------------------- Captured stdout --------------------------------

is it simply a case of a missing encode()?

astrofrog · 2012-12-06T07:52:21Z

Interestingly, some Python 2 tests that were not failing before are now failing:

_______________________________ test_regression ________________________________

    def test_regression():
>       _test_regression(False)

astropy\io\votable\tests\vo_test.py:176:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_python_based = False

    def _test_regression(_python_based=False):
        # Read the VOTABLE
        votable = parse(
            get_pkg_data_filename('data/regression.xml'),
            pedantic=False,
            _debug_python_based_parser=_python_based)
        table = votable.get_first_table()

        assert table.array.dtype == [
            (('string test', 'string_test'), '|O8'),
            (('fixed string test', 'string_test_2'), '|S10'),
            ('unicode_test', '|O8'),
            (('unicode test', 'fixed_unicode_test'), '<U10'),
            (('string array test', 'string_array_test'), '|S4'),
            ('unsignedByte', '|u1'),
            ('short', '<i2'),
            ('int', '<i4'),
            ('long', '<i8'),
            ('double', '<f8'),
            ('float', '<f4'),
            ('array', '|O8'),
            ('bit', '|b1'),
            ('bitarray', '|b1', (3, 2)),
            ('bitvararray', '|O8'),
            ('bitvararray2', '|O8'),
            ('floatComplex', '<c8'),
            ('doubleComplex', '<c16'),
            ('doubleComplexArray', '|O8'),
            ('doubleComplexArrayFixed', '<c16', (2,)),
            ('boolean', '|b1'),
            ('booleanArray', '|b1', (4,)),
            ('nulls', '<i4'),
            ('nulls_array', '<i4', (2, 2)),
            ('precision1', '<f8'),
            ('precision2', '<f8'),
            ('doublearray', '|O8'),
            ('bitarray2', '|b1', (16,))
            ]

        votable.to_xml(join(TMP_DIR, "regression.tabledata.xml"),
                       _debug_python_based_parser=_python_based)
        assert_validate_schema(join(TMP_DIR, "regression.tabledata.xml"))
        votable.get_first_table().format = 'binary'
        # Also try passing a file handle
        with open(join(TMP_DIR, "regression.binary.xml"), "wb") as fd:
            votable.to_xml(fd, _debug_python_based_parser=_python_based)
        assert_validate_schema(join(TMP_DIR, "regression.binary.xml"))
        # Also try passing a file handle
        with open(join(TMP_DIR, "regression.binary.xml"), "rb") as fd:
            votable2 = parse(fd, pedantic=False,
                             _debug_python_based_parser=_python_based)
        votable2.get_first_table().format = 'tabledata'
        votable2.to_xml(join(TMP_DIR, "regression.bin.tabledata.xml"),
                        _astropy_version="testing",
                        _debug_python_based_parser=_python_based)
        assert_validate_schema(join(TMP_DIR, "regression.bin.tabledata.xml"))

        with io.open(
            get_pkg_data_filename('data/regression.bin.tabledata.truth.xml'),
            'rt', encoding='utf-8') as fd:
            truth = fd.readlines()
        with io.open(
            join(TMP_DIR, "regression.bin.tabledata.xml"),
            'rt', encoding='utf-8') as fd:
            output = fd.readlines()

        # If the lines happen to be different, print a diff
        # This is convenient for debugging
        for line in difflib.unified_diff(truth, output):
            sys.stdout.write(
                line.
                encode('unicode_escape').
                replace('\\n', '\n'))

>       assert truth == output
E       assert [u'<?xml vers... data\n', ...] == [u'<?xml versi... data\n', ...]
E         At index 193 diff: u'      <TD>9999999999999999455752309870428160.000</TD>\n' != u'      <TD>9999999999999999500000000000000000.000</TD>\n'

astropy\io\votable\tests\vo_test.py:159: AssertionError
------------------------------- Captured stdout --------------------------------
WARNING: W39: ?:?:?: W39: Bit values can not be masked [astropy.io.votable.converters]
WARNING: W39: ?:?:?: W39: Bit values can not be masked [astropy.io.votable.converters]
WARNING: E02: None:16:1: E02: Incorrect number of elements in array. Expected multiple of 0, got 1 [astropy.io.votable.converters]
WARNING: W50: None:29:2: W50: Invalid unit string 'foo' [astropy.io.votable.tree]
WARNING: W48: None:46:5: W48: Unknown attribute 'value' on OPTION [astropy.io.votable.tree]
WARNING: E02: None:144:7: E02: Incorrect number of elements in array. Expected multiple of 4, got 1 [astropy.io.votable.converters]
WARNING: W49: None:144:7: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
---
+++
@@ -191,7 +191,7 @@
       <TD>2</TD>
       <TD>-9 0 -9 1</TD>
       <TD>     1e+34</TD>
-      <TD>9999999999999999455752309870428160.000</TD>
+      <TD>9999999999999999500000000000000000.000</TD>
       <TD/>
       <TD>0000000000000000</TD>
      </TR>
_____________________ test_regression_python_based_parser ______________________

    def test_regression_python_based_parser():
>       _test_regression(True)

astropy\io\votable\tests\vo_test.py:180:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_python_based = True

    def _test_regression(_python_based=False):
        # Read the VOTABLE
        votable = parse(
            get_pkg_data_filename('data/regression.xml'),
            pedantic=False,
            _debug_python_based_parser=_python_based)
        table = votable.get_first_table()

        assert table.array.dtype == [
            (('string test', 'string_test'), '|O8'),
            (('fixed string test', 'string_test_2'), '|S10'),
            ('unicode_test', '|O8'),
            (('unicode test', 'fixed_unicode_test'), '<U10'),
            (('string array test', 'string_array_test'), '|S4'),
            ('unsignedByte', '|u1'),
            ('short', '<i2'),
            ('int', '<i4'),
            ('long', '<i8'),
            ('double', '<f8'),
            ('float', '<f4'),
            ('array', '|O8'),
            ('bit', '|b1'),
            ('bitarray', '|b1', (3, 2)),
            ('bitvararray', '|O8'),
            ('bitvararray2', '|O8'),
            ('floatComplex', '<c8'),
            ('doubleComplex', '<c16'),
            ('doubleComplexArray', '|O8'),
            ('doubleComplexArrayFixed', '<c16', (2,)),
            ('boolean', '|b1'),
            ('booleanArray', '|b1', (4,)),
            ('nulls', '<i4'),
            ('nulls_array', '<i4', (2, 2)),
            ('precision1', '<f8'),
            ('precision2', '<f8'),
            ('doublearray', '|O8'),
            ('bitarray2', '|b1', (16,))
            ]

        votable.to_xml(join(TMP_DIR, "regression.tabledata.xml"),
                       _debug_python_based_parser=_python_based)
        assert_validate_schema(join(TMP_DIR, "regression.tabledata.xml"))
        votable.get_first_table().format = 'binary'
        # Also try passing a file handle
        with open(join(TMP_DIR, "regression.binary.xml"), "wb") as fd:
            votable.to_xml(fd, _debug_python_based_parser=_python_based)
        assert_validate_schema(join(TMP_DIR, "regression.binary.xml"))
        # Also try passing a file handle
        with open(join(TMP_DIR, "regression.binary.xml"), "rb") as fd:
            votable2 = parse(fd, pedantic=False,
                             _debug_python_based_parser=_python_based)
        votable2.get_first_table().format = 'tabledata'
        votable2.to_xml(join(TMP_DIR, "regression.bin.tabledata.xml"),
                        _astropy_version="testing",
                        _debug_python_based_parser=_python_based)
        assert_validate_schema(join(TMP_DIR, "regression.bin.tabledata.xml"))

        with io.open(
            get_pkg_data_filename('data/regression.bin.tabledata.truth.xml'),
            'rt', encoding='utf-8') as fd:
            truth = fd.readlines()
        with io.open(
            join(TMP_DIR, "regression.bin.tabledata.xml"),
            'rt', encoding='utf-8') as fd:
            output = fd.readlines()

        # If the lines happen to be different, print a diff
        # This is convenient for debugging
        for line in difflib.unified_diff(truth, output):
            sys.stdout.write(
                line.
                encode('unicode_escape').
                replace('\\n', '\n'))

>       assert truth == output
E       assert [u'<?xml vers... data\n', ...] == [u'<?xml versi... data\n', ...]
E         At index 193 diff: u'      <TD>9999999999999999455752309870428160.000</TD>\n' != u'      <TD>9999999999999999500000000000000000.000</TD>\n'

astropy\io\votable\tests\vo_test.py:159: AssertionError
------------------------------- Captured stdout --------------------------------
WARNING: W17: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:90:2: W17: GROUP element contains more than one DESCRIPTION element [astropy.io.votable.tree]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:104:28: W46: char value is too long for specified length of 10 [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:106:28: W46: unicodeChar value is too long for specified length of 10 [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:107:11: W46: char value is too long for specified length of 4 [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:126:7: E02: Incorrect number of elements in array. Expected multiple of 4, got 1 [astropy.io.votable.converters]
WARNING: W49: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:126:7: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:134:17: W46: char value is too long for specified length of 10 [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:137:17: W46: char value is too long for specified length of 4 [astropy.io.votable.converters]
WARNING: W49: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:141:6: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
WARNING: W01: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:144:18: W01: Array uses commas rather than whitespace [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:160:7: E02: Incorrect number of elements in array. Expected multiple of 16, got 0 [astropy.io.votable.converters]
WARNING: W49: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:160:7: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
WARNING: W49: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:160:7: W49: Empty cell illegal for integer fields. (suppressing further warnings of this type...) [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:166:17: W46: unicodeChar value is too long for specified length of 10 [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:190:7: E02: Incorrect number of elements in array. Expected multiple of 16, got 0 [astropy.io.votable.converters]
WARNING: W01: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:204:13: W01: Array uses commas rather than whitespace [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:206:7: E02: Incorrect number of elements in array. Expected multiple of 6, got 0 [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:214:7: E02: Incorrect number of elements in array. Expected multiple of 4, got 1 [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:220:7: E02: Incorrect number of elements in array. Expected multiple of 16, got 0 [astropy.io.votable.converters]
WARNING: W01: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:234:12: W01: Array uses commas rather than whitespace [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:236:7: E02: Incorrect number of elements in array. Expected multiple of 6, got 0 [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:244:7: E02: Incorrect number of elements in array. Expected multiple of 4, got 1 [astropy.io.votable.converters]
WARNING: E02: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:246:7: E02: Incorrect number of elements in array. Expected multiple of 4, got 1 (suppressing further warnings of this type...) [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:264:28: W46: char value is too long for specified length of 10 [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:266:28: W46: unicodeChar value is too long for specified length of 10 [astropy.io.votable.converters]
WARNING: W46: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/build/lib.win32-2.6/astropy/io/votable/tests/data/regression.xml>:267:11: W46: char value is too long for specified length of 4 [astropy.io.votable.converters]
WARNING: W39: ?:?:?: W39: Bit values can not be masked (suppressing further warnings of this type...) [astropy.io.votable.converters]
WARNING: E02: None:144:12: E02: Incorrect number of elements in array. Expected multiple of 4, got 1 [astropy.io.votable.converters]
WARNING: W49: None:144:12: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
---
+++
@@ -191,7 +191,7 @@
       <TD>2</TD>
       <TD>-9 0 -9 1</TD>
       <TD>     1e+34</TD>
-      <TD>9999999999999999455752309870428160.000</TD>
+      <TD>9999999999999999500000000000000000.000</TD>
       <TD/>
       <TD>0000000000000000</TD>
      </TR>
 generated xml file: <http://brayxp.translab.stsci.edu:8080/job/astropy-winxp32-staging-astrofrog/./COMPILER=msvc,NUMPY_VER=numpy-1.5.0,PYTHON_VER=python-2.6/ws/junit.xml>
======== 2 failed, 3117 passed, 580 skipped, 1 xfailed in 66.33 seconds ========

If this becomes too hard to debug this way, I can start up the Windows VM I set up on AWS.

mdboom · 2012-12-06T15:20:04Z

The test_find_by_hash test should be passing encoding='binary' if it wants to get bytes back. I've fixed this here. As for the others, it's most likely related to the change in precision handling in votable (PR #527). Either the formatters are behaving differently on Windows than on Unix, or we're displaying precision beyond what we were before and now seeing that there is some truncation going on on Windows.

@iguananaut: I haven't used it in a while, and sorry if you explained this already and it slipped by me: do I need to do anything other than pushing to my staging branch to get the Windows Jenkins test to fire?

embray · 2012-12-06T21:13:13Z

Right now I don't even have a staging branch build for you on Windows. I had to delete them all at one point because the configs got messed up. I'll recreate it though. If you've already pushed to your staging branch I can then just manually kick off a build.

embray · 2012-12-06T21:15:36Z

Okay, I have it building ebd2778 right now.

mdboom · 2012-12-07T14:19:53Z

It looks like on Python 2.6 float formatting was done using system libraries, and these had different behavior on Linux vs. Windows. In Python 2.7/3.1 and later, Python includes its own float formatting code so that it's consistent everywhere. We're just seeing this failure now because the "precision" fields are being handled correctly for the first time in VOTable (the precision not being used properly before meant that this difference was hidden).

We could include our own copy of the "dtoa" included with Python 2.7 for use on Python 2.6. Whether we can integrate that into the built-in string % operator and format isn't clear, though, so we might need to call our own function when we care about precision.

There might be a way to leverage Numpy (which is some cases does its own string formatting of numbers) to do this, but I haven't found a way to do it -- is there a Numpy function to format a number with precision that doesn't fallback to exponential notation? (array_repr comes close, but it's not quite the right thing).

Lastly, we can just mark this test as known fail when the legacy float formatting is in use -- Python even provides sys.float_repr_style to check for this.

This is from "What's new in Python 2.7":

Conversions between floating-point numbers and strings are now correctly rounded on most platforms. These conversions occur in many different places: str() on floats and complex numbers; the float and complex constructors; numeric formatting; serializing and deserializing floats and complex numbers using the marshal, pickle and json modules; parsing of float and imaginary literals in Python code; and Decimal-to-float conversion.

Related to this, the repr() of a floating-point number x now returns a result based on the shortest decimal string that’s guaranteed to round back to x under correct rounding (with round-half-to-even rounding mode). Previously it gave a string based on rounding x to 17 decimal digits.

The rounding library responsible for this improvement works on Windows and on Unix platforms using the gcc, icc, or suncc compilers. There may be a small number of platforms where correct operation of this code cannot be guaranteed, so the code is not used on such systems. You can find out which code is being used by checking sys.float_repr_style, which will be short if the new code is in use and legacy if it isn’t.

embray · 2012-12-07T19:22:13Z

I seem to recall encountering this float formatting issue somewhere myself. IIRC my solution at the time was to just re-reinvent the wheel and write my own float-formatting--just enough to deal with the parts that were inconsistent (in this case I believe it had to do with the number of digits after the exponent symbol.

mdboom · 2012-12-07T20:14:55Z

Yeah -- I've seen that "digits after the exponent" thing before, too... I think adding our own function seems like the thing to do -- but it's rather complex to get right. This is the code that Python 2.7/3.1 based their implementation on:

http://www.netlib.org/fp/dtoa.c

Unlike glibc's implementation, this one has a BSD-like license.

Here's CPython's modification of it:

http://hg.python.org/cpython/file/026b9f5dd97f/Python/dtoa.c

And we would need to add a very simple Python wrapper around that, of course.

This shouldn't be horrible, but it does seem like a lot of work for a version of Python that is already seeing pretty light usage percentage-wise.

astrofrog · 2012-12-07T21:42:21Z

I would just xfail this test in cases where it's expected to fail on Python 2.6. It's not like the results are actually that wrong that it would matter to the user anyway, right?

mdboom · 2012-12-10T17:08:29Z

Agreed that it probably wouldn't matter to the user. However, if the user requested a particular precision in VOTable, I think they are probably doing so for a reason and would care about getting all of the precision they asked for.

I think the thing to do now is probably to xfail (so we can at least have tests passing) and open a new issue for the precision problem.

…binary mode on Python 3

mdboom · 2012-12-10T17:21:19Z

I've added the xfails and pushed to my staging branch to make sure everything passes on Windows. Once that's true, I think this is good to merge.

mdboom · 2012-12-10T20:10:55Z

The Jenkins failure seems to be a network issue on just one of the configurations, so I'm going to go ahead and merge this.

astrofrog · 2012-12-10T20:20:55Z

Sounds good!

Failures on Windows with Python 3 (possible fix)

…-reading Failures on Windows with Python 3 (possible fix)

mdboom added 3 commits December 10, 2012 12:08

Shot in the dark to fix astropy#538 -- files are not being opened in …

6b97928

…binary mode on Python 3

Fix test_find_by_hash on Python 3.x/Windows

ecb8a6e

xfail some tests if legacy_float_repr is in effect

3fa4964

mdboom mentioned this pull request Dec 10, 2012

precision in votable incorrect when writing on Python 2.6 / MS-Windows #550

Closed

astrofrog mentioned this pull request Dec 11, 2012

travis fails when remote data option is on in tests #491

Closed

mdboom added a commit that referenced this pull request Dec 11, 2012

Merge pull request #539 from mdboom/utils/windows-python3-file-reading

42147a3

Failures on Windows with Python 3 (possible fix)

mdboom merged commit 42147a3 into astropy:master Dec 11, 2012

keflavich pushed a commit to keflavich/astropy that referenced this pull request Oct 9, 2013

Merge pull request astropy#539 from mdboom/utils/windows-python3-file…

ec4fd17

…-reading Failures on Windows with Python 3 (possible fix)

mdboom deleted the utils/windows-python3-file-reading branch May 21, 2014 23:58

Uh oh!

Failures on Windows with Python 3 (possible fix) #539

Failures on Windows with Python 3 (possible fix) #539

Uh oh!

Conversation

mdboom commented Dec 6, 2012

Uh oh!

astrofrog commented Dec 6, 2012

Uh oh!

astrofrog commented Dec 6, 2012

Uh oh!

mdboom commented Dec 6, 2012

Uh oh!

embray commented Dec 6, 2012

Uh oh!

embray commented Dec 6, 2012

Uh oh!

mdboom commented Dec 7, 2012

Uh oh!

embray commented Dec 7, 2012

Uh oh!

mdboom commented Dec 7, 2012

Uh oh!

astrofrog commented Dec 7, 2012

Uh oh!

mdboom commented Dec 10, 2012

Uh oh!

mdboom commented Dec 10, 2012

Uh oh!

mdboom commented Dec 10, 2012

Uh oh!

astrofrog commented Dec 10, 2012

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants