Skip to content

use six.moves versions for map/range/zip (when used) - #5259

Merged
mhvk merged 1 commit into
astropy:masterfrom
MSeifert04:replace_xrange
Aug 26, 2016
Merged

mhvk merged 1 commit into
astropy:masterfrom
MSeifert04:replace_xrange

Conversation

@MSeifert04

Copy link
Copy Markdown
Contributor

This PR adds from extern.six.moves at all places where map, zip or range are used.

Originally I only wanted to replace them where python2 names or the try ... except ... was still used, i.e. from six.moves import xrange or izip. But then I thought it probably wouldn't hurt to use the generator versions.

This PR also removes some trailing whitespaces or rows containing only whitespaces. (that was accidental because my editor is setup to remove them "on saving", but that doesn't hurt either, right?)

@mhvk

mhvk commented Aug 24, 2016

Copy link
Copy Markdown
Contributor

Are we sure we want to do all? I think it does slow down things slightly and to me the benefits are not obvious. I do think it is good to get rid of xrange or other things that are not in python3.

... ... ... well, actually, I should check before I write: on python3

from astropy.extern import six
six.moves.range is range, six.moves.zip is zip, six.moves.map is map
# (True, True, True)

while it is (False, False, False) on python2. So, clearly it does not slow down anything on python3 (except importing perhaps).

Still, I think I slightly prefer to keep the boilerplate to the from __future__ import... and only do this for clear python2-isms.

@MSeifert04

MSeifert04 commented Aug 24, 2016 •

Copy link
Copy Markdown
Contributor Author

@mhvk Thank you for the fast answer.

Regarding the rationale: The six.moves versions of map and zip simply import from the itertools module (see https://pythonhosted.org/six/#module-six.moves) and just represent the generator (python3)-equivalent of map, zip and range:

>>> import six
>>> import itertools
>>> six.moves.map is itertools.imap, six.moves.zip is itertools.izip, six.moves.range is xrange
(True, True, True)

They are all imported from the standard library so the import overhead should be negligable but I only added them where they are actually used.
Also the generator versions are generally slightly faster and require much less memory than the python2 builtins (there are few exceptions) because they don't create an intermediate list.

However I don't feel strong about it so if you think that's not really necessary I can alter/close the PR again.

@MSeifert04 MSeifert04 changed the title use six.moves versions for map/range/zip "everywhere" use six.moves versions for map/range/zip (when used) Aug 24, 2016
from .common import setup_function, teardown_function
from ....tests.helper import pytest
from ....extern.six.moves import cStringIO
from ....extern.six.moves import range, cStringIO as StringIO

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is there a need to rename cStringIO? Won't that just be adding to the confusion?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it's now called StringIO. Like xrange, izip and imap it's a name that isn't used anymore in python3.

@pllim

pllim commented Aug 24, 2016 •

Copy link
Copy Markdown
Member

Given the plan to completely drop Python 2 support by v3 (see APE10), does this just add more stuff to undo in the near future?

EDIT: Corrected link.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

@pllim I think you must have chosen the wrong link: APE10 But you are right that it adds a lot of stuff that has to be undone. OTOH it's still more than one year and one LTS until then.

@pllim

pllim commented Aug 24, 2016

Copy link
Copy Markdown
Member

@MSeifert04 , haha, yes... On Linux, there are two different clipboards and my click pulled the wrong one. It is corrected. Good catch!

@pllim

pllim commented Aug 24, 2016

Copy link
Copy Markdown
Member

We just have to think about if this is worth all the undoing a year from now. This is not really fixing any bug, right? So, really, no gain, but a lot of extra work later. Please correct me if I am wrong.

@MSeifert04

MSeifert04 commented Aug 24, 2016 •

Copy link
Copy Markdown
Contributor Author

This is not really fixing any bug, right?

Right

So, really, no gain, but a lot of extra work later.

I think it probably won't have noticible gain for python2 and for python3 there is no gain at all.

However for several modules there is already a from six.moves import ... present so there is no additional work. Also in most files there already is a from extern import six present, so someone actually has to open that file to edit it nevertheless. And in some places names like xrange and izip or cStringIO (or even try/excepts) were used which are removed in python3 so in those places it actually reduces the amount of work later.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

I totally forgot: I removes a lot of unnecessary whitespaces! That shaves some bytes off the package 😅

@pllim

pllim commented Aug 24, 2016

Copy link
Copy Markdown
Member

Okay, all good points. Since the work is already done... why not? 👍

@mhvk

mhvk commented Aug 24, 2016

Copy link
Copy Markdown
Contributor

I guess overall the work is minimized any time python2-only names are no longer used, and if we go to a state where all we need to do to make it python3-only is to remove lines importing from __future__ and from six, then we're definitely ahead, since that can be done quite automatically. (Indeed, as a follow-up it may make sense to ensure that any python2-ism like definining __nonzero__ is properly guarded by a if six.PY2 statement....).

I'm still not completely sold on just adding imports where things worked fine with python2 and python3 using exactly the same code while effectively doing something slightly different, but on the other hand like @pllim I can see the advantage of minimizing those cases.

So, overall, I'm happy to merge this, but just in case let me ping @astrofrog as the one who has thought the move to python3 through most.

@mhvk mhvk added this to the v1.3.0 milestone Aug 24, 2016
@mhvk mhvk self-assigned this Aug 24, 2016
import numpy as np

from astropy.table import Table, Column
from astropy.extern.six.moves import range

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure: Should these be relative imports?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I don't see why they can't be relative.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed they should be relative.

@mhvk

mhvk commented Aug 26, 2016

Copy link
Copy Markdown
Contributor

OK, I think enough time for second thought has passed, so I'll merge! Thanks, @MSeifert04!

@mhvk
mhvk merged commit 5d6c475 into astropy:master Aug 26, 2016
@MSeifert04
MSeifert04 deleted the replace_xrange branch August 26, 2016 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants