Skip to content

Upgrade PLY to 3.6 - #4003

Merged
embray merged 3 commits into
astropy:masterfrom
mdboom:extern/upgrade-ply-36
Jul 28, 2015
Merged

embray merged 3 commits into
astropy:masterfrom
mdboom:extern/upgrade-ply-36

Conversation

@mdboom

@mdboom mdboom commented Jul 23, 2015

Copy link
Copy Markdown
Contributor

Reminded by #4001, we haven't upgraded PLY in a while. Notably 3.6 includes a Unicode-related patch to PLY that we've been keeping locally all this time. Moving to 3.6, though requiring one minor backward-incompatible change related to file generation, allows us to use a "stock" PLY where desired.

Note that PLY no longer officially supports Python 2.6. It works on our unit tests, so I think we're fine for now, but let that be another nail in the coffin of Python 2.6.

Related to this, PLY still generates code that isn't Python 2 and 3 cross-compatible, so it requires some modifications. I should submit a PR against PLY for that.

@mdboom mdboom assigned ebressert and embray and unassigned ebressert Jul 23, 2015
@mdboom mdboom added this to the v1.1.0 milestone Jul 23, 2015
@embray

embray commented Jul 24, 2015

Copy link
Copy Markdown
Member

Since we keep the PLY-generated files in the repo it's fine if it doesn't work on Python 2.6 I guess. 👍

@embray

embray commented Jul 24, 2015

Copy link
Copy Markdown
Member

Is this branched from #4001? Or if I merge #4001 first do you think there will be a conflict?

@mdboom

mdboom commented Jul 24, 2015

Copy link
Copy Markdown
Contributor Author

It might conflict, but I can always fix later...

Comment thread astropy/extern/ply/lex.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh. Turns out this function was the cause of the reference cycle I've been trying to track down in #4002 . I need to double check but I'm almost certain this should fix it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, confirmed. This is all it was.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow. Wow. And wow.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, there was also #4012, so there were actually 2 reference cycles involved in this. But I think this is the one that I was actually being affected by.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old version of this code is PLY is some serious voodoo magic.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a fairly nice trick to avoid having to use sys._getframe. I think they've given up on making sys._getframe a strictly non-public function though. Way too many people have discovered and made use of it in production code by this point :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs basically say "don't expect this to exist on anything but CPython". I shudder to think of all the other problems astropy would have running on anything but CPython, so I consider that a non-problem for us...

@embray

embray commented Jul 28, 2015

Copy link
Copy Markdown
Member

@mdboom Would you mind rebasing this so I can merge it? Thanks.

@mdboom
mdboom force-pushed the extern/upgrade-ply-36 branch from 9904502 to ef901f3 Compare July 28, 2015 15:37
@mdboom

mdboom commented Jul 28, 2015

Copy link
Copy Markdown
Contributor Author

Rebased.

Comment thread dev/clean_parse_tables.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, that's a pain. IIRC I did this by hand last time these were updated.

Maybe (if you haven't already) I'll submit a patch to PLY to do this? (Certainly worth keeping this script around in the meantime though.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah -- a patch to PLY would be helpful. I haven't got around to that yet.

embray added a commit that referenced this pull request Jul 28, 2015
@embray
embray merged commit d29573b into astropy:master Jul 28, 2015
@embray

embray commented Jul 28, 2015

Copy link
Copy Markdown
Member

I'll do the patch. I already sent one to objgraph the other day so I'm on a roll :p

@embray

embray commented Jul 28, 2015

Copy link
Copy Markdown
Member

Granted, a patch to PLY won't involve the "license" line, but maybe we could morph that to a general tool that checks source files for that and adds it where it's missing.

@embray

embray commented Jul 28, 2015

Copy link
Copy Markdown
Member

And on second thought, is this patch (or these touch-ups) really necessary in general?
The format and contents of these tab files is not likely to change much in the future. They don't contain any print statements so print_function is irrelevant. Similar for division and absolute_import. The only one that's relevant is unicode_literals, but as it is if it outputs unicode strings in the u'' format, that will still work on Python 3.3 and up, which I believe is all we support anyways (and for Python 2 it's a non-issue).

@mdboom

mdboom commented Jul 28, 2015

Copy link
Copy Markdown
Contributor Author

That's true. I guess I thought we were still supporting Python 3.2. If that's not the case then this doesn't matter. Why don't we keep the tables as-is for now, and I'll file a quick PR to remove the little dev tool.

@embray

embray commented Jul 28, 2015

Copy link
Copy Markdown
Member

Hmm, I think we're only supporting Python 3.2 for 1.0.x (and poorly, at that, since travis-ci doesn't run 3.2 anymore, though I still test locally....sometimes...).

I'm fine either way.

@mdboom

mdboom commented Jul 28, 2015

Copy link
Copy Markdown
Contributor Author

Ok -- so we'll just keep as-is for now until Python 3.2 support is dropped, I guess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants