Skip to content

Update generated lextab/parsetab files - #7174

Merged
eteq merged 8 commits into
astropy:masterfrom
astrofrog:update-ply
Feb 9, 2018
Merged

eteq merged 8 commits into
astropy:masterfrom
astrofrog:update-ply

Conversation

@astrofrog

Copy link
Copy Markdown
Member

This does the following:

  • Updates the bundled version of PLY to 3.10
  • Updates the code that auto-generates the lexers/parsers to add a header line with the license (which used to be handled by clean_parse_tables.py
  • Updates the comments to explain how to regenerate the file, and include that in the header of the lexers/parsers
  • Re-generates the files

Just to be clear, this does not use clean_parse_tables.py anymore.

Now one thing that isn't ideal is that in Python 3, it's more obvious that e.g. sets are returned in a non-deterministic order, so these files will change every time they are re-generated, which is not ideal. Some of it is under our control (see the addition of sorted for one of the sets), but I'm not sure how to fix them all. This may be something that has to be fixed in ply itself.

I think this should fix #7068

@astrofrog astrofrog added this to the v3.0.0 milestone Feb 9, 2018
@astrofrog
astrofrog requested review from bsipocz and eteq February 9, 2018 10:30
@astropy-bot

astropy-bot Bot commented Feb 9, 2018 •

Copy link
Copy Markdown

Hi there @astrofrog 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

lexer = lex.lex(optimize=True, lextab='cds_lextab',
outputdir=os.path.dirname(__file__),
reflags=re.UNICODE)
reflags=int(re.UNICODE))

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eteq @bsipocz - technically speaking this is the actual thing that was causing the bug in #7068 - should I split this out into a separate PR to backport to v2.0.x? Or should I do a separate PR to v2.0.x with this minimal fix?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohh, good catch.

I would say both approach is fine with me, but with the aim to get 3.0 out of the door, maybe the latter is better.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree a separate PR with just the minimal fix.

@astrofrog

Copy link
Copy Markdown
Member Author

Just for info, I opened a PR to ply to make the generated files deterministic (i.e. they will be the same every time they are re-generated): dabeaz/ply#151

@bsipocz bsipocz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bsipocz

bsipocz commented Feb 9, 2018

Copy link
Copy Markdown
Member

Updating ply to 3.10 actually preproduced the same error from #7068, so since this one passes I'm certain that that issue is solved by this PR. Thanks @astrofrog!

@astrofrog

Copy link
Copy Markdown
Member Author

By the way the re flag issue was actually due to a bug in PLY in Python 3.6, which I've opened a PR for here: dabeaz/ply#153

@astrofrog

Copy link
Copy Markdown
Member Author

@eteq - does this satisfy #4318 or do you also want a section in the dev docs themselves? I feel like having the instructions inside the files themselves and at the top of the Python files might be enough because stuff just gets lost in the vast dev docs...

@bsipocz

bsipocz commented Feb 9, 2018

Copy link
Copy Markdown
Member

@astrofrog - I think, for this case, having the instructions in the files themselves are more useful than having it somewhere in the docs.

@eteq eteq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM aside from one minor change. For #4318 I think we should just have a brief mention in the dev docs that amounts to "look at the header files". I.e., just a sentence or two probably in development/building - if you agree I can just put a commit straight up here adding that.

# remove it from this folder, then build astropy and run the tests in-place:
#
# python setup.py build_ext --inplace
# pytest astropy/coordinates

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about replacing this test part with:

python -c "from astropy.coordinates import angle_utilities;angle_utilities._AngleParser()"

? That seems a bit more fine-grained

@astrofrog astrofrog Feb 9, 2018 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mine is more general in case we add multiple parsers in future :) And more consistent with what's done for units.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think it's very dangerous to use the tests to do this... but I see your point that something more general is needed. So I guess this is fine for consistency, but I'm going to make an issue to try to address this better anyway.

# remove it from this folder, then build astropy and run the tests in-place:
#
# python setup.py build_ext --inplace
# pytest astropy/units

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one I don't see an obvious way to replace it with a more targeted call...

@eteq
eteq merged commit a92043d into astropy:master Feb 9, 2018
eteq added a commit that referenced this pull request Feb 9, 2018
Update generated lextab/parsetab files
@astrofrog
astrofrog deleted the update-ply branch November 14, 2018 15:30
@saimn saimn mentioned this pull request May 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can't build with Python 3.6: KeyError: '_unit'

3 participants