bpo-32861: urllib.robotparser fix incomplete __str__ methods. by michael-lazar · Pull Request #5711 · python/cpython

michael-lazar · 2018-02-17T01:56:14Z

The RobotFileParser's string representation was incomplete and missing some valid rule lines.

https://bugs.python.org/issue32861

The RobotFileParser's string representation was incomplete and missing some valid rule lines.

serhiy-storchaka

LGTM. I have added just few suggestions. And the two unnecessary trailing newlines should be kept in maintained releases.

Please add a news entry.

serhiy-storchaka · 2018-03-27T13:59:52Z


    def __str__(self):
-        return ''.join([str(entry) + "\n" for entry in self.entries])
+        ret = [str(entry) for entry in self.entries]


This may be faster:

entries = self.entries if self.default_entry is not None: entries = entries + [self.default_entry] return '\n\n'.join(map(str, entries))

serhiy-storchaka · 2018-03-27T14:00:36Z

        ret = []
        for agent in self.useragents:
-            ret.extend(["User-agent: ", agent, "\n"])
+            ret.append("User-agent: {0}".format(agent))


f-strings can be used in 3.6+.

serhiy-storchaka · 2018-03-27T14:03:57Z

        for line in self.rulelines:
-            ret.extend([str(line), "\n"])
-        return ''.join(ret)
+            ret.append(str(line))


Or just

ret.extend(map(str, self.rulelines))

michael-lazar · 2018-04-03T00:51:02Z

All suggestions have been implemented, I have no strong opinions on any of them. I also added a news entry.

Is backporting part of this PR, or do you merge this into 3.8 and then create separate issues for the other python branches? Is that something that I can help with?

serhiy-storchaka

LGTM.

Just add your credits.

serhiy-storchaka · 2018-05-13T10:37:11Z

@@ -0,0 +1,3 @@
+The urllib.robotparser's ``__str__`` representation now includes wildcard
+entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra
+newlines that were being appended to the end of the string.


Please add "Patch by yourname." and add your name into Misc/ACKS.

michael-lazar · 2018-05-14T01:12:56Z

Cool, I added my name to the news entry. I'm already in the ACKS file so all good there.

miss-islington · 2018-05-14T14:10:44Z

Thanks @michael-lazar for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 2.7, 3.6, 3.7.
🐍🍒⛏🤖

…GH-5711) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra newlines that were being appended to the end of the string. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]>

bedevere-bot · 2018-05-14T14:11:52Z

GH-6795 is a backport of this pull request to the 3.7 branch.

miss-islington · 2018-05-14T14:12:48Z

Sorry, @michael-lazar and @serhiy-storchaka, I could not cleanly backport this to 2.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker bd08a0af2d88c590ede762102bd42da3437e9980 2.7

bedevere-bot · 2018-05-14T14:13:53Z

GH-6796 is a backport of this pull request to the 3.6 branch.

…GH-5711) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra newlines that were being appended to the end of the string. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]>

…H-5711) (GH-6795) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]>

…ythonGH-5711) (pythonGH-6795) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]> (cherry picked from commit c3fa1f2) Co-authored-by: Miss Islington (bot) <[email protected]>

…ythonGH-5711) (pythonGH-6795) The robotparser's __str__ representation now includes wildcard entries. (cherry picked from commit c3fa1f2) Co-authored-by: Michael Lazar <[email protected]>.

…ythonGH-5711) (pythonGH-6795) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]> (cherry picked from commit c3fa1f2) Co-authored-by: Miss Islington (bot) <[email protected]>

…H-5711) (GH-6795) (GH-6818) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit c3fa1f2) Co-authored-by: Michael Lazar <[email protected]>

GH-6795) (GH-6817) The robotparser's __str__ representation now includes wildcard entries. (cherry picked from commit c3fa1f2) Co-authored-by: Michael Lazar <[email protected]>.

bpo-32861: urllib.robotparser fixed incomplete __str__ methods.

6bf354c

The RobotFileParser's string representation was incomplete and missing some valid rule lines.

the-knights-who-say-ni added the CLA signed label Feb 17, 2018

bedevere-bot added the awaiting review label Feb 17, 2018

serhiy-storchaka reviewed Mar 27, 2018

View reviewed changes

serhiy-storchaka added type-bug An unexpected behavior, bug, or error needs backport to 3.6 awaiting changes and removed awaiting changes labels Mar 27, 2018

bpo-32861: Code review cleanup and adding news entry (#5711)

301451c

serhiy-storchaka approved these changes May 13, 2018

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels May 13, 2018

serhiy-storchaka self-assigned this May 13, 2018

Update 2018-04-02-20-44-54.bpo-32861.HeBjzN.rst

e9e8259

serhiy-storchaka merged commit bd08a0a into python:master May 14, 2018

bedevere-bot removed the awaiting merge label May 14, 2018

bedevere-bot removed the needs backport to 3.7 label May 14, 2018

bedevere-bot removed the needs backport to 3.6 label May 14, 2018

serhiy-storchaka removed the needs backport to 2.7 label May 14, 2018

serhiy-storchaka removed their assignment Dec 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bpo-32861: urllib.robotparser fix incomplete str methods.#5711

bpo-32861: urllib.robotparser fix incomplete str methods.#5711
serhiy-storchaka merged 3 commits intopython:masterfrom
michael-lazar:fix-issue-32861

michael-lazar commented Feb 17, 2018 •

edited by bedevere-bot

Loading

Uh oh!

serhiy-storchaka left a comment

Uh oh!

serhiy-storchaka Mar 27, 2018

Uh oh!

serhiy-storchaka Mar 27, 2018

Uh oh!

serhiy-storchaka Mar 27, 2018

Uh oh!

michael-lazar commented Apr 3, 2018 •

edited

Loading

Uh oh!

serhiy-storchaka left a comment

Uh oh!

serhiy-storchaka May 13, 2018

Uh oh!

michael-lazar commented May 14, 2018

Uh oh!

miss-islington commented May 14, 2018

Uh oh!

bedevere-bot commented May 14, 2018

Uh oh!

miss-islington commented May 14, 2018

Uh oh!

bedevere-bot commented May 14, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

michael-lazar commented Feb 17, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Mar 27, 2018

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Mar 27, 2018

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Mar 27, 2018

Choose a reason for hiding this comment

Uh oh!

michael-lazar commented Apr 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka May 13, 2018

Choose a reason for hiding this comment

Uh oh!

michael-lazar commented May 14, 2018

Uh oh!

miss-islington commented May 14, 2018

Uh oh!

bedevere-bot commented May 14, 2018

Uh oh!

miss-islington commented May 14, 2018

Uh oh!

bedevere-bot commented May 14, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

michael-lazar commented Feb 17, 2018 •

edited by bedevere-bot

Loading

michael-lazar commented Apr 3, 2018 •

edited

Loading