Skip to content

Use BufferedReader instead of BytesIO in utils subpackage - #7981

Closed
jairideout wants to merge 1 commit into
astropy:masterfrom
jairideout:issue-6652-utils
Closed

jairideout wants to merge 1 commit into
astropy:masterfrom
jairideout:issue-6652-utils

Conversation

@jairideout

Copy link
Copy Markdown

Addresses part of #6652 by using io.BufferedReader instead of io.BytesIO in the utils subpackage. This change was only necessary in the astropy.utils.data.get_readable_fileobj function.

Addresses part of astropy#6652 by using `io.BufferedReader` instead of `io.BytesIO` in the `utils` subpackage. This change was only necessary in the `astropy.utils.data.get_readable_fileobj` function.
@astropy-bot

astropy-bot Bot commented Oct 26, 2018 •

Copy link
Copy Markdown

Hi there @jairideout 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

Comment thread astropy/utils/data.py
return nbytes

stream = RawStream(fileobj)
fileobj = io.BufferedReader(stream)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why the additional RawStream layer is needed, instead of using directly BufferedReader ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constructor for io.BufferedReader requires an object that's a subclass (or at least a duck type) of io.RawIOBase.

In this part of the codepath for get_readable_fileobj(), fileobj is only guaranteed to have a .read() method, so an io.RawIOBase subclass is required to wrap fileobj in an appropriate interface for io.BufferedReader.

@pllim pllim modified the milestones: v3.2, v4.0 Apr 19, 2019
@codecov

codecov Bot commented Apr 19, 2019 •

Copy link
Copy Markdown

Codecov Report

Merging #7981 into master will increase coverage by <.01%.
The diff coverage is 93.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7981      +/-   ##
==========================================
+ Coverage   86.86%   86.86%   +<.01%     
==========================================
  Files         377      377              
  Lines       57174    57189      +15     
  Branches     1016     1016              
==========================================
+ Hits        49663    49678      +15     
  Misses       6927     6927              
  Partials      584      584
Impacted Files Coverage Δ
astropy/utils/data.py 80.99% <93.75%> (+0.41%) ⬆️
astropy/samp/hub.py 77.52% <0%> (+0.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6890f3...dd1fe5f. Read the comment docs.

3 similar comments
@codecov

codecov Bot commented Apr 19, 2019 •

Copy link
Copy Markdown

Codecov Report

Merging #7981 into master will increase coverage by <.01%.
The diff coverage is 93.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7981      +/-   ##
==========================================
+ Coverage   86.86%   86.86%   +<.01%     
==========================================
  Files         377      377              
  Lines       57174    57189      +15     
  Branches     1016     1016              
==========================================
+ Hits        49663    49678      +15     
  Misses       6927     6927              
  Partials      584      584
Impacted Files Coverage Δ
astropy/utils/data.py 80.99% <93.75%> (+0.41%) ⬆️
astropy/samp/hub.py 77.52% <0%> (+0.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6890f3...dd1fe5f. Read the comment docs.

@codecov

codecov Bot commented Apr 19, 2019

Copy link
Copy Markdown

Codecov Report

Merging #7981 into master will increase coverage by <.01%.
The diff coverage is 93.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7981      +/-   ##
==========================================
+ Coverage   86.86%   86.86%   +<.01%     
==========================================
  Files         377      377              
  Lines       57174    57189      +15     
  Branches     1016     1016              
==========================================
+ Hits        49663    49678      +15     
  Misses       6927     6927              
  Partials      584      584
Impacted Files Coverage Δ
astropy/utils/data.py 80.99% <93.75%> (+0.41%) ⬆️
astropy/samp/hub.py 77.52% <0%> (+0.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6890f3...dd1fe5f. Read the comment docs.

@codecov

codecov Bot commented Apr 19, 2019 •

Copy link
Copy Markdown

Codecov Report

Merging #7981 into master will increase coverage by <.01%.
The diff coverage is 93.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7981      +/-   ##
==========================================
+ Coverage   86.86%   86.86%   +<.01%     
==========================================
  Files         377      377              
  Lines       57174    57189      +15     
  Branches     1016     1016              
==========================================
+ Hits        49663    49678      +15     
  Misses       6927     6927              
  Partials      584      584
Impacted Files Coverage Δ
astropy/utils/data.py 80.99% <93.75%> (+0.41%) ⬆️
astropy/samp/hub.py 77.52% <0%> (+0.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6890f3...dd1fe5f. Read the comment docs.

@pllim

pllim commented Apr 24, 2019

Copy link
Copy Markdown
Member

I am proposing a simpler approach at #8634.

@jairideout

Copy link
Copy Markdown
Author

@pllim #8634 is indeed simpler than this PR (just wrap BytesIO in BufferedReader). I chose the implementation here to avoid reading the entire file into a BytesIO object, specifically to address the code comments on lines 200-204:

# Check if the file object supports random access, and if not,
# then wrap it in a BytesIO buffer.  It would be nicer to use a
# BufferedReader to avoid reading loading the whole file first,
# but that is not compatible with streams or urllib2.urlopen
# objects on Python 2.x.

Just wanted to note the differences between our solutions -- thanks!

@pllim

pllim commented Apr 24, 2019

Copy link
Copy Markdown
Member

@jairideout , gotcha. However, I am not sure anymore that we even need either PR, as the current code is not causing any problem AFAIK and the intent of the original comment was lost with time. 🤷‍♀️

@jairideout

Copy link
Copy Markdown
Author

Sounds good! Feel free to close this PR if you decide it's not worth including due to the additional complexity 👍

@saimn

saimn commented May 24, 2019

Copy link
Copy Markdown
Contributor

I had a look but could not find a real use case for this while running the tests. Sorry for the useless work, but I think that we should close, and wait for a real need to consider using this, given the additional complexity.

@saimn saimn added the Close? Tell stale bot that this issue/PR is stale label May 24, 2019
@pllim

pllim commented May 25, 2019

Copy link
Copy Markdown
Member

I concur with @saimn . Thanks for the work anyway, @jairideout . It is much appreciated and we hope you continue to contribute!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Close? Tell stale bot that this issue/PR is stale no-changelog-entry-needed Refactoring utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants