Skip to content

Swapping out google.streaming for google-resumable-media in GCS downloads. #3323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dhermes merged 3 commits into from
Apr 24, 2017
Merged

Swapping out google.streaming for google-resumable-media in GCS downloads. #3323

dhermes merged 3 commits into from
Apr 24, 2017

Conversation

@dhermes
Copy link
Contributor

@dhermes dhermes commented Apr 24, 2017

@dhermes dhermes added the api: storage Issues related to the Cloud Storage API. label Apr 24, 2017
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Apr 24, 2017
theacodes
theacodes reviewed Apr 24, 2017
View reviewed changes
storage/setup.py Outdated

REQUIREMENTS = [
'google-cloud-core >= 0.24.0, < 0.25dev',
'google-auth',

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

storage/setup.py Outdated
'google-cloud-core >= 0.24.0, < 0.25dev',
'google-auth',
'google-resumable-media == 0.0.2',
'requests',

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

storage/google/cloud/storage/blob.py Outdated
import six
from six.moves.urllib.parse import quote

import google.auth.transport.requests as tr_requests

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

tseaver
tseaver reviewed Apr 24, 2017
View reviewed changes
storage/google/cloud/storage/blob.py Outdated
import six
from six.moves.urllib.parse import quote

import google.auth.transport.requests as tr_requests

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

storage/google/cloud/storage/blob.py
if self.chunk_size is None:
download = resumable_media.Download(download_url, headers=headers)
response = download.consume(transport)
file_obj.write(response.content)

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

tseaver
tseaver reviewed Apr 24, 2017
View reviewed changes
storage/google/cloud/storage/blob.py
download.chunksize = self.chunk_size

# Get the download URL.
download_url = self._get_download_url()

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

@dhermes dhermes mentioned this pull request Apr 24, 2017
Closed
@dhermes
Copy link
Contributor Author

dhermes commented Apr 24, 2017

I filed googleapis/google-resumable-media-python#5 to carry on the discussion about streaming the contents in a more memory-efficient manner (when possible).

For now, I am going to merge this based on an LGTM from @jonparrott (given in chat)

@dhermes dhermes merged commit d0ef6a9 into googleapis:master Apr 24, 2017
@dhermes dhermes deleted the resumable-media-for-GCS-downloads branch April 24, 2017 22:36
This was referenced Apr 27, 2017
Merged
Closed
richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017
@dhermes
Swapping out google.streaming for google-resumable-media in GCS downl…
153c4b9 
…oads. (googleapis#3323)

* Swapping out google.streaming for google-resumable-media in GCS downloads.

* Dropping tr_requests import alias in storage blob module.

* Putting lower bounds on new deps in storage package.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@tseaver tseaver tseaver left review comments

@theacodes theacodes theacodes left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@lukesneeringer lukesneeringer

@tseaver tseaver

Labels

api: storage Issues related to the Cloud Storage API. cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dhermes @tseaver @theacodes @lukesneeringer @googlebot