Skip to content

Replace hand-rolled upload/download impl. w/ apitools utilities. #407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tseaver merged 5 commits into from
Dec 10, 2014
Merged

Replace hand-rolled upload/download impl. w/ apitools utilities. #407

tseaver merged 5 commits into from
Dec 10, 2014

Conversation

@tseaver
Copy link
Contributor

@tseaver tseaver commented Dec 4, 2014

Note that in order to use the 'Upload' / 'Download' utilities provided
by apitools without its 'base_api' module (not vendored), we have to unwind
some of the generated request setup code.

Fixes #334.

@tseaver
Replace hand-rolled upload/download impl. w/ apitools utilities.
08e0c9e 
Note that in order to use the 'Upload' / 'Download' utilities provided
by apitools without its 'base_api' module (not vendored), we have to unwind
some of the generated request setup code.

Fixes #334.
silvolu
silvolu reviewed Dec 4, 2014
View reviewed changes
gcloud/storage/key.py Outdated

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

@coveralls
Copy link

coveralls commented Dec 5, 2014

Coverage Status

Coverage remained the same when pulling 71780e3 on tseaver:334-use_apitools_transfer_for_upload_download into 03fc101 on GoogleCloudPlatform:master.

@tseaver
Garden in 'python-modernize' updates to apitools.
0416e93 
Catches up our vendored-in source to commit:
5a1bab1df7a474cff57f7f5cc066b19a1c286a21.
@tseaver
Copy link
Contributor Author

tseaver commented Dec 8, 2014

0416e93 pulls in @craigcitro's merge of my 'python-modernize' PR to the vendored-in sources.

@coveralls
Copy link

coveralls commented Dec 8, 2014

Coverage Status

Coverage remained the same when pulling 0416e93 on tseaver:334-use_apitools_transfer_for_upload_download into 03fc101 on GoogleCloudPlatform:master.

@tseaver
Copy link
Contributor Author

tseaver commented Dec 9, 2014

@silvolu, @craigcitro, @dhermes: any further review? Do we need to add regression tests which explicitly target either "simple" or "resumable" uploads?

@silvolu
Copy link
Contributor

silvolu commented Dec 9, 2014

LGTM

Do we need to add regression tests which explicitly target either "simple" or "resumable" uploads?

Please do.

@tseaver
Add a regression test for small-file writes.
9a68d90 
In conjunction with the existing large-file test, this one ensures
that we exercise both the 'simple'/'media' upload API, and the
'resumable' one.
@tseaver
Copy link
Contributor Author

tseaver commented Dec 9, 2014

@silvolu I added a "small file" regression test in 9a68d90, exercising the "simple" / "media" upload API. @dhermes already had a "large file" one, which now exercises the "resumable" upload API.

@coveralls
Copy link

coveralls commented Dec 9, 2014

Coverage Status

Coverage remained the same when pulling 9a68d90 on tseaver:334-use_apitools_transfer_for_upload_download into 03fc101 on GoogleCloudPlatform:master.

tseaver added a commit that referenced this pull request Dec 10, 2014
@tseaver
Merge pull request #407 from tseaver/334-use_apitools_transfer_for_up…
41286a8 
…load_download

Replace hand-rolled upload/download impl. w/ apitools utilities.
@tseaver tseaver merged commit 41286a8 into googleapis:master Dec 10, 2014
@tseaver tseaver deleted the 334-use_apitools_transfer_for_upload_download branch December 10, 2014 02:32
@tseaver
Copy link
Contributor Author

tseaver commented Dec 10, 2014

Ugh, another set of likely-transient regression test failures post-merge. I have restarted the build on Travis..

@silvolu
Copy link
Contributor

silvolu commented Dec 10, 2014

@tseaver I can reproduce the failures. Looking into fixing the tests.

@dhermes dhermes mentioned this pull request Dec 30, 2014
Closed
@dhermes dhermes added the api: storage Issues related to the Cloud Storage API. label Dec 31, 2015
parthea pushed a commit that referenced this pull request Jun 4, 2023
@renovate-bot
chore(deps): update dependency google-cloud-documentai to v2.1.0 (#407)
3fcd645
parthea pushed a commit that referenced this pull request Jun 4, 2023
@gcf-owl-bot
chore: resolve issue with prerelease presubmit [autoapprove] (#407)
930a54e 
Source-Link: googleapis/synthtool@1b9ad76
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:9db98b055a7f8bd82351238ccaacfd3cda58cdf73012ab58b8da146368330021
parthea added a commit that referenced this pull request Jun 4, 2023
@gcf-owl-bot @parthea
feat: Add files field to finding's list of attributes (#407)
b246140 
* chore: update to gapic-generator-python 1.5.0

feat: add support for `google.cloud.<api>.__version__`
PiperOrigin-RevId: 484665853

Source-Link: googleapis/googleapis@8eb249a

Source-Link: googleapis/googleapis-gen@c8aa327
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzhhYTMyN2I1ZjQ3ODg2NWZjM2ZkOTFlM2MyNzY4ZTU0ZTI2YWQ0NCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update version in gapic_version.py

* add .release-please-manifest.json with correct version

* add owlbot.py to exclude generated gapic_version.py

* set manifest to true in .github/release-please.yml

* add release-please-config.json

* chore: Update to gapic-generator-python 1.6.0

feat(python): Add typing to proto.Message based class attributes

feat(python): Snippetgen handling of repeated enum field

PiperOrigin-RevId: 487326846

Source-Link: googleapis/googleapis@da380c7

Source-Link: googleapis/googleapis-gen@61ef576
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNjFlZjU3NjJlZTY3MzFhMGNiYmZlYTIyZmQwZWVjZWU1MWFiMWM4ZSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* feat: new APIs added to reflect updates to the filestore service

- Add ENTERPRISE Tier
- Add snapshot APIs: RevertInstance, ListSnapshots, CreateSnapshot, DeleteSnapshot, UpdateSnapshot
- Add multi-share APIs: ListShares, GetShare, CreateShare, DeleteShare, UpdateShare
- Add ConnectMode to NetworkConfig (for Private Service Access support)
- New status codes (SUSPENDED/SUSPENDING, REVERTING/RESUMING)
- Add SuspensionReason (for KMS related suspension)
- Add new fields to Instance information: max_capacity_gb, capacity_step_size_gb, max_share_count, capacity_gb, multi_share_enabled

PiperOrigin-RevId: 487492758

Source-Link: googleapis/googleapis@5be5981

Source-Link: googleapis/googleapis-gen@ab0e217
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYWIwZTIxN2Y1NjBjYzJjMWFmYzExNDQxYzJlYWI2YjY5NTBlZmQyYiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update path to snippet metadata json

* chore: Update gapic-generator-python to v1.6.1

PiperOrigin-RevId: 488036204

Source-Link: googleapis/googleapis@08f275f

Source-Link: googleapis/googleapis-gen@555c094
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTU1YzA5NDVlNjA2NDllMzg3MzlhZTY0YmM0NTcxOWNkZjcyMTc4ZiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* feat: Add files field to finding's list of attributes

PiperOrigin-RevId: 488290111

Source-Link: googleapis/googleapis@7349e1c

Source-Link: googleapis/googleapis-gen@721049d
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNzIxMDQ5ZDUxMmRlZWJhYTk2ZDQzOGFhMjNlOTcwMjk5ZDk3NzEyYiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <[email protected]>
parthea pushed a commit that referenced this pull request Jun 4, 2023
@gcf-owl-bot
chore(deps): Update nox in .kokoro/requirements.in [autoapprove] (#407)
8964345 
Source-Link: https://togithub.com/googleapis/synthtool/commit/92006bb3cdc84677aa93c7f5235424ec2b157146
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:2e247c7bf5154df7f98cce087a20ca7605e236340c7d6d1a14447e5c06791bd6
parthea pushed a commit that referenced this pull request Sep 20, 2023
@renovate-bot
chore(deps): update all dependencies (#407)
8ab06fb
vchudnov-g pushed a commit that referenced this pull request Sep 20, 2023
@release-please
chore: release 2.9.0 (#407)
9d45771 
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Sep 22, 2023
@gcf-owl-bot
chore: add prerelease nox session (#407)
0494e8c 
Source-Link: googleapis/synthtool@050953d
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:65e656411895bff71cffcae97246966460160028f253c2e45b7a25d805a5b142

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea added a commit that referenced this pull request Sep 22, 2023
@gcf-owl-bot @parthea
chore(python): upgrade gcp-releasetool in .kokoro [autoapprove] (#407)
60b12d3 
* chore(python): upgrade gcp-releasetool in .kokoro [autoapprove]

Source-Link: googleapis/synthtool@5f2a608
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:8555f0e37e6261408f792bfd6635102d2da5ad73f8f09bcb24f25e6afb5fac97

* trigger ci

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <[email protected]>
parthea pushed a commit that referenced this pull request Oct 21, 2023
@gcf-owl-bot
build(deps): [autoapprove] bump certifi from 2022.12.7 to 2023.7.22 (#…
7b6663d 
…407)

Source-Link: https://togithub.com/googleapis/synthtool/commit/395d53adeeacfca00b73abf197f65f3c17c8f1e9
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:6c1cbc75c74b8bdd71dada2fa1677e9d6d78a889e9a70ee75b93d1d0543f96e1
parthea pushed a commit that referenced this pull request Oct 21, 2023
@gcf-owl-bot
chore: add prerelease nox session (#407)
640ad16 
Source-Link: googleapis/synthtool@050953d
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:65e656411895bff71cffcae97246966460160028f253c2e45b7a25d805a5b142

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Oct 21, 2023
@gcf-owl-bot
build(deps): bump cryptography to 41.0.0 [autoapprove] (#407)
ee0b865 
Source-Link: https://togithub.com/googleapis/synthtool/commit/d0f51a0c2a9a6bcca86911eabea9e484baadf64b
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:240b5bcc2bafd450912d2da2be15e62bc6de2cf839823ae4bf94d4f392b451dc
parthea pushed a commit that referenced this pull request Aug 21, 2025
@gcf-owl-bot
chore(deps): update actions/download-artifact action to v3 (#407)
f9f0753 
Source-Link: googleapis/synthtool@38e11ad
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:4e1991042fe54b991db9ca17c8fb386e61b22fe4d1472a568bf0fcac85dcf5d3
parthea pushed a commit that referenced this pull request Sep 16, 2025
@gcf-owl-bot @parthea
chore(deps): update actions/download-artifact action to v3 (#407)
0cecd8a 
Source-Link: googleapis/synthtool@38e11ad
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:4e1991042fe54b991db9ca17c8fb386e61b22fe4d1472a568bf0fcac85dcf5d3
parthea pushed a commit that referenced this pull request Nov 22, 2025
@gcf-owl-bot
feat: Publish new fields to support cluster group routing for Cloud B…
e27989c 
…igtable (#407)

* feat: Publish new fields to support cluster group routing for Cloud Bigtable

Committer: @garye
PiperOrigin-RevId: 391576441

Source-Link: googleapis/googleapis@5f76113

Source-Link: googleapis/googleapis-gen@e01b342

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Nov 22, 2025
@release-please
chore: release 2.4.0 (#408)
cc0feac 
🤖 I have created a release \*beep\* \*boop\*
---
## [2.4.0](https://www.github.com/googleapis/python-bigtable/compare/v2.3.3...v2.4.0) (2021-09-24)


### Features

* Publish new fields to support cluster group routing for Cloud Bigtable ([#407](https://www.github.com/googleapis/python-bigtable/issues/407)) ([66af554](https://www.github.com/googleapis/python-bigtable/commit/66af554a103eea0139cb313691d69f4c88a9e87f))


### Bug Fixes

* add 'dict' annotation type to 'request' ([160bfd3](https://www.github.com/googleapis/python-bigtable/commit/160bfd317a83561821acc0212d3514701a031ac6))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
parthea pushed a commit that referenced this pull request Nov 24, 2025
@renovate-bot @software-dov
chore(deps): update dependency dataclasses to v0.7 (#407)
61cddb4 
Co-authored-by: Dov Shlachter <[email protected]>
parthea pushed a commit that referenced this pull request Nov 24, 2025
@nolanleastin
fix: Let OS select an available port when running TestInstalledAppFlow (
5c1a0d2 
#407)

tests.unit.test_flow.TestInstalledAppFlow attempts to create a new
server at the specified host:port. New server creation occasionally
results in 'address already in use' because the socket will be
unavailable for a period of time after the socket is closed, resulting
in flaky test failures.

Work around this in the tests by letting the OS pick an available port each time.

Fixes #381
parthea pushed a commit that referenced this pull request Nov 24, 2025
@gcf-owl-bot
docs: Add documentation for enums (#407)
8aae4be 
* docs: Add documentation for enums

fix: Add context manager return types

chore: Update gapic-generator-python to v1.8.1
PiperOrigin-RevId: 503210727

Source-Link: googleapis/googleapis@a391fd1

Source-Link: googleapis/googleapis-gen@0080f83
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDA4MGY4MzBkZWMzN2MzMzg0MTU3MDgyYmNlMjc5ZTM3MDc5ZWE1OCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Nov 24, 2025
@gcf-owl-bot
docs: Add documentation for enums (#407)
0d9f624 
* docs: Add documentation for enums

fix: Add context manager return types

chore: Update gapic-generator-python to v1.8.1
PiperOrigin-RevId: 503210727

Source-Link: googleapis/googleapis@a391fd1

Source-Link: googleapis/googleapis-gen@0080f83
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDA4MGY4MzBkZWMzN2MzMzg0MTU3MDgyYmNlMjc5ZTM3MDc5ZWE1OCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Nov 24, 2025
@craiglabenz @gcf-owl-bot @crwilcox
feat: add support for recursive queries (#407)
783fb03 
* refactor: added BaseQuery.copy method

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* responded to code review

* feat: added recursive query

* tidied up

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* more tidying up

* fixed error with path compilation

* fixed async handling in system tests

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* Update google/cloud/firestore_v1/base_collection.py

Co-authored-by: Christopher Wilcox <[email protected]>

* reverted error message changes

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* comment updates

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Christopher Wilcox <[email protected]>
parthea pushed a commit that referenced this pull request Nov 24, 2025
@gcf-owl-bot
chore: use gapic-generator-python 0.52.0 (#407)
ede879e 
- [ ] Regenerate this pull request now.

fix: improper types in pagers generation
PiperOrigin-RevId: 399773015

Source-Link: googleapis/googleapis@410c184

Source-Link: googleapis/googleapis-gen@290e883
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMjkwZTg4MzU0NWUzYWM5ZmYyYmQwMGNkMGRhY2IyOGYxYjhjYTk0NSJ9
parthea pushed a commit that referenced this pull request Nov 24, 2025
@cguardia
fix: support same options in model.query as query (#407)
ce9945b 
For maintaining consistency.
parthea pushed a commit that referenced this pull request Nov 25, 2025
@gcf-owl-bot
chore: use gapic-generator-python 0.52.0 (#407)
992d5ef 
- [ ] Regenerate this pull request now.

fix: improper types in pagers generation
PiperOrigin-RevId: 399773015

Source-Link: googleapis/googleapis@410c184

Source-Link: googleapis/googleapis-gen@290e883
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMjkwZTg4MzU0NWUzYWM5ZmYyYmQwMGNkMGRhY2IyOGYxYjhjYTk0NSJ9
parthea pushed a commit that referenced this pull request Nov 26, 2025
@busunkim96
fix: revert "feat: send quota project id in x-goog-user-project heade…
e9d73ac 
…r for OAuth2 credentials (#400)" (#407)

This reverts commit ab3dc1e26f5240ea3456de364c7c5cb8f40f9583.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Use apitools for uploads/downloads

5 participants

@tseaver @coveralls @silvolu @craigcitro @dhermes