Implicit dataset from environ #444

dhermes · 2014-12-19T01:06:15Z

~~This has #436 as a diff-base, so feel free to hold off review until it is merged.~~
This adds helper methods for implicit datastore environ.

By setting two environment variables

GCLOUD_DATASET_ID
GOOGLE_APPLICATION_CREDENTIALS

the user can call convenience methods in gcloud.datastore directly
without worrying about auth or the name of the dataset. The
goal is that in places like App Engine and Compute Engine, the
dataset ID can be implied without any user intervention.

Partially addresses #337.

NOTE: This still needs to be documented, but it's unclear where is appropriate. We also need to have documentation for auth (outside of CONTRIBUTING.md).

coveralls · 2014-12-19T01:09:15Z

Coverage remained the same when pulling 2f227ac on dhermes:implicit-dataset-from-environ into 89ded75 on GoogleCloudPlatform:master.

dhermes · 2014-12-19T01:11:13Z

Also note this can't be merged until the comment in Entity.__init__ is resolved:

    def __init__(self, dataset=None, kind=None, exclude_from_indexes=()):
        super(Entity, self).__init__()
        # DJH: Need to decide if this inherits from object/dict. Notice that
        #      `Entity` objects are False-y even if they have protected
        #      variables set.

This is the question put forth in #396. The short term solution is to inherit from both dict and the helper that Query and Transaction inherit from:

class Entity(dict, _implicit_environ._DatastoreBase):
    ...

Though branching MRO is something to avoid if we can.

coveralls · 2014-12-19T01:13:09Z

Coverage remained the same when pulling 13c727b on dhermes:implicit-dataset-from-environ into 21e5f45 on GoogleCloudPlatform:master.

dhermes · 2014-12-19T01:17:21Z

@silvolu This was one of the "convenience" things I mentioned before.

coveralls · 2014-12-19T20:17:04Z

Changes Unknown when pulling f0a7825 on dhermes:implicit-dataset-from-environ into * on GoogleCloudPlatform:master*.

dhermes · 2014-12-19T21:32:00Z

@tseaver PTAL

coveralls · 2014-12-20T01:47:14Z

Coverage remained the same when pulling 153f306 on dhermes:implicit-dataset-from-environ into b343acf on GoogleCloudPlatform:master.

coveralls · 2014-12-22T20:36:46Z

Coverage remained the same when pulling c5457aa on dhermes:implicit-dataset-from-environ into b343acf on GoogleCloudPlatform:master.

By setting two environment variables - GCLOUD_DATASET_ID - GOOGLE_APPLICATION_CREDENTIALS the user can call convenience methods in `gcloud.datastore` directly without worrying about auth or the name of the dataset. The goal is that in places like App Engine and Compute Engine, the dataset ID can be implied without **any** user intervention. Partially addresses googleapis#337. NOTE: This still needs to be documented, but it's unclear where is appropriate. We also need to have documentation for auth (outside of CONTRIBUTING.md).

coveralls · 2014-12-22T21:08:16Z

Coverage remained the same when pulling d85d200 on dhermes:implicit-dataset-from-environ into b343acf on GoogleCloudPlatform:master.

gcloud/datastore/__init__.py

tseaver · 2014-12-23T20:15:29Z

I'm pretty opposed to "implicit majyk" that triggers at import time, with no way for the application to control or opt out of. I'd much rather have the application do an explicit configuration call, either:

from gcloud.datastore import guess_default_dataset
guess_default_dataset(os.environ)

or have a way to register the dataset ID explicity (e.g., from a value in a config file)::

from gcloud.datastore import set_default_dataset
set_default_dataset(config.settings['gcloud.dataset_id'])

Yes, it requires a tiny bit more typing on a host where the environment contains the "right stuff", but it makes the process explicit, and a lot easier to explain / understand.

@tseaver

Incorporates feedback from @tseaver. Also - Fixing bad tests that accidentally leave side-effects. - Making the method to set the default dataset public.

coveralls · 2014-12-23T21:07:37Z

Coverage remained the same when pulling 7cd77f2 on dhermes:implicit-dataset-from-environ into b343acf on GoogleCloudPlatform:master.

dhermes · 2014-12-23T21:12:08Z

@tseaver I agree explicit is better, PTAL at latest changes. Thanks for the insight, I could have put more effort into the design of the interface.

RE: The MRO comment in Entity.__init__ it's moot since we are dropping dataset from Entity long term.

Also note RE: defaults set by free functions in datastore.__init__: when we remove Dataset we will need to also provide a method which sets a default connection and then have things like

q = datastore.query('Kind')

which will use both the default dataset ID and the default connection (bound to the query).

It's more likely that it'll only be necessary to use the default Connection in

q.fetch()

and just provide a

q.fetch(connection=connection)

to override.

tseaver · 2014-12-30T14:26:51Z

LGTM, with the note on raising an exception in 'set_default_dataset' if no value is passed and we can't figure one out from the environment.

dhermes · 2014-12-30T16:45:53Z

Going to merge this, then rebase in #456 and merge that.

I'm not 100% sure about raising when set_default_dataset fails. You're probably right, but that's an easy change and GCE and GAE support still need to be added to the method so there is a lot to do.

Implicit dataset from environ

* feat: exposed GetProcessorType to v1beta3 PiperOrigin-RevId: 502616281 Source-Link: googleapis/googleapis@04e6db9 Source-Link: googleapis/googleapis-gen@f4bc77f Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjRiYzc3ZjkwNWU2ODA1N2RiMDliNzgzOTdhZmJkZGI5Yjg2YzgwZSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* chore: Update gapic-generator-python to v1.9.0 PiperOrigin-RevId: 517425588 Source-Link: googleapis/googleapis@33c93eb Source-Link: googleapis/googleapis-gen@d5f5978 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZDVmNTk3ODlkMTlmYzQzMjcwZmYyMTI0OTY3ZDRlYzg5OTJiOGU4ZiJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Source-Link: googleapis/synthtool@0941ef3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:2f90537dd7df70f6b663cd654b1fa5dee483cf6a4edcfd46072b2775be8a23ec Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Source-Link: https://togithub.com/googleapis/synthtool/commit/5f2a6089f73abf06238fe4310f6a14d6f6d1eed3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:8555f0e37e6261408f792bfd6635102d2da5ad73f8f09bcb24f25e6afb5fac97

Co-authored-by: Anthonios Partheniou <[email protected]>

Source-Link: https://togithub.com/googleapis/synthtool/commit/0ddbff8012e47cde4462fe3f9feab01fbc4cdfd6 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:bced5ca77c4dda0fd2f5d845d4035fc3c5d3d6b81f245246a36aee114970082b

* chore: Update gapic-generator-python to v1.11.7 PiperOrigin-RevId: 573230664 Source-Link: googleapis/googleapis@93beed3 Source-Link: googleapis/googleapis-gen@f4a4eda Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjRhNGVkYWE4MDU3NjM5ZmNmNmFkZjkxNzk4NzIyODBkMWE4ZjY1MSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * chore: Update gapic-generator-python to v1.11.8 PiperOrigin-RevId: 574178735 Source-Link: googleapis/googleapis@7307199 Source-Link: googleapis/googleapis-gen@ce3af21 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiY2UzYWYyMWI3YzU1OWE4N2MyYmVmYzA3NmJlMGUzYWVkYTNhMjZmMCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * chore: Update gapic-generator-python to v1.11.9 PiperOrigin-RevId: 574520922 Source-Link: googleapis/googleapis@5183984 Source-Link: googleapis/googleapis-gen@a59af19 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTU5YWYxOWQ0YWM2NTA5ZmFlZGYxY2MzOTAyOTE0MWI2YTViODk2OCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * update post processor image; remove unused files * update docs/index.rst * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>

* chore: test minimum dependencies in python 3.7 * update constraints

Co-authored-by: Anthonios Partheniou <[email protected]>

* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>

* fix(deps): allow protobuf 3.19.5 * explicitly exclude protobuf 4.21.0

…00:00:00` (#444) * fix: read out-of-bounds DATETIME values such as `0001-01-01 00:00:00` deps: require google-cloud-bigquery 1.26.1 or later * feat: accepts a table ID, which downloads the table without a query * revert tests for read_gbq fix which isn't yet resolved * Revert "revert tests for read_gbq fix which isn't yet resolved" This reverts commit 2a76982df7cff48e58d8b1ad7eae19477665cb76. * add todo for next steps * add unit test for table ID read_gbq * add helper for is_query * implement read_gbq with table id * fix remaining tests, don't localalize out-of-bounds timestamp columns * Update pandas_gbq/gbq.py * fix 3.7 unit tests * correct coverage * skip coverage for optional test skip * fix docs build * improve test coverage for error case * as of google-cloud-bigquery 1.11.0, get_table before list_rows is unnecessary * refactor tests * add more scalars * add more types * add failing time test * add test for bignumeric * add test for null values * add epoch timestamps to tests * add post-download dtype conversions * add failing test for desired fix * fix the issue with extreme datetimes * fix constraints * fix tests for empty dataframe * fix tests for older google-cloud-bigquery * ignore index on empty dataframe * add db-dtypes to runtime import checks * document dependencies * remove TODO, since done * remove unnecessary special case for empty dataframe Fixes prerelease test run * remove redundant 'deprecated' from comment

🤖 I have created a release *beep* *boop* --- ## [0.17.0](googleapis/python-bigquery-pandas@v0.16.0...v0.17.0) (2022-01-19) ### ⚠ BREAKING CHANGES * use nullable Int64 and boolean dtypes if available (#445) ### Features * accepts a table ID, which downloads the table without a query ([#443](googleapis/python-bigquery-pandas#443)) ([bf0e863](googleapis/python-bigquery-pandas@bf0e863)) * use nullable Int64 and boolean dtypes if available ([#445](googleapis/python-bigquery-pandas#445)) ([89078f8](googleapis/python-bigquery-pandas@89078f8)) ### Bug Fixes * `read_gbq` supports extreme DATETIME values such as `0001-01-01 00:00:00` ([#444](googleapis/python-bigquery-pandas#444)) ([d120f8f](googleapis/python-bigquery-pandas@d120f8f)) * `to_gbq` allows strings for DATE and floats for NUMERIC with `api_method="load_parquet"` ([#423](googleapis/python-bigquery-pandas#423)) ([2180836](googleapis/python-bigquery-pandas@2180836)) * allow extreme DATE values such as `datetime.date(1, 1, 1)` in `load_gbq` ([#442](googleapis/python-bigquery-pandas#442)) ([e13abaf](googleapis/python-bigquery-pandas@e13abaf)) * avoid iteritems deprecation in pandas prerelease ([#469](googleapis/python-bigquery-pandas#469)) ([7379cdc](googleapis/python-bigquery-pandas@7379cdc)) * use data project for destination in `to_gbq` ([#455](googleapis/python-bigquery-pandas#455)) ([891a00c](googleapis/python-bigquery-pandas@891a00c)) ### Miscellaneous Chores * release 0.17.0 ([#470](googleapis/python-bigquery-pandas#470)) ([29ac8c3](googleapis/python-bigquery-pandas@29ac8c3)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Source-Link: googleapis/synthtool@993985f Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:1894490910e891a385484514b22eb5133578897eb5b3c380e6d8ad475c6647cd

Source-Link: googleapis/synthtool@0ddbff8 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:bced5ca77c4dda0fd2f5d845d4035fc3c5d3d6b81f245246a36aee114970082b Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* chore: reference main branch of google-cloud-python * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update README.rst Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

autosynth cannot find the source of changes triggered by earlier changes in this repository, or by version upgrades to tools such as linters.

This reverts commit 054db75734756b0e82e7984ca07fa80025edc908.

dhermes force-pushed the implicit-dataset-from-environ branch from 2f227ac to 13c727b Compare December 19, 2014 01:09

dhermes mentioned this pull request Dec 19, 2014

DISCUSSION PR: Re-writing Entity to subclass object instead of dict. #445

Closed

dhermes force-pushed the implicit-dataset-from-environ branch from 13c727b to f0a7825 Compare December 19, 2014 20:14

dhermes force-pushed the implicit-dataset-from-environ branch from f0a7825 to 153f306 Compare December 20, 2014 01:44

dhermes mentioned this pull request Dec 21, 2014

Fix 3 and 451 #454

Closed

dhermes force-pushed the implicit-dataset-from-environ branch from 153f306 to c5457aa Compare December 22, 2014 20:34

dhermes force-pushed the implicit-dataset-from-environ branch from c5457aa to d85d200 Compare December 22, 2014 21:05

dhermes mentioned this pull request Dec 22, 2014

Address first part of 451: Remove Key.from_path #456

Merged

tseaver reviewed Dec 23, 2014
View reviewed changes

gcloud/datastore/__init__.py Outdated

This comment was marked as spam.

Sign in to view

Making cached dataset ID require explicit interaction.

7cd77f2

Incorporates feedback from @tseaver. Also - Fixing bad tests that accidentally leave side-effects. - Making the method to set the default dataset public.

dhermes added a commit that referenced this pull request Dec 30, 2014

Merge pull request #444 from dhermes/implicit-dataset-from-environ

563a19f

Implicit dataset from environ

dhermes merged commit 563a19f into googleapis:master Dec 30, 2014

dhermes deleted the implicit-dataset-from-environ branch December 30, 2014 16:53

dhermes mentioned this pull request Dec 30, 2014

Add GAE/GCE support for datastore.set_default_dataset and add exception on failure #475

Closed

dhermes mentioned this pull request Jan 14, 2015

Allow implicit dataset / bucket / other behavior from environment #337

Closed

dhermes added the api: datastore Issues related to the Datastore API. label Dec 31, 2015

parthea added a commit that referenced this pull request Sep 22, 2023

chore(deps): update all dependencies (#444)

51f2d7b

Co-authored-by: Anthonios Partheniou <[email protected]>

parthea added a commit that referenced this pull request Oct 21, 2023

chore: test minimum dependencies in python 3.7 (#444)

8db03d2

* chore: test minimum dependencies in python 3.7 * update constraints

parthea added a commit that referenced this pull request Oct 21, 2023

chore(deps): update dependency google-cloud-translate to v3.8.2 (#444)

381fd99

Co-authored-by: Anthonios Partheniou <[email protected]>

parthea added a commit that referenced this pull request Oct 21, 2023

fix(deps): allow protobuf 3.19.5 (#444)

10975b5

* fix(deps): allow protobuf 3.19.5 * explicitly exclude protobuf 4.21.0

parthea pushed a commit that referenced this pull request Aug 21, 2025

chore(deps): update all dependencies (#444)

6152e20

parthea pushed a commit that referenced this pull request Sep 16, 2025

chore(deps): update all dependencies (#444)

4a56a5d

parthea pushed a commit that referenced this pull request Nov 24, 2025

chore(deps): update dependency google-api-core to v1.19.0 (#444)

54f5fef

parthea pushed a commit that referenced this pull request Nov 24, 2025

fix: allow reading logs from non-project paths (#444)

85041ee

parthea pushed a commit that referenced this pull request Nov 24, 2025

changes without context (#444)

f9d6390

autosynth cannot find the source of changes triggered by earlier changes in this repository, or by version upgrades to tools such as linters.

parthea pushed a commit that referenced this pull request Nov 25, 2025

fix: allow reading logs from non-project paths (#444)

9e42ec6

parthea pushed a commit that referenced this pull request Nov 26, 2025

Revert "fix: update _GOOGLE_OAUTH2_CERTS_URL (#365)" (#444)

9996ae3

This reverts commit 054db75734756b0e82e7984ca07fa80025edc908.

Implicit dataset from environ #444

Implicit dataset from environ #444

Uh oh!

Conversation

dhermes commented Dec 19, 2014

Uh oh!

coveralls commented Dec 19, 2014

Uh oh!

dhermes commented Dec 19, 2014

Uh oh!

coveralls commented Dec 19, 2014

Uh oh!

dhermes commented Dec 19, 2014

Uh oh!

coveralls commented Dec 19, 2014

Uh oh!

dhermes commented Dec 19, 2014

Uh oh!

coveralls commented Dec 20, 2014

Uh oh!

coveralls commented Dec 22, 2014

Uh oh!

coveralls commented Dec 22, 2014

Uh oh!

This comment was marked as spam.

Uh oh!

tseaver commented Dec 23, 2014

Uh oh!

coveralls commented Dec 23, 2014

Uh oh!

dhermes commented Dec 23, 2014

Uh oh!

tseaver commented Dec 30, 2014

Uh oh!

dhermes commented Dec 30, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants