Implements group_by, projection and offset on datastore Query. #282

dhermes · 2014-10-22T07:39:15Z

In addition, __eq__ was implemented on datastore.key.Key to allow for easy comparison.

See #281 for some context. As mentioned there, I looked and didn't see any equivalent bugs for these features.

tseaver · 2014-10-22T12:13:35Z

How is this PR different from #281?

dhermes · 2014-10-22T15:32:38Z

From #281:

As new Query features are outside the scope of functional / regression testing, I am going
to peel them off into a separate PR and then rebase this one onto it after that is reviewed.
However, please feel free to dig into this review.

This PR has only the new features and none of the regression tests.

dhermes · 2014-10-22T21:34:14Z

Rebased after the Travis fix.

@tseaver PTAL

coveralls · 2014-10-22T22:03:21Z

Coverage remained the same when pulling 8df6f41 on dhermes:implement-groupby-projection-offset into 9884f77 on GoogleCloudPlatform:master.

tseaver · 2014-10-22T22:44:27Z

Implementation LGTM, although I don't know these bits of the API at all. Does 'offset()' conflict with the cursor stuff?

dhermes · 2014-10-23T00:45:57Z

No, they are complimentary. We can get final sign-off from one of the datastore folks if you like?

coveralls · 2014-10-23T00:58:05Z

Coverage remained the same when pulling 3d4ce5d on dhermes:implement-groupby-projection-offset into 2da8654 on GoogleCloudPlatform:master.

dhermes · 2014-10-23T01:14:06Z

@tseaver while fixing some failures in #281 I realized #274 made key equality a bit of a different animal.

I added b61ba2e to address some corner cases.

You'll notice that this is a bit unpleasant. ISTM we should work harder to make the underlying representation of a Key simpler. /cc @pcostell

coveralls · 2014-10-23T01:20:29Z

Coverage remained the same when pulling b61ba2e on dhermes:implement-groupby-projection-offset into 2da8654 on GoogleCloudPlatform:master.

tseaver · 2014-10-23T17:42:47Z

I still don't see why two keys, each with 'dataset_id == None' but having the same path, should be unequal (they are both in the "default" dataset for the application's connection).

Likewise for when both keys have 'namespace == None' but the same path.

dhermes · 2014-10-23T17:55:19Z

They would both be equal. See test_key___eq__.

The failures happen if (dataset as an example)

        if not (self._dataset_id == other._dataset_id or
                self._dataset_id is None or other._dataset_id is None):

which is equivalent to

        if (self._dataset_id != other._dataset_id and
            self._dataset_id is not None and other._dataset_id is not None):

In other words, failure only occurs if the values don't match AND both values are non-null.

tseaver · 2014-10-23T18:02:20Z

I don't understand why it isn't just:

if self._dataset_id != other._dataset_id:

If they are both None, they will be equal; if either is not None but the other is, they won't be equal. None compares for equality fine with arbitrary objects.

dhermes · 2014-10-23T18:34:27Z

Dataset: Because of Fix #121: don't copy project ID to the Key protobuf's dataset_id. #276 (sorry was referring to Revert "Fix #121: Drop munging of 's~' prefix onto keys." #274 before) the values may be unset.
Namespace: The same happens with values of None (local) vs u'' (from retrieved protobuf).

To give a simplified idea of what is happening in test_post_with_generated_id, test_post_with_id and test_post_with_name:

>>> import regression.regression_utils
>>> dataset = regression.regression_utils.get_dataset()
>>> entity = dataset.entity(kind='SomeKind')
>>> entity['key'] = 'value'
>>> key = entity.key().name('some_name')  # Same for partial key or key id.
>>> entity = entity.key(key)
>>> entity.save()
<Entity[{'kind': 'SomeKind', 'name': 'some_name'}] {'key': 'value'}>
>>> retrieved_entity = dataset.get_entity(entity.key())

then we'll have

>>> entity.key().path()
[{'kind': 'SomeKind', 'name': 'some_name'}]
>>> retrieved_entity.key().path()
[{'kind': u'SomeKind', 'name': u'some_name'}]
>>> 
>>> print entity.key()._dataset_id
None
>>> retrieved_entity.key()._dataset_id
u's~redacted-dataset-id'
>>>
>>> print entity.key()._namespace
None
>>> retrieved_entity.key()._namespace
u''

coveralls · 2014-10-23T18:52:42Z

Coverage remained the same when pulling c27a8d5 on dhermes:implement-groupby-projection-offset into 09859db on GoogleCloudPlatform:master.

dhermes · 2014-10-23T20:45:15Z

Rebased after #289.

@tseaver does the explanation above make sense for allowing null values in __eq__? Do you think we should do some more work to make these values actually agree?

For Datasets: I suppose much of this is what was removed in Fix #121: don't copy project ID to the Key protobuf's dataset_id. #276
For namespaces we can either switch our default from None to '' or we can set _namespace to None when it is '' on the protobuf

coveralls · 2014-10-23T20:50:48Z

Coverage remained the same when pulling 330ffc5 on dhermes:implement-groupby-projection-offset into 8a21978 on GoogleCloudPlatform:master.

tseaver · 2014-10-23T21:03:13Z

Yes, I see what you are trying to accomplish now. WRT the key's _datset_id, I'm not positive we want None to compare equal to one with an ID set by the back-end: it is sort of like comparing a naive datetime to a zoned one: they will be the same now, but later (if multi-dataset access comes into play) they might not be.

WRT namespace: we could switch the default value from None to '', and then just compare them naturally.

coveralls · 2014-10-23T23:10:42Z

Coverage remained the same when pulling 95528d2 on dhermes:implement-groupby-projection-offset into b6d3e74 on GoogleCloudPlatform:master.

This issue surfaced during review for googleapis#282.

dhermes · 2014-10-24T00:25:40Z

@tseaver it turns out we weren't handling namespaces quite perfectly. See #292.

As for dataset ID, the comment I added in Entity.save() to #292 may be the best strategy (i.e. if the key starts implicit, it should stay implicit).

To fully support this, we'd (at least) need to change Dataset.get_entities since it just creates each entity via helpers.entity_from_protobuf without any regard to the original values in keys().

A simple solution to this could be changing

        for entity_pb in entity_pbs:
            entities.append(helpers.entity_from_protobuf(
                entity_pb, dataset=self))

to

        for entity_pb, curr_key in zip(entity_pbs, keys):
            curr_entity = helpers.entity_from_protobuf(
                entity_pb, dataset=self)
            # If original key was implicit, maintain this on the
            # returned entity.
            if curr_key._dataset_id is None:
                curr_entity.key()._dataset_id = None
            else:
                if curr_entity.key() != curr_key:
                    raise ValueError('Key returned disagrees with key used '
                                     'in query.')
            entities.append(curr_entity)

🤖 I have created a release \*beep\* \*boop\* --- ### [3.3.1](https://www.github.com/googleapis/python-dlp/compare/v3.3.0...v3.3.1) (2021-11-05) ### Bug Fixes * **deps:** drop packaging dependency ([84181e9](https://www.github.com/googleapis/python-dlp/commit/84181e971ee04b46a603119d44410816fd7f04be)) * **deps:** require google-api-core >= 1.28.0 ([84181e9](https://www.github.com/googleapis/python-dlp/commit/84181e971ee04b46a603119d44410816fd7f04be)) * fix extras_require typo in setup.py ([84181e9](https://www.github.com/googleapis/python-dlp/commit/84181e971ee04b46a603119d44410816fd7f04be)) ### Documentation * list oneofs in docstring ([84181e9](https://www.github.com/googleapis/python-dlp/commit/84181e971ee04b46a603119d44410816fd7f04be)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

* chore: Update gapic-generator-python to v1.11.2 PiperOrigin-RevId: 546510849 Source-Link: googleapis/googleapis@736073a Source-Link: googleapis/googleapis-gen@deb64e8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZGViNjRlOGVjMTlkMTQxZTMxMDg5ZmU5MzJiM2E5OTdhZDU0MWM0ZCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Source-Link: googleapis/synthtool@56da63e Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:993a058718e84a82fda04c3177e58f0a43281a996c7c395e0a56ccc4d6d210d7

fix(deps): require proto-plus>=1.15.0

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

This adds region tags in order to support the Cloud Code API Explorer pilot

Source-Link: https://togithub.com/googleapis/synthtool/commit/352b9d4c068ce7c05908172af128b294073bf53c Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:3e3800bb100af5d7f9e810d48212b37812c1856d20ffeafb99ebe66461b61fc7

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

#282) Tests were failing because the `str` method for the context returned by `pytest.raises` no longer prints the contained exception. Instead, use `match=regex_value` to check for the desired error message.

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

Co-authored-by: Daniel Sanche <[email protected]>

add test cases for interceptor

…p/templates/python_library/.kokoro (#282) Source-Link: googleapis/synthtool@d0f51a0 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:240b5bcc2bafd450912d2da2be15e62bc6de2cf839823ae4bf94d4f392b451dc Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

…282) * chore: add default_version and codeowner_team to .repo-metadata.json * Assign @googleapis/actools-python as codeowner

* feat: add support for IN/NOT_IN/!= operator * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * chore: fix tests * chore: fix tests Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>

Source-Link: googleapis/synthtool@9ae0785 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:52210e0e0559f5ea8c52be148b33504022e1faef4e95fbe4b32d68022af2fa7e Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Lingqing Gan <[email protected]>

Source-Link: googleapis/synthtool@9ae0785 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:52210e0e0559f5ea8c52be148b33504022e1faef4e95fbe4b32d68022af2fa7e Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

dhermes mentioned this pull request Oct 22, 2014

Functional test - Add datastore query tests #281

Merged

tseaver added the api: datastore Issues related to the Datastore API. label Oct 22, 2014

dhermes force-pushed the implement-groupby-projection-offset branch 2 times, most recently from 8dcf9f4 to 104345a Compare October 22, 2014 21:33

dhermes force-pushed the implement-groupby-projection-offset branch from 104345a to 8df6f41 Compare October 22, 2014 21:53

dhermes force-pushed the implement-groupby-projection-offset branch from 8df6f41 to 3d4ce5d Compare October 23, 2014 00:50

dhermes force-pushed the implement-groupby-projection-offset branch from b61ba2e to c27a8d5 Compare October 23, 2014 18:38

dhermes force-pushed the implement-groupby-projection-offset branch from c27a8d5 to 330ffc5 Compare October 23, 2014 20:40

dhermes force-pushed the implement-groupby-projection-offset branch from 330ffc5 to 95528d2 Compare October 23, 2014 23:01

dhermes added a commit to dhermes/google-cloud-python that referenced this pull request Oct 23, 2014

Only using explicit namespace when deserializing key pb.

12695c2

This issue surfaced during review for googleapis#282.

dhermes mentioned this pull request Oct 23, 2014

Only using explicit namespace when deserializing key pb. #292

Merged

parthea pushed a commit that referenced this pull request Sep 22, 2023

chore(deps): update dependency google-cloud-dataproc to v3 (#282)

b2a1251

parthea added a commit that referenced this pull request Sep 22, 2023

fix(deps): require google-api-core>=1.31.5, >=2.3.2 (#282)

a0a36b4

fix(deps): require proto-plus>=1.15.0

parthea pushed a commit that referenced this pull request Oct 21, 2023

chore: release 3.5.0 (#282)

355d30a

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

parthea pushed a commit that referenced this pull request Oct 21, 2023

chore(deps): update dependency pytest to v7.1.2 (#282)

0514105

parthea pushed a commit that referenced this pull request Oct 21, 2023

docs(samples): Add Cloud Code tags for API Explorer pilot (#282)

56098f4

This adds region tags in order to support the Cloud Code API Explorer pilot

parthea added a commit that referenced this pull request Oct 21, 2023

chore: update .repo-metadata.json (#282)

4b2aa9e

parthea pushed a commit that referenced this pull request Oct 21, 2023

chore(deps): update dependency google-analytics-admin to v0.11.1 (#282)

091b895

parthea pushed a commit that referenced this pull request Oct 21, 2023

chore(deps): update dependency pytest to v7.1.3 (#282)

b17bb6f

parthea pushed a commit that referenced this pull request Oct 22, 2023

chore(main): release 1.8.0 (#282)

9ae5ec5

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

parthea pushed a commit that referenced this pull request Nov 22, 2025

chore: release 2.0.0 (#282)

0b83d6d

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

parthea pushed a commit that referenced this pull request Nov 24, 2025

chore(deps): update actions/checkout action to v5 (#282)

3eeab7f

Co-authored-by: Daniel Sanche <[email protected]>

parthea pushed a commit that referenced this pull request Nov 24, 2025

add test cases for interceptor (#282)

517c45b

add test cases for interceptor

parthea added a commit that referenced this pull request Nov 24, 2025

chore: add default_version and codeowner_team to .repo-metadata.json (#…

28adde4

…282) * chore: add default_version and codeowner_team to .repo-metadata.json * Assign @googleapis/actools-python as codeowner

parthea pushed a commit that referenced this pull request Nov 24, 2025

chore(deps): update dependency google-cloud-bigquery to v3.19.0 (#282)

2106eba

parthea pushed a commit that referenced this pull request Nov 24, 2025

chore(deps): update dependency google-cloud-bigquery to v2.16.0 (#282)

ddafd59

parthea pushed a commit that referenced this pull request Nov 24, 2025

chore(deps): update dependency google-cloud-bigquery to v2.16.0 (#282)

40bc589

parthea pushed a commit that referenced this pull request Nov 25, 2025

chore(deps): update dependency google-cloud-bigquery to v2.16.0 (#282)

4544e8d

parthea pushed a commit that referenced this pull request Nov 25, 2025

chore(deps): update dependency google-cloud-bigquery to v2.16.0 (#282)

7ac6623

parthea pushed a commit that referenced this pull request Nov 26, 2025

Release v1.5.1 (#282)

96c0803

Implements group_by, projection and offset on datastore Query. #282

Implements group_by, projection and offset on datastore Query. #282

Uh oh!

Conversation

dhermes commented Oct 22, 2014

Uh oh!

tseaver commented Oct 22, 2014

Uh oh!

dhermes commented Oct 22, 2014

Uh oh!

dhermes commented Oct 22, 2014

Uh oh!

coveralls commented Oct 22, 2014

Uh oh!

tseaver commented Oct 22, 2014

Uh oh!

dhermes commented Oct 23, 2014

Uh oh!

coveralls commented Oct 23, 2014

Uh oh!

dhermes commented Oct 23, 2014

Uh oh!

coveralls commented Oct 23, 2014

Uh oh!

tseaver commented Oct 23, 2014

Uh oh!

dhermes commented Oct 23, 2014

Uh oh!

tseaver commented Oct 23, 2014

Uh oh!

dhermes commented Oct 23, 2014

Uh oh!

coveralls commented Oct 23, 2014

Uh oh!

dhermes commented Oct 23, 2014

Uh oh!

coveralls commented Oct 23, 2014

Uh oh!

tseaver commented Oct 23, 2014

Uh oh!

coveralls commented Oct 23, 2014

Uh oh!

dhermes commented Oct 24, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants