Coverage remained the same when pulling 78bd1a2 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

Coverage remained the same when pulling e50b270 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

Pretty soon the datastore will normalize all types of keys, so you should never need to provide anything except the project id. Importantly, _find_true_dataset_id will only work with the current dataset which means it will break for users that store foreign keys.

In the near-future we don't want anyone to have to deal with the "s~", it should be purely an implementation detail of the Datastore. I think the client libraries should probably be written pretending the "s~" doesn't exist since it is cleaner that way (with the caveat that it is slightly broken atm).

Thanks for the quick followup. That is great news! It will certainly be much cleaner if we can ignore s~. All the deleted code in this PR will still be able to be deleted even without _find_true_dataset_id.

How do you mean normalize? Currently it seems some normalization is happening, but that means if I send no dataset ID in the request and foo as the dataset ID in the URI, then I get back s~foo on the keys in the response. This still causes some issues / confusion / surprise.

I believe the goal is that everywhere you will specify foo or an empty dataset and when you get the key back you will also only see foo in the dataset.

@eddavisson can you confirm?

That's correct. Note that if you want to do a one-time RPC in the mean time, you can do a Lookup request and check both the found and missing fields in the response for the resolved dataset id (basically what you're doing in this pull request but it's a read-only RPC).

I didn't think to use missing! Good call. With the __key__ projection that'd be cheap enough, even if somehow the user had stored Key('__Foo', 1).

UPDATE: DERP, no projection on lookup.

I think we should go forward with @eddavisson suggestion while waiting for the API updates to land.

Coverage remained the same when pulling 3184184 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

@pcostell RE: foreign keys. I'm not sure why that's an issue if the application is managing those keys.

I double checked and the backend does not change the foreign dataset ID after storing (seems it shouldn't, for privacy reasons). From 0525530:

>>> from gcloud import datastore
>>> from gcloud.datastore import _implicit_environ
>>> from gcloud.datastore.entity import Entity
>>> from gcloud.datastore.key import Key
>>>
>>> datastore.set_default_connection()
>>> dataset_id = 'foo'
>>> datastore.set_default_dataset_id(dataset_id)
>>>
>>> e = Entity(key=Key('Foo', 1))
>>> e['bar'] = Key('Baz', 1, dataset_id='foreignappid')
>>> e.save()
>>>
>>> e_retrieved = datastore.get(e.key)
>>> e_retrieved['bar'].dataset_id
u'foreignappid'
>>>
>>> entity_pb, = _implicit_environ.CONNECTION.lookup(
...     dataset_id=dataset_id,
...     key_pbs=[e.key.to_protobuf()],
... )
>>> entity_pb.property
[<gcloud.datastore.datastore_v1_pb2.Property object at 0x7f4a5006d398>]
>>> print entity_pb.property[0].value.key_value
partition_id {
  dataset_id: "foreignappid"
}
path_element {
  kind: "Baz"
  id: 1
}

>>> e.key.delete()

FWIW I tried this both with a foreign dataset that my (user, not service account) OAuth 2.0 token had access to and one that the token did not.

I believe we want to remove the notion of "s~" completely. So anywhere the user interacts with a dataset it should never have this extra prefix, including foreign keys.

Note that this is not implemented, so it isn't finalized and definitely won't work yet.

Coverage remained the same when pulling 0cf4333 on dhermes:non-fuzzy-app-id into dcb73d3 on GoogleCloudPlatform:master.

Had to rebase after #548 and #550, so did the renames in each commit through the rebase.

Changes Unknown when pulling 31fd6fd on dhermes:non-fuzzy-app-id into * on GoogleCloudPlatform:master*.

LGTM.

@tseaver We can't / shouldn't merge this until the discussion above is resolved.

Coverage remained the same when pulling 5d631a6 on dhermes:non-fuzzy-app-id into 55285f9 on GoogleCloudPlatform:master.

Rebased and squashed into single commit (had to hack around after rebasing).

Changes Unknown when pulling 0c4f4c6 on dhermes:non-fuzzy-app-id into * on GoogleCloudPlatform:master*.

Coverage remained the same at 100.0% when pulling 9b0f87a on dhermes:non-fuzzy-app-id into 9518db7 on GoogleCloudPlatform:master.

I'm closing this (since it is so out of date) and moving to discussion / tracking in #821.

Part of it was merged in #792 as well.