Had a few discussions with @GoogleCloudPlatform/cloud-datastore (particularly @pcostell) so I wanted to summarize the things we covered. This issue is to clarify the goals for Datastore support in gcloud-python, and discuss and decide on a course of action that everyone likes for how to reconcile what we have today with the other libraries out there.

Current state of the world

Currently, I see two styles that people want to use when interacting with Datastore:

1. a "simple" more key-value-based style (similar to Amazon's Simple DB) where you're CRUDding dictionaries and then adding some querying
2. a more advanced ORM style where you're create models, and have some sort of schema defined in a Python file.

For the former, gcloud.datastore has had the goal of covering this use case. For the latter, ndb is the latest way (supported) way of doing this -- with others potentially existing, but ndb seems to be the clear leader.

We also have a unique situation where our code currently might have trouble running in App Engine -- whereas ndb can't run outside of App Engine. The layout sort of looks like this:

Looking forward

If our goals are ....

1. We agree that both styles of interacting with Datastore matter (and both should exist)
2. gcloud.datastore and ndb are our choices for each style respectively
3. gcloud.datastore and ndb should both run in App Engine and non-App Engine runtimes
4. gcloud.datastore is where all the recommended Python stuff to talk to Datastore should live (it is the "official source of truth")
5. People who want to write new Python libraries for Datastore can rely on code that exists in gcloud.datastore (and set gcloud as a Python dependency)

... then I'd like to suggest that we....

1. Port ndb over as gcloud.datastore.ndb (bringing with it datastore_rpc and datastore_query)
2. Rewrite gcloud.datastore to run on top of datastore_query
3. Rename gcloud.datastore to be a peer with ndb (using "simple" in this diagram, not set on that at all though).

which makes things look like this:

What do you guys think?

/cc @GoogleCloudPlatform/cloud-datastore @dhermes @tseaver @silvolu @proppy