This repository was archived by the owner on Feb 20, 2026. It is now read-only.
docs(samples): Query profiling feature samples#1253
Merged
danieljbruce merged 95 commits intogoogleapis:mainfrom Jun 19, 2024
Merged
docs(samples): Query profiling feature samples#1253danieljbruce merged 95 commits intogoogleapis:mainfrom
danieljbruce merged 95 commits intogoogleapis:mainfrom
Conversation
Copy paste over the files from the preview branch
Thread the query mode through the client library. Allow calls to get made using the query mode.
This code allows basic query profiling to work and creates a basic code snippet that can be used to make it work.
This will be useful for debugging grpc issues.
This mock server will be super useful for ensuring that the right data gets transmitted in private preview so that we can confirm the client library is working correctly.
We use the ts file instead.
See if Explain gets passed along to the mock server.
Protos.json is needed to pass the plan along.
Make sure changes don’t break the use of this callback by adding this test.
Define it with a more general data structure.
…into query-profiling-feature # Conflicts: # protos/google/datastore/v1/query_profile.proto # protos/protos.d.ts # protos/protos.js # protos/protos.json # src/v1/datastore_client.ts
The merge caused generated files to have a diff. The diff should not be in the PR.
only should not be used here
Data structures need to be added for the return values that contain stats.
Add stats to info if it is available. If there are no results then end the stream and send the info back. This way, stats will always be sent in info if they are available and the program won ’t break if there are no results.
Explain what happens when the result set is empty. Just send the stats back.
The mock server code was used for debugging and now we don’t need it since the service is working.
Calls to nightly don’t have a second database to work with. Regular calls work now so nightly calls are not necessary.
Bring the query profiling tests back.
This reverts commit 040d0a5.
Stats do not necessarily come from the server.
Each query profiling mode needs to be explored.
Stats returned needs to be parsed by a special library that will remove the complexities of the Struct object.
A library is needed for parsing the Struct values. Add the libraries and use them.
Expected plan and expected stats should be used in the tests. This ensures the tests check for the proper stats values.
Add a specific type for the runAggregationQuery callback so that it can now support the info parameter. In order to allow runAggregationQuery to make use of creating info, we also refactor info construction into a separate function.
The parser for runAggregationQuery should have a deterministic process for computing results.
Make sure that the entities and the plan are correct.
kolea2
reviewed
May 28, 2024
| const [entities, info] = await datastore.runQuery(q, { | ||
| explainOptions: {analyze: true}, | ||
| }); | ||
| entities.sort((e1, e2) => { |
Contributor
Author
There was a problem hiding this comment.
Much like the or query test the entities are expected to be in a specific order for the test output. runQuery doesn't guarantee entities in a certain order so if we don't sort them then we end up with a flakey test.
Contributor
There was a problem hiding this comment.
if this is only meant to assist in tests, I would prefer we change the test itself to assert on the contents of the results, ignoring the order.
Co-authored-by: kolea2 <[email protected]>
Co-authored-by: kolea2 <[email protected]>
danieljbruce
added
the
kokoro:force-run
yoshi-kokoro
removed
the
kokoro:force-run
danieljbruce
added
the
kokoro:force-run
yoshi-kokoro
removed
the
kokoro:force-run
danieljbruce
added
the
owlbot:run
gcf-owl-bot
bot
removed
the
owlbot:run
kolea2
previously requested changes
May 29, 2024
Contributor
kolea2
left a comment
kolea2
left a comment
There was a problem hiding this comment.
looking good! A few small TODOs.
| const [entities, info] = await datastore.runQuery(q, { | ||
| explainOptions: {analyze: true}, | ||
| }); | ||
| entities.sort((e1, e2) => { |
Contributor
There was a problem hiding this comment.
if this is only meant to assist in tests, I would prefer we change the test itself to assert on the contents of the results, ignoring the order.
| 'use strict'; | ||
|
|
||
| // sample-metadata: | ||
| // title: Run query profiling (aggregation query) |
Contributor
There was a problem hiding this comment.
please fix to match other title (and in other places)
Make the tests indifferent about the order so that sorting isn’t required in the samples.
danieljbruce
added
the
owlbot:run
gcf-owl-bot
bot
removed
the
owlbot:run
cindy-peng
approved these changes
Jun 4, 2024
| }; | ||
|
|
||
| // Ensure the datastore database has no existing data. | ||
| const query = datastore.createQuery('Task'); |
Contributor
There was a problem hiding this comment.
Nit: does it make more sense to add these clean up to after() in L53?
Contributor
Author
There was a problem hiding this comment.
No because then the samples tests won't work. This code is responsible for clearing the database and setting it to a specific state so that when the samples run the output is a predictable value.
danieljbruce
dismissed
kolea2’s stale review
Not reviewing PR anymore as discussed in chat.
bhshkh
reviewed
Jun 6, 2024
| const [, info] = await datastore.runQuery(q, { | ||
| explainOptions: {analyze: false}, | ||
| }); | ||
| console.log(`info: ${Object.keys(info.explainMetrics)}`); |
Contributor
There was a problem hiding this comment.
When I run this code, I'm seeing error:
client.ts:32
console.log(`info: ${Object.keys(info.explainMetrics)}`);
^
TypeError: Cannot convert undefined or null to object
at Function.keys (<anonymous>)
at quickstart (/usr/local/google/home/bahaaiman/Documents/cfdb-workspace-01/clients/node/nodejs-datastore-client/client.ts:32:35)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Node.js v20.14.0
What am I doing wrong in the code?
// Imports the Google Cloud client library
const { Datastore } = require('@google-cloud/datastore');
// Creates a client
const datastore = new Datastore({projectId: "my-project"});
async function quickstart() {
// The Cloud Datastore key for the new entity
const taskKey = datastore.key(['Task', 'task1']);
// Prepares the new entity
const task = {
key: taskKey,
data: {
Description: 'Buy milk2',
Done: false,
Slice: [100, 200, 300]
},
};
// Saves the entity
await datastore.save(task);
console.log(`Saved ${task.key.name}: ${task.data.Description}`);
const query = datastore.createQuery('Task');
const [entities, info] = await datastore.runQuery(query, {
explainOptions: {analyze: true},
});
for (const entity of entities) {
console.log(`Entity found: ${entity['description']}`);
}
console.log(`info: ${Object.keys(info.explainMetrics)}`);
}
quickstart();
Contributor
There was a problem hiding this comment.
Checked with @danieljbruce offline. Updating the datastore version to 9.0.0 gets rid of the error
bhshkh
approved these changes
Jun 11, 2024
danieljbruce
added
the
owlbot:run
gcf-owl-bot
bot
removed
the
owlbot:run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes:
In samples/queryFilterOr.js, the sort isn't working properly. This results in a flakey test for OR queries because runQuery doesn't guarantee an order for results and relies the sort so that the results are in the right order for the assert statement. The sort function is fixed so it now works properly.
Samples files samples/queryProfileExplain.js, samples/queryProfileExplainAggregation.js, samples/queryProfileExplainAnalyze.js, samples/queryProfileExplainAnalyzeAggregation.js are added which contain the code samples that execute runQuery and runAggregationQuery for both query planning and query profiling. The filenames match the filenames in java-datastore.
In samples/test/queryFilterOr.test we pull out the code that sets the database to a specific state into a samples/test/helpers/populate-data.js file because we want to reuse this code to set the database to a specific state for the query profiling samples as well.
In samples/test/queryProfiling.test.js we add samples tests that run each of the 4 files with query profiling code samples and compare the output in each sample to the expected output for each code sample.