…ration (#1251)

* feat: New PropertyMask field which allows partial commits, lookups, and query results

PiperOrigin-RevId: 635449160

Source-Link: googleapis/googleapis@dde0ec1

Source-Link: googleapis/googleapis-gen@8caa60d
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOGNhYTYwZDlhZWE4Mjk2NGExOWNkZjhmYWY5MTM4NDkxMWRiOGJkZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* build: update gapic generator to allow individual location mixin generation

PiperOrigin-RevId: 638462084

Source-Link: googleapis/googleapis@e5d5636

Source-Link: googleapis/googleapis-gen@d02b2f9
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZDAyYjJmOTIzOWMzZTFkZTY0N2VmMzdmYjEwYTVlNDI0YWEwYjdhNSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: danieljbruce <[email protected]>

* Add the right files from the preview branch

Copy paste over the files from the preview branch

* Thread Query mode through the library

Thread the query mode through the client library. Allow calls to get made using the query mode.

* Create a basic workflow for testing profiling grpc

This code allows basic query profiling to work and creates a basic code snippet that can be used to make it work.

* This test makes a call to the mock server

This will be useful for debugging grpc issues.

* Now the test works with the mock server in repo

This mock server will be super useful for ensuring that the right data gets transmitted in private preview so that we can confirm the client library is working correctly.

* Delete js file

We use the ts file instead.

* Try explain

See if Explain gets passed along to the mock server.

* Update protos.json

Protos.json is needed to pass the plan along.

* Try an aggregate query

* Add test to catch breaking change to return type

Make sure changes don’t break the use of this callback by adding this test.

* Try redefining RunQueryCallback

Define it with a more general data structure.

* Revert Gapic changes to match head branch

The merge caused generated files to have a diff. The diff should not be in the PR.

* Remove only

only should not be used here

* Add data structures for return values

Data structures need to be added for the return values that contain stats.

* Add plumbing to send stats info back

Add stats to info if it is available. If there are no results then end the stream and send the info back. This way, stats will always be sent in info if they are available and the program won ’t break if there are no results.

* Set test to only that we intend to explore

* Add a comment about stats

Explain what happens when the result set is empty. Just send the stats back.

* Delete the mock server code

The mock server code was used for debugging and now we don’t need it since the service is working.

* Remove calls to nightly

Calls to nightly don’t have a second database to work with. Regular calls work now so nightly calls are not necessary.

* Introduce profiling tests again

Bring the query profiling tests back.

* Revert "Remove calls to nightly"

This reverts commit 040d0a5.

* Stats are optional

Stats do not necessarily come from the server.

* Write some tests to test each mode

Each query profiling mode needs to be explored.

* Add code for parsing the stats returned

Stats returned needs to be parsed by a special library that will remove the complexities of the Struct object.

* Add dependencies necessary for parsing

A library is needed for parsing the Struct values. Add the libraries and use them.

* Add assertions for the expected plan/stats

Expected plan and expected stats should be used in the tests. This ensures the tests check for the proper stats values.

* Refactor info with stats build and add info to cb

Add a specific type for the runAggregationQuery callback so that it can now support the info parameter. In order to allow runAggregationQuery to make use of creating info, we also refactor info construction into a separate function.

* Modify the parser for runAggregationQuery

The parser for runAggregationQuery should have a deterministic process for computing results.

* Add asserts for the return values of runAggregate

Make sure that the entities and the plan are correct.

* Complete tests for runQuery and aggregation query

The assertion checks for runQuery and runAggregationQuery should be done and they test each mode.

* Add tests for Query and AggregateQuery

Tests should ensure run functions work properly for the run function on both of these objects.

* Add initial transaction tests for runQuery

The runQuery tests have been added which get the right results. Next the right assert statements and info collection will be added to the tests.

* Add checks on info to the transaction tests

Checks against info are needed to be sure that stats are passed back to the caller properly.

* Fix tests for aggregate queries in transactions

Add tests for running aggregate queries inside transactions.

* Ran linter, added all tests for runQueryStream

Added a temporary test for runQueryStream. Also ran the linter.

* Change parsing of return values

Return values are going to look different for users. Change the code so that the parsing is done differently.

* Reformat the info function

This function is more readable if we eliminate some of the tertiary arguments and complex logic for building the info object.

* Change tests as a result of structure changes

The structure of the QueryInfo object is changed. Modify the tests to work with the new structure.

* Use import and not require

import is better for catching compile time errors and is more customary.

* Better spacing for imports

Change the spacing so that the imports are all in one place.

* Introduce a single function for checking the execution stats. Make sure all the tests use this function. Pull out the run query plan and the run aggregation query plan.

* Fix the tests so that they call the right fns

Add assertion checks to check the query plan against some expected value and make sure the right assertion checks are done for the right tests.

* Finish the tests for the streaming call

Finish the tests for specifying no mode, specifying normal mode, EXPLAIN mode and EXPLAIN_ANALYZE mode. Make sure the tests  pass.

* Delete code that will not be used anymore

There is a lot of boilerplate code that was needed for the streaming call. Get rid of it here.

* Make changes to match new proto

Code change to use new proto was made so that code will compile.

* Add Explain Metrics interface

Make slight change to withBeginTransaction so that code compiles under new structure. Also group plan and statistics under the new explainMetrics interface.

* Remove bytesReturned from test

Proto should not be passing along bytesReturned anymore.

* Fix system tests to use values matching new struct

* Remove calls to nightly

Feature is now fully ready so nightly tests should not be done because the feature is expected to work in production.

* Query profiling

Add test for runQuery. Send back plan summary and execution stats.

* Add a test for runAggregationQuery

runAggregationQuery needs a unit test to validate request/return data.

* Parameterize the query profiling tests

* run the linter

* Export Query Mode in index.ts

Query mode needs to be exported so that it can be accessed by the user.

* Query profiling samples

Add query profiling samples for runQuery and runAggregationQuery.

* Change data structure types to match return values

* Remove TODO

* remove import

* delete the query profiling samples

* Remove abstraction for RunQueryCallback

* Change the comment to describe new data types

* Remove TODO

* linting fixes

* Update type to include runAggregationQuery

* Put else back in

This change is actually simpler because it doesn’t introduce a let. It is also a much smaller diff.

* mode is not needed in sharedQueryOptions

* Revert "mode is not needed in sharedQueryOptions"

This reverts commit b8d0c63.

* Rearrange imports

rearrange the imports to simplify the diff.

* Undo old query profiling changes

* Remove dependencies that are not needed

* Pass analyze true in

* Fix the flakey test - sort values

* Move code out that is responsible for importing data into the datastore database in preparation for the samples.

Ensure query profiling output in the script matches what the sample tests expect.

* Write the sample for aggregate queries

* Add a description

* Rename the sample files

* Set files up so that they can be parsed

Rename the files too

* Add some scripts for just the explain option

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Remove logging of entities from the samples

* Update samples/queryProfileExplain.js

Co-authored-by: kolea2 <[email protected]>

* Update samples/queryProfileExplain.js

Co-authored-by: kolea2 <[email protected]>

* Update descriptions

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Don’t sort in the sample.

Make the tests indifferent about the order so that sorting isn’t required in the samples.

* Fix titles

* Add header

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: kolea2 <[email protected]>

* Move the state up, make it protected

Transaction state should move up to the super class

* Build new transaction options method

Add a method for building transaction options. Also add a method for detecting if a request is a transaction. Build new transaction options

* Correct the fn that builds transactions

Corrected getTransactionRequest. Build the request options properly.

* Set transaction to readOnly in system test.

When this system test isn’t set to be a readOnly transaction then the error occurs which says too much contention.

* Change #parseRunSuccess

Change parseRunSuccess so that it can be used more universally.

* Move parseRunSuccess up to protected

parseRunSuccess should move up to the super class because it now needs to be used there.

* Saves the transaction id

This change saves the transaction id returned from the server for read calls

* Add tests for testing read only transactions

Read only tests are needed

* Add tests that measure read time

Make sure that code using the new transaction option has better performance than code that doesn’t have it.

* ran linter

* Add a test for testing requests

Use the MockedTransactionWrapper to test requests being passed into the Gapic layer.

* Final changes to make test work

Mock out begin transaction. Test for newTransaction consistency type. Mock out commit.

* Add the transaction.run test

A transaction.run test is needed  for lookup, lookup, put, commit.

* run linter

* runQuery, lookup, put, commit

Add a test for this sequence of operations and ensure it works properly.

* runAggregationQuery, lookup, put, commit

Four operations that get all the results for running an aggregation query. Adding another test for these four operations.

* put, put, lookup, commit

Last test suite regarding new transaction unit tests.

* Add tests for the commits

Add a bunch of tests for the commit case. Check the commit gapic input.

* Add testing to ensure begin tx is called

Begin transaction should be called at least once. Add code here to increment the counter.

* Document #blockWithMutex

Remove commented code too.

* Document transaction state

* feat: new transaction feature branch

* Add a check for expired

Check for expired on most functions and write tests for the expired check.

* Add commit and rollback blocks

Check for expired state in the commit and rollback blocks.

* run the linter

* Don’t allow readtime to be specified in a txn

Matches the python client.

* throw error for both read time and consistency

* Add test for specifying readtime

* Remove the console logs

* Refactor the test

* Improve test to make sure Gapic layer isn’t called

Also, fix bug with read consistency.

* Run linter
Reorganize tests
New test for readtime and consistency

* Make change to allow the runAggregationQuery go up

* Should error when get is used

* Move tests over and refactor initialized datastore

* Remove only and remove imports

* Introduce parameterized testing for errors

* Use parameters in parameterized tests

* Migrate error tests to parameterized testing

* Change description

* Prepare second describe block for tests

* Revert "Prepare second describe block for tests"

This reverts commit 974117b.

* Always return after the error is sent back

This ensures that an extra call does not get made to the server.

* Add headers

* Tests and implementation for expired on rollback

* fix test

* Wrap rollbacks with a withBeginTransaction.

* Update the test so that it begins the tx before

* Throw error if transaction not started on rollback

* Remove only

* Add a comment to the test regarding new txn.run()

* Ensure that the errors get bubbled up

* Run. linter

* read time and consistency error

* Remove unnecessary change

* Eliminate the call to withBeginTransaction

* Throw error reported in documentation

* Work on streaming errors

Fix the concurrency tests and move the error reporting outside of the stream, add two tests for the streams to make sure errors get reported.

* Move the error throwing up the stack

This makes it so that it is reported in the streams.

* Fix linting errors

* Generate unit tests for getTransactionRequest

I am going to rewrite the getTransactionRequest function so it is a good idea not to change how it works.

* Generate unit tests

Fix the compiler error
Change code so that when readOnly is specified then readwrite options are ignored.
Fix the unit test to adopt this change.

* Make function one ternary operation

Add comments, simplify logic in the function.

* Add two comments

* Update the comment

* Run the linter

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>