The add(entity) method in a Transaction object is not multi-thread safe. Other methods (e.g. put, delete) may also not be safe but we have traced a failure in our server specifically to add. Our server uses Scala Futures and so uses multiple threads of execution.

The particular scenario we tracked down was in our new user creation processing where we add 4 entities in a transaction. However there was a probability that one of the entities didn't get saved. It wasn't always the same entity kind that "disappeared". We traced this by adding debug logging after the 'add' calls that shows the adds successfully completing - but subsequent processing would find one of the entities missing. We did note that it always seemed to be just one entity missing and when an entity did go missing it seemed to be the first entity kind to be added (thread execution order was not consistent across invocations).

We added synchronize on each of our DataStore calls and haven't encountered this issue since. So at a minimum this issue can serve as a warning to others. Maybe a note could be added to the documentation. But we are also patiently waiting for issue googleapis/google-cloud-java#12003 (async support) to be implemented and would hope that explicit synchronization to not be needed.