Skip to content

Add StateStore query API#1153

Merged
artursouza merged 1 commit into
dapr:masterfrom
dmitsh:ds-query
Oct 18, 2021
Merged

Add StateStore query API#1153
artursouza merged 1 commit into
dapr:masterfrom
dmitsh:ds-query

Conversation

@dmitsh

@dmitsh dmitsh commented Sep 16, 2021

Copy link
Copy Markdown
Contributor

Description

Added StateStore query API and its implementation for MongoDB

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: dapr/dapr#3662

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

@dmitsh
dmitsh requested review from a team as code owners September 16, 2021 23:00
@dmitsh dmitsh changed the title Add StateStore query API Add StateStore query API (WIP, do not merge) Sep 16, 2021
@dmitsh dmitsh changed the title Add StateStore query API (WIP, do not merge) Add StateStore query API Sep 21, 2021
token string
}

func (q *Query) VisitEQ(f *query.EQ) (string, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to the query text, can this also contain the map of parameters as a response? I think we should support prepared statement natively, otherwise, user code would still be vulnerable to SQL injection if the value is hand crafted for that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function argument query.EQ already contains the exact key/value. What is the purpose and content of the suggested map of parameters?

return str, nil
}

func (q *Query) visitFilters(op string, filters []query.Filter) (string, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, instead of returning string, it could return interface{}, so for databases like MongoDb, it would already be the struct that the filter expects.

@dmitsh dmitsh Sep 24, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function returns string because it generates a textual query

opts = append(opts, documentdb.Continuation(q.token))
}
items := []CosmosItem{}
resp, err := client.QueryDocuments(collection.Self, &q.query, &items, opts...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


func (q *Query) VisitEQ(f *query.EQ) (string, error) {
// { <key>: <val> }
return fmt.Sprintf("{ %q: %q }", f.Key, f.Val), nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key or value could have a content that could turn this string into an invalid JSON. Instead, build the query object directly in the visitor methods.

daixiang0
daixiang0 previously approved these changes Oct 15, 2021

@daixiang0 daixiang0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall.

Add StateStore query API implementation for MongoDB, CosmosDB
@codecov

codecov Bot commented Oct 15, 2021

Copy link
Copy Markdown

Codecov Report

Merging #1153 (392fa5d) into master (310b4fd) will increase coverage by 0.65%.
The diff coverage is 48.60%.

❗ Current head 392fa5d differs from pull request most recent head 29f6f05. Consider uploading reports for the commit 29f6f05 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1153      +/-   ##
==========================================
+ Coverage   34.34%   35.00%   +0.65%     
==========================================
  Files         140      145       +5     
  Lines       11993    12479     +486     
==========================================
+ Hits         4119     4368     +249     
- Misses       7450     7651     +201     
- Partials      424      460      +36     
Impacted Files Coverage Δ
bindings/azure/cosmosdb/cosmosdb.go 21.42% <0.00%> (-1.36%) ⬇️
bindings/azure/storagequeues/storagequeues.go 36.93% <0.00%> (-0.68%) ⬇️
internal/component/redis/redis.go 0.00% <0.00%> (ø)
internal/component/redis/settings.go 71.42% <ø> (ø)
pubsub/azure/servicebus/subscription.go 0.00% <0.00%> (ø)
state/azure/cosmosdb/cosmosdb.go 15.32% <0.00%> (-0.72%) ⬇️
state/mongodb/mongodb.go 14.50% <0.00%> (-1.80%) ⬇️
state/requests.go 0.00% <ø> (ø)
state/store.go 61.53% <ø> (ø)
pubsub/azure/servicebus/servicebus.go 30.29% <12.50%> (+0.09%) ⬆️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67d5a8c...29f6f05. Read the comment docs.

@artursouza artursouza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it with MongoDB and the result is base64 encoded:image

The bulk Get API returns the document correctly:
image

@artursouza artursouza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've found the fix for the base64 encoded result. We basically need to create a separate response object and use jasonite.RawMessage:

https://github.com/dmitsh/dapr/commit/71efbe23fb9299928cc20073a0120644458334ff

For gRPC API it seems to simply be byte[] in data field.

Comment thread state/responses.go
Key string `json:"key"`
Data []byte `json:"data"`
ETag *string `json:"etag,omitempty"`
Error string `json:"error,omitempty"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why there can be error per key? For bulk get it makes sense but why for query?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why there can be error per key? For bulk get it makes sense but why for query?

If the data is encoded, we decode each entry separately (similarly to bulk-get), hence the error could happen for individual keys.

@dmitsh

dmitsh commented Oct 18, 2021

Copy link
Copy Markdown
Contributor Author

I've found the fix for the base64 encoded result. We basically need to create a separate response object and use jasonite.RawMessage:

dmitsh/dapr@71efbe2

For gRPC API it seems to simply be byte[] in data field.

Thanks, I tried this fix and it worked!!!, but your change belongs to the corresponding PR in dapr/dapr.
Once this PR is merged, I could finalize dapr/dapr#3695

@artursouza
artursouza merged commit e369fec into dapr:master Oct 18, 2021
@dmitsh
dmitsh deleted the ds-query branch October 18, 2021 20:58
amimimor pushed a commit to amimimor/components-contrib that referenced this pull request Dec 9, 2021
Add StateStore query API implementation for MongoDB, CosmosDB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

State Store Query API

3 participants