Skip to content

Add RM_ReplyWithBigNumber module API#9639

Merged
oranagra merged 18 commits intoredis:unstablefrom
sjpotter:bignumber
Oct 25, 2021
Merged

Add RM_ReplyWithBigNumber module API#9639
oranagra merged 18 commits intoredis:unstablefrom
sjpotter:bignumber

Conversation

@sjpotter
Copy link
Contributor

@sjpotter sjpotter commented Oct 17, 2021

Let modules use additional type of RESP3 response (unused by redis so far)

Also fix tests that where introduced in #8521 but didn't actually run.

Copy link
Collaborator

@yossigo yossigo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sjpotter Please also add test coverage for this API.

@oranagra oranagra changed the title support returing big numbers from modules Add RM_ReplyWithBigNumber module API Oct 17, 2021
@oranagra oranagra added release-notes indication that this issue needs to be mentioned in the release notes state:major-decision Requires core team consensus state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten labels Oct 17, 2021
Copy link
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@redis/core-team please approve

@oranagra oranagra added the approval-needed Waiting for core team approval to be merged label Oct 17, 2021
Copy link
Collaborator

@yossigo yossigo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some typo / whitespace issues - other than this LGTM.

@sjpotter
Copy link
Contributor Author

fyi, I was confused by how it compiled / ran as well, but I was just mirroring the existing test code.

@sjpotter
Copy link
Contributor Author

once I update PR, tests should be good to go, as had to fix a bunch of things to get them to work now which indicates that they are being used correctly.

@sjpotter
Copy link
Contributor Author

note that while the reply module tests are now running, I had to comment out 2 of them, to get them to work for now. I'll remove that to get this PR to fail after I see it passing

@sjpotter
Copy link
Contributor Author

sjpotter commented Oct 18, 2021 via email

@oranagra
Copy link
Member

note that while the reply module tests are now running, I had to comment out 2 of them, to get them to work for now. I'll remove that to get this PR to fail after I see it passing

the map test fails because it expects doubles (with .0 suffix) and gets plain integers, i think you should just fix the test.
i.e. the test expected to get a list rather than a dict, but it didn't take into account that in resp2 these doubles are strings and not real doubles (or maybe back then the test infra handled them differently). either way, you should just fix the test.

the attribute test actually expects the error in RESP2, but it puts the catch on the wrong line (should be on the execution of the rw.attribute command, rw.error is part of another test).

please fix these.

@sjpotter
Copy link
Contributor Author

so the map test was easy to fix, having trouble with the attribute test.

I've tried to simplify it to

        test {RM_ReplyWithAttribute: an set reply} {
            if {$proto == 2} {
                catch [r rw.attribute 3] e
                assert_match "Attributes aren't supported by RESP 2" $e
            } else {
                set res [r rw.attribute 3]
                assert_equal [dict create 0 0.0 1 1.5 2 3.0] $res
            }
        }

but it still fails on the catch in the proto == 2 run. when I comment out the proto == 2 run, it blocks forever.

@oranagra
Copy link
Member

i don't understand what failure you're getting. the code looks correct. maybe post the exact error?

i also don't understand why it blocks forever, IIUC you're running the resp3 branch of the test code with hello 2 so the rw.attribute is executed without a catch and it should cause the tcl code to crash (exception isn't caught)

p.s. you can use assert_error to verify the failure rather than catch it yourself.

@oranagra
Copy link
Member

ohh wait, i think your catch line should be using curly braces, not rectangular.
what your code currently does is execute the code in the brackets and pass the result to catch. rather than pass the script to catch so that it can execute it.

@sjpotter
Copy link
Contributor Author

yes, already figured out the curley bracket thing. still hangs though. I was wondering if it was because of an odd number of elements in rw_attribute() (i.e. its a map in practice), the function didn't line up with the expected response as it is) so I fixed that, but still hangs.

i.e. this

int rw_attribute(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    if (argc != 2) return RedisModule_WrongArity(ctx);

    long long integer;
    if (RedisModule_StringToLongLong(argv[1], &integer) != REDISMODULE_OK)
        return RedisModule_ReplyWithError(ctx, "Arg cannot be parsed as a integer");

    if (RedisModule_ReplyWithAttribute(ctx, integer) != REDISMODULE_OK) {
        return RedisModule_ReplyWithError(ctx, "Attributes aren't supported by RESP 2");
    }

    for (int i = 0; i < integer; ++i) {
        RedisModule_ReplyWithLongLong(ctx, i);
    }

    RedisModule_ReplyWithSimpleString(ctx, "OK");
    return REDISMODULE_OK;
}

I believe is wrong for multiple reasons, the not being a map and the OK return. However, fixing those things, doesn't cause it not to hang. but I am missing understanding of what attribute types are supposed to be. It seems to be addon data to normal requests, but that doesn't make sense as its own command/response then?

@sjpotter
Copy link
Contributor Author

current test tcl code is

        test {RM_ReplyWithAttribute: an set reply} {
            if {$proto == 2} {
                catch {[r rw.attribute 3]} e
                assert_match "Attributes aren't supported by RESP 2" $e
            } else {
                set res [r rw.attribute 3]
                assert_equal [dict create 0 0.0 1 1.5 2 3.0] $res
            }
        }

@sjpotter
Copy link
Contributor Author

perhaps because of the redis support code

          | {
              # ignore attributes for now (nowhere to store them)
              redis_read_map $id $fd
              continue
          }

@sjpotter
Copy link
Contributor Author

yes, changing it to

| {return [redis_read_map $id $fd]}

means no more hanging

@sjpotter
Copy link
Contributor Author

I have no idea why non module tests are failing. As there's no modificatoin to any code outside of module tcl tests and test module code.

@sjpotter
Copy link
Contributor Author

ok, figured it out my change for the attribute output, needs to go all rawread

-            | {
-                # ignore attributes for now (nowhere to store them)
-                redis_read_map $id $fd
-                continue
-            }
+            | {return [redis_read_map $id $fd]}

@sjpotter
Copy link
Contributor Author

sjpotter commented Oct 19, 2021

failing a defrag test, dont know why (perhaps only happened locally, and not in github checks)

Copy link
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, what's left is to handle the doc comments:
#9639 (comment)

Copy link
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with minor typos (assuming by English is any good)

@oranagra oranagra merged commit 12ce2c3 into redis:unstable Oct 25, 2021
oranagra added a commit that referenced this pull request Nov 1, 2021
The module test in reply.tcl was introduced by #8521 but didn't run until recently (see #9639)
and then it started failing with valgrind.
This is because valgrind uses 64 bit long double (unlike most other platforms that have at least 80 bits)
But besides valgrind, the tests where also incompatible with ARM32, which also uses 64 bit long doubles.

We now use appropriate value to avoid issues with either valgrind or ARM32

In all the double tests, i use 3.141, which is safe since since addReplyDouble uses
`%.17Lg` which is able to represent this value without adding any digits due to precision loss. 

In the long double, since we use `%.17Lf` in ld2string, it preserves 17 significant
digits, rather than 17 digit after the decimal point (like in `%.17Lg`).
So to make these similar, i use value lower than 1 (no digits left of
the period)

Lastly, we have the same issue with TCL (no long doubles) so we read
raw protocol in that test.

Note that the only error before this fix (in both valgrind and ARM32 is this:
```
*** [err]: RM_ReplyWithLongDouble: a float reply in tests/unit/moduleapi/reply.tcl
Expected '3.141' to be equal to '3.14100000000000001' (context: type eval line 2 cmd {assert_equal 3.141 [r rw.longdouble 3.141]} proc ::test)
```
so the changes to debug.c and scripting.tcl aren't really needed, but i consider them a cleanup
(i.e. scripting.c validated a different constant than the one that's sent to it from debug.c).

Another unrelated change is to add the RESP version to the repeated tests in reply.tcl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approval-needed Waiting for core team approval to be merged release-notes indication that this issue needs to be mentioned in the release notes state:major-decision Requires core team consensus state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants