Proposed changes

Issue Number: close #10683

Problem Summary:

Currently, vectorized udaf can only be implemented locally based on c++ or Java. There are some problems need to be considered:

• The cost for getting started is high
• The requirements for the implementation quality of functions are relatively high, otherwise it will affect the stability of be, such as leading to core

To solove this problem , I think the remote udaf function is a good way it can benifits:

• the cost for getting started is low, for it can use any language to impl udaf interface
• decouple with doris kernel， it good for be stability

We need to consider the performance, and whether the remote call will have a lot of overhead， Here I did a test

Test the time-consuming of summation operation with different amount of data

Step1. deploy with 2 BE node, Table scheam:

CREATE TABLE `table1` (
  `event_day` date NULL,
  `siteid` int(11) NULL DEFAULT "10",
  `citycode` smallint(6) NULL,
  `username` varchar(32) NULL DEFAULT "",
  `pv` bigint(20) SUM NULL DEFAULT "0"
) ENGINE=OLAP
AGGREGATE KEY(`event_day`, `siteid`, `citycode`, `username`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`siteid`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
)

Step2 create remote udaf

CREATE AGGREGATE FUNCTION  rpc_myadd(int) RETURNS int PROPERTIES (
    "TYPE"="RPC",
    "OBJECT_FILE"="127.0.0.1:9000",
    "update_fn"="rpc_sum_update",
    "merge_fn"="rpc_sum_merge",
    "finalize_fn"="rpc_sum_finalize"
);

Step3 import data with diffrent rows

Result

we can concluded:

• RPC udaf performance is better than that of non vectorized aggregate function by 6% - 29%
• The performance of RPC udaf is significantly lower than that of native vectorized aggregate functions, but when the number of data rows is 1000W, it can still return in seconds, and theoretically it can support most business scenarios

Checklist(Required)

1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...