Skip to content

Feature/create simple lambda function#23978

Merged
kitaisreal merged 46 commits intoClickHouse:create-user-defined-lambda-functionfrom
Realist007:feature/create-simple-lambda-function
Aug 17, 2021
Merged

Feature/create simple lambda function#23978
kitaisreal merged 46 commits intoClickHouse:create-user-defined-lambda-functionfrom
Realist007:feature/create-simple-lambda-function

Conversation

@Realist007
Copy link
Copy Markdown
Contributor

@Realist007 Realist007 commented May 9, 2021

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add feature for creating user-defined functions.

Detailed description / Documentation draft:

You can create your function with command 'create function MyFunc as ({parameters}) -> {function core}'. For example: 'create function sum as (a, b) -> a + b'

@robot-clickhouse robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels May 9, 2021
@alexey-milovidov alexey-milovidov marked this pull request as draft May 11, 2021 09:31
@alexey-milovidov
Copy link
Copy Markdown
Member

What is missing:

  • persistent storage of functions (in the functions directory as a list of .sql files);
  • DROP FUNCTION query;
  • access rights for functions manipulation.

@kitaisreal kitaisreal self-requested a review May 11, 2021 19:06
@kitaisreal kitaisreal self-assigned this May 11, 2021
@kitaisreal
Copy link
Copy Markdown
Contributor

@Realist007 please add more tests for functionality.

  1. Need to check how function will behave with constant arguments.
    For example CREATE FUNCTION custom_plus as (a, b) -> a + b;
    SELECT isConstant(custom_plus(1, 2)) should be true.
  2. With non constant arguments.
  3. Also need more tests to check for errors, it would be nice to add additional context for example SELECT custom_plus ('test', 'test').

@kitaisreal kitaisreal removed their request for review May 11, 2021 19:10
@alexey-milovidov
Copy link
Copy Markdown
Member

alexey-milovidov commented May 11, 2021

  • check for correct error messages when trying to create a function with the name of already existing builtin function;
  • check for correct error messages when trying to drop existing builtin function;
  • order of initialization on server startup if functions are dependent on each other;
  • check for correct error messages when trying to create recursive function;
  • the case for distributed queries when cluster nodes contain different implementation of the same function;
  • make sure that the function is JIT compiled.

class UserDefinedFunction;
using UserDefinedFunctionPtr = std::shared_ptr<UserDefinedFunction>;

class UserDefinedFunction : public IFunction
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be logical to use IFunctionOverloadResolver interface.

This UDF is polymorphic, and can be used with different types. However, we don't need to resolve types for every processing block. OverloadResolver can check types and build ready ExpressionActions. Those actions should be used in IFunctionBase.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, looks like we need to disable all useDefaultImplementation flags.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have specific values of arguments inside ExpressionActions. Why it will work if we build ExpressionAction once?

@filimonov
Copy link
Copy Markdown
Contributor

#11

@Realist007 Realist007 marked this pull request as ready for review August 16, 2021 13:57
@kitaisreal kitaisreal changed the base branch from master to create-user-defined-lambda-function August 17, 2021 18:22
@kitaisreal kitaisreal merged commit 8403f38 into ClickHouse:create-user-defined-lambda-function Aug 17, 2021
@kitaisreal kitaisreal mentioned this pull request Aug 17, 2021
@sevirov
Copy link
Copy Markdown
Contributor

sevirov commented Aug 18, 2021

Internal documentation ticket: DOCSUP-13375

kitaisreal added a commit that referenced this pull request Aug 25, 2021
@l1t1
Copy link
Copy Markdown

l1t1 commented Sep 18, 2021

will you support recursive function?

CREATE FUNCTION f3 AS a -> multiIf(a < 2, 1, f3(a) + f3(a + 1))

Query id: 38b7b875-5877-4081-ac79-ad0493cc6923

0 rows in set. Elapsed: 0.001 sec. 

Received exception from server (version 21.11.1):
Code: 611. DB::Exception: Received from localhost:9000. DB::Exception: You cannot create recursive function. (CANNOT_CREATE_RECURSIVE_FUNCTION)

@alexey-milovidov
Copy link
Copy Markdown
Member

@l1t1 Not in near time.

User defined SQL functions are substituted (as expression) firstly and only then optionally compiled and evaluated.
We cannot do infinite substitution, that's why recursion is not supported.

@ZendeberNucana
Copy link
Copy Markdown

ZendeberNucana commented Oct 18, 2021

Any ideas of how to support this?

CREATE FUNCTION UDFTestArrayMap AS (d) -> arrayMap( x -> JSONExtractKeysAndValues(x, 'String'), d )

DB::Exception: Identifier `x` does not exist in arguments. (UNKNOWN_IDENTIFIER) (version 21.10.2.15 (official build))

@SeaAndHillMe
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants