Skip to content

Add aggregate function for exponential smoothing#28914

Merged
alexey-milovidov merged 16 commits intomasterfrom
exp-smooth
Sep 21, 2021
Merged

Add aggregate function for exponential smoothing#28914
alexey-milovidov merged 16 commits intomasterfrom
exp-smooth

Conversation

@alexey-milovidov
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov commented Sep 12, 2021

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add aggregate function exponentialMovingAverage that can be used as window function. This closes #27511.

@robot-clickhouse robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Sep 12, 2021
@KochetovNicolai KochetovNicolai self-assigned this Sep 12, 2021
@alexey-milovidov
Copy link
Copy Markdown
Member Author

@Mergifyio update

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 15, 2021

Command update: success

Branch has been successfully updated

@mathalex mathalex self-assigned this Sep 16, 2021
@alexey-milovidov
Copy link
Copy Markdown
Member Author

alexey-milovidov commented Sep 18, 2021

Looks like "moving average" that I've implemented does not make any physical (natural) sense and also ill-defined, and "moving sum" should have the last value multiplied by 1 / k, so the sum over sequence of ..., 1, 1, 1, 1 will yield 1.

0.0009775171065493646
0.0009775171065493646
0.0009775171065493646
0.0009775171065493646
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not clear to me why this value is always the same, reading from _mt storage.

Copy link
Copy Markdown
Member Author

@alexey-milovidov alexey-milovidov Sep 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now calculation of moving average is made that way, so it does not depend on order of values.
(Only depends on what will be the latest value by time)

AggregateFunctionExponentialMovingAverage(const DataTypes & argument_types_, const Array & params)
: IAggregateFunctionDataHelper<ExponentiallySmoothedAverage, AggregateFunctionExponentialMovingAverage>(argument_types_, params)
{
if (params.size() != 1)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe set default param = 1 / ln(2) so that it would be exp(-time)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it will make window size fractional that feels unnatural.
(Window size analogue to simple sliding window is 2 * half_decay_time)

@alexey-milovidov
Copy link
Copy Markdown
Member Author

I will merge and maybe @mathalex will have any thoughts later...

@alexey-milovidov alexey-milovidov merged commit 3203fa4 into master Sep 21, 2021
@alexey-milovidov alexey-milovidov deleted the exp-smooth branch September 21, 2021 20:52
@sevirov
Copy link
Copy Markdown
Contributor

sevirov commented Sep 22, 2021

Internal documentation ticket: DOCSUP-15237

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exponentially smoothed moving average as aggregate function.

5 participants