Datatype Date32, support range 1925 to 2283 by liuneng1994 · Pull Request #25774 · ClickHouse/ClickHouse

liuneng1994 · 2021-06-28T08:57:44Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add a new datatype Date32 (store data as Int32), support date range same with Datetime64
support load parquet date32 to ClickHouse Date32
Add new function toDate32 like toDate
Support some function :

toYear
toMonth
toQuarter
toDayOfMonth
toDayOfWeek
toDayOfYear
toStartOfDay
toMonday
toISOWeek
toISOYear
toWeek
toYearWeek
toStartOfWeek
toStartOfMonth
toStartOfQuarter
toStartOfYear
toStartOfISOYear
toRelativeYearNum
toRelativeQuarterNum
toRelativeMonthNum
toRelativeWeekNum
toRelativeDayNum
toRelativeHourNum
toRelativeMinuteNum
toRelativeSecondNum
toYYYYMM
toYYYYMMDD
toYYYYMMDDhhmmss
addSeconds
addMinutes
addHours
addDays
addWeeks
addMonths
addQuarters
addYears
subtractSeconds
subtractMinutes
subtractHours
subtractDays
subtractWeeks
subtractMonths
subtractQuarters
subtractYears

Detailed description / Documentation draft:

I will add more docs in the future

Information about CI checks: https://clickhouse.tech/docs/en/development/continuous-integration/

# Conflicts: # src/DataTypes/IDataType.h # src/Functions/CustomWeekTransforms.h

liuneng1994 · 2021-06-28T11:07:51Z

retest this please

# Conflicts: # src/DataStreams/PostgreSQLBlockInputStream.cpp

liuneng1994 · 2021-07-05T10:13:45Z

@kssenii I fix some functional tests.There are still two failure cases, but they seem to have nothing to do with my code. And AST fuzzer failed， I didn't change the wrong line of code. I don't know how to fix it

kssenii · 2021-07-05T10:20:57Z

Ok! I'll try to review as soon as I can (most likely tomorrow).

base/common/DateLUTImpl.h

src/Functions/FunctionDateOrDateTimeAddInterval.h

src/IO/ReadHelpers.h

base/common/DateLUTImpl.h

src/AggregateFunctions/AggregateFunctionSequenceMatch.cpp

…kHouse into datatype-date32

kssenii · 2021-07-08T17:26:21Z

I think that here

ClickHouse/base/common/DateLUTImpl.h

Lines 1156 to 1164 in 9c2aad6

    
           inline ExtendedDayNum NO_SANITIZE_UNDEFINED addMonths(ExtendedDayNum d, Int64 delta) const 
        
           { 
        
               return toDayNum(addMonthsIndex(d, delta)); 
        
           } 
        
           inline Time NO_SANITIZE_UNDEFINED addQuarters(Time t, Int64 delta) const 
        
           { 
        
               return addMonths(t, delta * 3); 
        
           }

data types need to be adjusted (hint from fuzzer ../base/common/../common/DateLUTImpl.h:1173:35: runtime error), the max now will be Int32 after adding Date32, there is no need in int64. Lets make it Int32 in methods which then call inside addMonths. Please, check similar places there.

liuneng1994 · 2021-07-09T10:31:33Z

I think that here

ClickHouse/base/common/DateLUTImpl.h

Lines 1156 to 1164 in 9c2aad6

inline ExtendedDayNum NO_SANITIZE_UNDEFINED addMonths(ExtendedDayNum d, Int64 delta) const

{

return toDayNum(addMonthsIndex(d, delta));

}

inline Time NO_SANITIZE_UNDEFINED addQuarters(Time t, Int64 delta) const

{

return addMonths(t, delta * 3);

}

data types need to be adjusted (hint from fuzzer ../base/common/../common/DateLUTImpl.h:1173:35: runtime error), the max now will be Int32 after adding Date32, there is no need in int64. Lets make it Int32 in methods which then call inside addMonths. Please, check similar places there.

I fix this issue, fuzzer test is ok

base/common/DateLUTImpl.h

src/Functions/FunctionDateOrDateTimeAddInterval.h

src/IO/ReadHelpers.h

kssenii · 2021-07-09T11:39:20Z

Also please run ClickHouse/utils/generate-ya-make/generate-ya-make.sh script to fix Yandex third-party checks . It will add some changes to ya.make files.

liuneng1994 · 2021-07-12T01:55:12Z

Also please run ClickHouse/utils/generate-ya-make/generate-ya-make.sh script to fix Yandex third-party checks . It will add some changes to ya.make files.

I have run this script, but third-party checks still fail

src/Core/ya.make

liuneng1994 · 2021-07-12T13:16:40Z

@kssenii failed again...

kssenii · 2021-07-12T13:23:33Z

Yandex third-party checks (only for Yandex employees) failure not related to changes, its broken in master: Code: 46. DB::Exception: Received from localhost:38565. DB::Exception: Unknown function h3ToChildren: While processing h3ToChildren(599405990164561919, 100).

gyuton · 2021-07-15T02:45:17Z

Internal documentation ticket: DOCSUP-11585.

alexey-milovidov · 2021-07-15T07:53:30Z

@kssenii One bug was found: #26351
I will revert this PR before the bug will be fixed.

Continuation of #25774

liuneng1994 · 2021-07-28T03:44:13Z

I found that my github id disappeared from the code base. This is a sad story. @kssenii

alexey-milovidov · 2021-07-28T08:11:17Z

@liuneng1994 Your commits are recorded:

milovidov@milovidov-desktop:~/work/ClickHouse$ git log | grep 'neng.liu'
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>

And you will appear in the system.contributors table and in the authors in changelog.
The full history of development is kept, including your commits, revert and revert of revert.

liuneng1994 · 2021-07-28T08:21:28Z

@liuneng1994 Your commits are recorded:

milovidov@milovidov-desktop:~/work/ClickHouse$ git log | grep 'neng.liu'
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>
Author: neng.liu <[email protected]>
Author: Neng Liu <[email protected]>

And you will appear in the system.contributors table and in the authors in changelog.
The full history of development is kept, including your commits, revert and revert of revert.

OK, thanks very much

liuneng1994 and others added 9 commits June 7, 2021 06:41

date32 initial code

120da1f

add datetype32 part2

88a0d0b

date32

2bd7e6e

add datetype32 function support

b6eab5f

fix some compile problem

45e9e9f

Merge branch 'master' into datatype-date32

7ed1728

# Conflicts: # src/DataTypes/IDataType.h # src/Functions/CustomWeekTransforms.h

add some functional test

aadc2af

add more test case

4769edb

add parquet test and fix add interval error when type is date32

ec7604b

robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Jun 28, 2021

liuneng1994 changed the title ~~Datatype date32~~ Datatype Date32, support range 1925 to 2283 Jun 28, 2021

Avogar added the can be tested label Jun 28, 2021

fix test error

107616b

Neng Liu and others added 3 commits June 29, 2021 10:11

run test reference in Europe/Moscow timezhone

2f6aed7

fix style error

00377e7

fix some test error

17bbfd7

kssenii self-assigned this Jul 2, 2021

liuneng1994 added 4 commits July 3, 2021 22:15

fix test error

b878142

Merge remote-tracking branch 'origin/master' into datatype-date32

dc0c65e

Merge branch 'master' into datatype-date32

8b88988

# Conflicts: # src/DataStreams/PostgreSQLBlockInputStream.cpp

resolve conflicts

e36ce3e

liuneng1994 closed this Jul 7, 2021

liuneng1994 reopened this Jul 7, 2021

kssenii reviewed Jul 7, 2021

View reviewed changes

liuneng1994 added 2 commits July 8, 2021 00:06

fix pr comment

100d30a

Merge branch 'datatype-date32' of https://github.com/liuneng1994/Clic…

e47aadf

…kHouse into datatype-date32

liuneng1994 added 3 commits July 9, 2021 11:35

change addQuarter delta's datatype from Int64 to Int32

e172ca2

change addWeeks delta's datatype from Int64 to Int32

1d7d973

add type cast

78f0bb9

liuneng1994 requested a review from kssenii July 9, 2021 10:31

kssenii approved these changes Jul 9, 2021

View reviewed changes

base/common/DateLUTImpl.h Outdated Show resolved Hide resolved

src/Functions/FunctionDateOrDateTimeAddInterval.h Show resolved Hide resolved

src/IO/ReadHelpers.h Show resolved Hide resolved

liuneng1994 added 2 commits July 11, 2021 04:55

add comment and update ya.make

4669839

fix test error

dfe05ca

liuneng1994 force-pushed the datatype-date32 branch from 9a0e97a to dfe05ca Compare July 11, 2021 13:14

kssenii reviewed Jul 12, 2021

View reviewed changes

src/Core/ya.make Outdated Show resolved Hide resolved

Update ya.make

ec9a30d

kssenii merged commit 4a6b4d6 into ClickHouse:master Jul 12, 2021

liuneng1994 mentioned this pull request Jul 14, 2021

A new datatype Date32 need support housepower/ClickHouse-Native-JDBC#355

Closed

alexey-milovidov mentioned this pull request Jul 15, 2021

Revert "Datatype Date32, support range 1925 to 2283" #26352

Merged

kssenii mentioned this pull request Jul 15, 2021

Continuation of #25774 #26364

Merged

kssenii added a commit that referenced this pull request Jul 18, 2021

Merge pull request #26364 from kssenii/fixing-date32

b0ddc4f

Continuation of #25774

adevyatova mentioned this pull request Aug 20, 2021

DOCSUP-11585: Document Date32 data type #27919

Merged

liuneng1994 mentioned this pull request Aug 27, 2021

Extend range of Datetime64 and Date32 #28216

Closed

baibaichen mentioned this pull request Aug 27, 2021

Can't load parquet file which contains date before 1970 #21542

Closed

Conversation

liuneng1994 commented Jun 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liuneng1994 commented Jun 28, 2021

Uh oh!

liuneng1994 commented Jul 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kssenii commented Jul 5, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kssenii commented Jul 8, 2021

Uh oh!

liuneng1994 commented Jul 9, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kssenii commented Jul 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liuneng1994 commented Jul 12, 2021

Uh oh!

Uh oh!

liuneng1994 commented Jul 12, 2021

Uh oh!

kssenii commented Jul 12, 2021

Uh oh!

gyuton commented Jul 15, 2021

Uh oh!

alexey-milovidov commented Jul 15, 2021

Uh oh!

liuneng1994 commented Jul 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Jul 28, 2021

Uh oh!

liuneng1994 commented Jul 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

liuneng1994 commented Jun 28, 2021 •

edited

Loading

liuneng1994 commented Jul 5, 2021 •

edited

Loading

kssenii commented Jul 9, 2021 •

edited

Loading

liuneng1994 commented Jul 28, 2021 •

edited

Loading