Skip to content

feat(sql): add generate_series() functions#5761

Merged
bluestreak01 merged 17 commits intomasterfrom
nw_generate_series
Jun 27, 2025
Merged

feat(sql): add generate_series() functions#5761
bluestreak01 merged 17 commits intomasterfrom
nw_generate_series

Conversation

@nwoolmer
Copy link
Copy Markdown
Contributor

@nwoolmer nwoolmer commented Jun 18, 2025

There have been some pain points with timestamp_sequence for a while:

  1. It only supports microsecond increments, rather than string units (i.e. '5d', '15m')
  2. It is a stateful function, with the state incrementing on every touch, rather than only on hasNext calls.
  3. The function is driven by long_sequence.

For example:

SELECT timestamp_sequence(  
    '2025-06-13T09:00:00'::timestamp,
    60000000L
  ) AS ts FROM long_sequence(60) 
  ORDER BY ts;

This should return timestamps in minutes, but it actually returns them in two minute increments.

ts
2025-06-13T09:00:00.000000Z
2025-06-13T09:02:00.000000Z
2025-06-13T09:04:00.000000Z
2025-06-13T09:06:00.000000Z
2025-06-13T09:08:00.000000Z
...

This is because the function is accessed both for the ORDER BY ts clause and in the ts projection.

There are workarounds, for example:

SELECT ('2025-06-13T09:00:00'::timestamp + (x * 6_000_000_0L)) as ts FROM (
  SELECT x - 1 as x FROM long_sequence(61)
)
ORDER BY ts;

Instead, we can now generate timestamps as a pseudo-table, using generate_series(NNL/NNS).

 generate_series(
    '2025-06-13T09:00:00', 
    '2025-06-13T10:00:00', 
    '1m'
  );
generate_series
2025-06-13T09:00:00.000000Z
2025-06-13T09:01:00.000000Z
2025-06-13T09:02:00.000000Z
2025-06-13T09:03:00.000000Z
2025-06-13T09:04:00.000000Z
...

In Postgres, the range is inclusive. Therefore, you must use '2025-06-13T10:00:00'::timestamp - 1 to omit the final entry, or just adjust the timestamp directly i.e '2025-06-13T09:59:59' or use dateadd i.e dateadd('u', -1, '2025-06-13T10:00:00').

This PR also includes variants for generating LONG and DOUBLE: generate_series(LL/LLL) and generate_series(DD/DDD). The default increment for the DD and LL cases will be 1 or 1.0.

If the third parameter is negative, the result set will be returned in reverse order:

generate_series(1, 5, -2);

generate_series
5
3
1

If the start/end values are back to front, they will be automatically swapped for you:

generate_series(1, 5, -2);
generate_series(5, 1, -2);
-- ^ these are equivalent ^

All variants support constants or bind variable constants.

This PR additionally fixed a long-standing bu when using non-keyed sample by fill prev, wherein dirty state would be left behind in SimpleMapValuePeeker. In the FILL(PREV) case, since this was not a constant fill, subsequent calls to the function would return different data.

Also, there was an issue in the same cursor where the first timestamp of the range would be skipped, which is now also fixed. This wasn't caught in testing because assertFactoryCursor would always execute assertTimestamp first, so both of the cursor calls were run with already 'dirtied' state.

Also fixes a dirty state bug in LastValueDoubleWindowFunctionFactory.

@nwoolmer nwoolmer added the SQL Issues or changes relating to SQL execution label Jun 18, 2025
@jerrinot jerrinot self-requested a review June 20, 2025 11:03
@jerrinot
Copy link
Copy Markdown
Contributor

jerrinot commented Jun 23, 2025

error position reporting looks fishy, see what's highlighted:
image

edit: I think we should store positions of all parameters, otherwise errors are hard to act on.

Copy link
Copy Markdown
Contributor

@jerrinot jerrinot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some easy to fix issues should be resolved before releasing it

@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 361 / 396 (91.16%)

file detail

path covered line new line coverage
🔵 io/questdb/griffin/SqlException.java 0 3 00.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesTimestampStringRecordCursorFactory.java 125 156 80.13%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesDoubleRecordCursorFactory.java 43 44 97.73%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesTimestampStringFunctionFactory.java 7 7 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesLongFunctionFactory.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesTimestampRecordCursorFactory.java 50 50 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesDoubleDefaultFunctionFactory.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/constants/LongConstant.java 1 1 100.00%
🔵 io/questdb/griffin/engine/groupby/SimpleMapValue.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesTimestampLongFunctionFactory.java 7 7 100.00%
🔵 io/questdb/cairo/sql/Function.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/constants/DoubleConstant.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/date/AbstractGenerateSeriesRecordCursorFactory.java 30 30 100.00%
🔵 io/questdb/griffin/FunctionParser.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesDoubleFunctionFactory.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesLongRecordCursorFactory.java 45 45 100.00%
🔵 io/questdb/griffin/engine/groupby/SimpleMapValuePeeker.java 5 5 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesLongDefaultFunctionFactory.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/date/TimestampAddFunctionFactory.java 11 11 100.00%
🔵 io/questdb/griffin/engine/groupby/SampleByFillValueNotKeyedRecordCursor.java 4 4 100.00%
🔵 io/questdb/griffin/engine/functions/window/LastValueDoubleWindowFunctionFactory.java 15 15 100.00%

@bluestreak01 bluestreak01 merged commit 2c5a34b into master Jun 27, 2025
37 checks passed
@bluestreak01 bluestreak01 deleted the nw_generate_series branch June 27, 2025 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

SQL Issues or changes relating to SQL execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants