Skip to content

MOD-12347 Add support for NaN values#1853

Merged
AvivDavid23 merged 21 commits into
masterfrom
NaN-Values
Jan 14, 2026
Merged

MOD-12347 Add support for NaN values#1853
AvivDavid23 merged 21 commits into
masterfrom
NaN-Values

Conversation

@AvivDavid23

@AvivDavid23 AvivDavid23 commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Note

Adds comprehensive NaN support across ingestion, storage, queries, and compaction.

  • Introduces COUNTNAN and COUNTALL aggregations (CREATERULE/RANGE/REVRANGE/MRANGE; parsing, enum, dispatch, and info wired) and per-aggregator isValueValid to control NaN participation
  • Aggregations now skip NaNs by default; COUNTNAN counts only NaNs, COUNTALL counts all samples; vectorized max falls back when NaNs present
  • TWA: treats NaN-only buckets as empty; computes values via neighboring samples when EMPTY; skips NaNs when fetching boundary samples; tracks validSamplesInBucket to suppress NaN-only outputs unless EMPTY
  • Ingestion/API: accept NaN (case-insensitive) in TS.ADD/MADD; RESP2 prints "NaN", RESP3 returns float NaN; INCRBY/DECRBY reject NaN operands; IGNORE never deduplicates on NaN; clearer duplicate-policy error when NaN with MIN/MAX/SUM
  • Storage: normalize NaN to a canonical bit pattern in Gorilla compression; SIMD max path checks for NaNs; RDB version bumped with new rule field persisted
  • Misc: TS.INFO reflects new aggs; enums/string mappings extended; minor JSON schema tidy
  • Tests: extensive coverage for NaN ingestion, aggregations (incl. TWA), MRANGE/MGET, compactions, RDB compatibility, RESP3, and compression

Written by Cursor Bugbot for commit 25b0c8a. This will update automatically on new commits. Configure here.

Comment thread src/module.c
Comment thread src/filter_iterator.c
Comment thread src/filter_iterator.c Outdated
Comment thread src/compaction.c
Comment thread commands.json Outdated
Comment thread src/compaction.c Outdated
Comment thread src/compaction.c
Comment thread src/compaction.c Outdated
Comment thread src/consts.h Outdated
Comment thread src/filter_iterator.c Outdated
Comment thread src/generic_chunk.c Outdated
Comment thread src/gorilla.h
Comment thread src/module.c
Comment thread src/module.c Outdated
Comment thread src/module.c Outdated
Comment thread src/filter_iterator.c
Comment thread src/module.c
rule->startCurrentTimeBucket = currentTimestampNormalized;
RedisModule_CloseKey(key);
}
rule->aggClass->appendValue(rule->aggContext, value, timestamp);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TWA compaction loses interpolation data through NaN-only buckets

Medium Severity

When a TWA compaction rule processes a bucket containing only NaN samples, subsequent buckets lose the previous valid sample information needed for interpolation. The condition at line 651 checks hadValidSamples, but when a NaN-only bucket is skipped, hadValidSamples is false even though getLastSample still returns valid data from the bucket before the NaN-only one. This causes addPrevBucketLastSample to be skipped, breaking TWA interpolation for the next bucket.

Fix in Cursor Fix in Web

@AvivDavid23 AvivDavid23 merged commit ec82081 into master Jan 14, 2026
24 checks passed
@AvivDavid23 AvivDavid23 deleted the NaN-Values branch January 14, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants