panic: capacity overflow in generate_series / range with large i64 arguments

Describe the bug

generate_series and range panic with capacity overflow when given an integer range so large the count exceeds isize::MAX bytes. The panic comes from Vec::reserve inside the integer-range implementation, hit during planning (constant folding of the table-valued function).

To Reproduce

use datafusion::prelude::SessionContext;

#[tokio::main]
async fn main() {
    let ctx = SessionContext::new();
    let _ = ctx
        .sql("SELECT generate_series(0, 9223372036854775807)")
        .await
        .unwrap()
        .create_physical_plan()
        .await;
}

Panic:

thread 'main' panicked at .../alloc/src/raw_vec/mod.rs:28:5:
capacity overflow

Also reproduces with:

SELECT range(0, 9223372036854775807)
SELECT range(9223372036854775807)
SELECT generate_series(-9223372036854775808, 9223372036854775807)

Bounded ranges like SELECT generate_series(1, 100) are fine.

Expected behavior

Return a planning/execution error along the lines of "range too large to materialize" (or, ideally, a streaming implementation that does not need to materialize the full sequence eagerly). The public SQL API should never panic on user-supplied SQL.

Root cause

datafusion/functions-nested/src/range.rs, in generate_range_values:

// line 563-565   (step > 0 branch)
let count =
    (start.abs_diff(limit) / step.unsigned_abs()).saturating_add(1) as usize;
values.reserve(count);                                  // ← panics here

// line 583-585   (step < 0 branch — identical pattern)
let count =
    (start.abs_diff(limit) / step.unsigned_abs()).saturating_add(1) as usize;
values.reserve(count);

For generate_series(0, i64::MAX, 1) the count is ~u64::MAX/8 (after saturating_add(1)), which on a 64-bit target turns into a usize of ~9.2 × 10^18. Vec::<i64>::reserve multiplies by size_of::<i64>() = 8, sees that exceeds isize::MAX, and panics.

Suggested fix

Bound count at allocation time:

const MAX_RANGE_ELEMENTS: usize = isize::MAX as usize / std::mem::size_of::<i64>();
if count > MAX_RANGE_ELEMENTS {
    return exec_err!(
        "range too large: would produce {count} elements (max {MAX_RANGE_ELEMENTS})"
    );
}
values.reserve(count);

A friendlier limit (say, 1 GiB / 8 B = 128 M elements, configurable) would also stop this from being a memory-exhaustion DoS.

Additional context

Found by a cargo fuzz target (fuzz/fuzz_targets/sql_physical_plan.rs) seeded with SQL extracted from datafusion/sqllogictest/test_files/. The fuzzer triggered it from a mutated generate_series example by replacing a small numeric literal with 9223372036854775807 (i64::MAX).

@Dandandan

Thanks @Dandandan I also came up DataFusion is prone to overflows in #22137

Btw, what tool did you use to test this?

Found by a cargo fuzz target (fuzz/fuzz_targets/sql_physical_plan.rs) seeded with SQL extracted from datafusion/sqllogictest/test_files/. The fuzzer triggered it from a mutated generate_series example by replacing a small numeric literal with 9223372036854775807 (i64::MAX).

Was this a general-purpose fuzzer that mutates raw bytes, with SLT queries used as seeds?

Thanks @Dandandan I also came up DataFusion is prone to overflows in #22137

Btw, what tool did you use to test this?

This issue was using https://github.com/rust-fuzz/cargo-fuzz + Claude to come up with the code / handling the reports.

The further reports come from using /goal and asking to produce 20 unique reproducable crashes.

Found by a cargo fuzz target (fuzz/fuzz_targets/sql_physical_plan.rs) seeded with SQL extracted from datafusion/sqllogictest/test_files/. The fuzzer triggered it from a mutated generate_series example by replacing a small numeric literal with 9223372036854775807 (i64::MAX).

Was this a general-purpose fuzzer that mutates raw bytes, with SLT queries used as seeds?

Yeah raw bytes with SLT queries as seeds, indeed. Also it uses the generated binary AFAIK to find tokens / branches.

Generated Code:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

#![no_main]

use std::sync::Arc;
use std::sync::OnceLock;
use std::time::Duration;

use datafusion::arrow::array::{
    BooleanArray, Float64Array, Int32Array, Int64Array, StringArray,
    TimestampNanosecondArray,
};
use datafusion::arrow::datatypes::{DataType, Field, Schema, TimeUnit};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::catalog::MemTable;
use datafusion::error::Result;
use datafusion::execution::context::SessionContext;
use datafusion::prelude::SessionConfig;
use datafusion_common::instant::Instant;
use libfuzzer_sys::fuzz_target;

const DEFAULT_PLANNING_TIMEOUT_MS: u64 = 1_000;

fn runtime() -> &'static tokio::runtime::Runtime {
    static RT: OnceLock<tokio::runtime::Runtime> = OnceLock::new();
    RT.get_or_init(|| {
        tokio::runtime::Builder::new_current_thread()
            .enable_time()
            .build()
            .expect("failed to create tokio runtime")
    })
}

fn planning_timeout() -> Duration {
    static T: OnceLock<Duration> = OnceLock::new();
    *T.get_or_init(|| {
        std::env::var("DATAFUSION_FUZZ_PLANNING_TIMEOUT_MS")
            .ok()
            .and_then(|value| value.parse::<u64>().ok())
            .map(Duration::from_millis)
            .unwrap_or_else(|| Duration::from_millis(DEFAULT_PLANNING_TIMEOUT_MS))
    })
}

fuzz_target!(|data: &[u8]| {
    let sql = String::from_utf8_lossy(data);
    let started = Instant::now();

    runtime().block_on(async {
        // SQL, schema, and physical planning errors are expected fuzzer noise.
        // Panics and slow planning are the findings this target reports.
        let _ = plan_sql(&sql).await;
    });

    let elapsed = started.elapsed();
    let limit = planning_timeout();
    if elapsed > limit {
        eprintln!(
            "query exceeded {} ms after {} ms for SQL:\n{}",
            limit.as_millis(),
            elapsed.as_millis(),
            sql
        );
        panic!(
            "query exceeded {} ms after {} ms",
            limit.as_millis(),
            elapsed.as_millis()
        );
    }
});

async fn plan_sql(sql: &str) -> Result<()> {
    let ctx = fuzz_context()?;
    let dataframe = ctx.sql(sql).await?;
    let _batches = dataframe.collect().await?;
    Ok(())
}

fn fuzz_context() -> Result<SessionContext> {
    let config = SessionConfig::new()
        .with_information_schema(true)
        .with_target_partitions(1);
    let ctx = SessionContext::new_with_config(config);

    register_demo_table(&ctx, "t")?;
    register_demo_table(&ctx, "t1")?;
    register_demo_table(&ctx, "t2")?;

    Ok(ctx)
}

fn register_demo_table(ctx: &SessionContext, table_name: &str) -> Result<()> {
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int64, false),
        Field::new("a", DataType::Int64, true),
        Field::new("b", DataType::Float64, true),
        Field::new("c", DataType::Utf8, true),
        Field::new("x", DataType::Int32, true),
        Field::new("y", DataType::Boolean, true),
        Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true),
    ]));
    let id = Int64Array::from(vec![1, 2, 3, 4, 5]);
    let a = Int64Array::from(vec![Some(10), None, Some(-3), Some(0), Some(i64::MAX)]);
    let b = Float64Array::from(vec![
        Some(1.5),
        Some(-2.5),
        None,
        Some(0.0),
        Some(f64::INFINITY),
    ]);
    let c = StringArray::from(vec![Some("alpha"), Some(""), None, Some("z"), Some("héllo")]);
    let x = Int32Array::from(vec![Some(100), Some(-1), Some(0), None, Some(i32::MIN)]);
    let y = BooleanArray::from(vec![Some(true), Some(false), None, Some(true), Some(false)]);
    let ts = TimestampNanosecondArray::from(vec![
        Some(0),
        Some(1_000_000_000),
        None,
        Some(-1),
        Some(i64::MAX),
    ]);
    let batch = RecordBatch::try_new(
        Arc::clone(&schema),
        vec![
            Arc::new(id),
            Arc::new(a),
            Arc::new(b),
            Arc::new(c),
            Arc::new(x),
            Arc::new(y),
            Arc::new(ts),
        ],
    )?;
    let table = MemTable::try_new(schema, vec![vec![batch]])?;
    ctx.register_table(table_name, Arc::new(table))?;
    Ok(())
}

take

fix: return error instead of capacity overflow panic in generate_seri…

Navigation Menu

Repository navigation

panic: capacity overflow in generate_series / range with large i64 arguments #22188

Describe the bug

To Reproduce

Expected behavior

Root cause

Suggested fix

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

panic: capacity overflow in generate_series / range with large i64 arguments #22188

Description

Describe the bug

To Reproduce

Expected behavior

Root cause

Suggested fix

Additional context

Activity

comphead commented on May 15, 2026

2010YOUY01 commented on May 16, 2026

Dandandan commented on May 16, 2026

Dandandan commented on May 16, 2026

Dandandan commented on May 16, 2026

sweb commented on May 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions