spark: array_contains compares floats by bits, where Spark treats -0.0 as 0.0 and all NaNs as equal #34

Describe the bug

SparkArrayContains in the datafusion-spark crate (datafusion/spark/src/function/array/array_contains.rs) delegates the element comparison to array_has, which compares values by their bits. Spark's ArrayContains compares elements with genEqual, under which -0.0 equals 0.0 and all NaNs are equal, at any depth of a struct or array element. So the two disagree on those values:

The null semantics patched in on top of array_has are right; only the equality differs.

To Reproduce

use std::sync::Arc;

use arrow::array::{Float64Array, ListArray};
use arrow::datatypes::{DataType, Field};
use datafusion_common::ScalarValue;
use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
use datafusion_spark::function::array::array_contains::SparkArrayContains;

let array = ListArray::new(
    Arc::new(Field::new("item", DataType::Float64, true)),
    arrow::buffer::OffsetBuffer::new(vec![0, 1].into()),
    Arc::new(Float64Array::from(vec![-0.0])),
    None,
);
let result = SparkArrayContains::default().invoke_with_args(ScalarFunctionArgs {
    args: vec![
        ColumnarValue::Array(Arc::new(array)),
        ColumnarValue::Scalar(ScalarValue::Float64(Some(0.0))),
    ],
    arg_fields: vec![],
    number_rows: 1,
    return_field: Arc::new(Field::new("result", DataType::Boolean, true)),
    config_options: Default::default(),
})?;
// Spark returns true for array_contains(array(-0.0D), 0.0D); this returns false.

Expected behavior

array_contains matches Spark's genEqual for float and double elements: -0.0 equals 0.0, and any NaN equals any other NaN, including inside struct and array elements.

Additional context

Found while working on apache/datafusion-comet#6520. Comet currently routes array_contains on float elements away from this function because of the mismatch, and is adding its own kernel with Spark's equality in the meantime; a fix here would let it drop that kernel. arrays_overlap (#20781) in the same crate likely has the same equality.