Equality semantics in Vortex #8826

It is not immediately obvious what we define as "equal" in Vortex, especially with nullability involved.

A good demonstration of our current semantics on develop are:

impl PartialEq for Scalar {
    fn eq(&self, other: &Self) -> bool {
        self.dtype.eq_ignore_nullability(&other.dtype)
            && self.value == other.value                // where `value` is `Option`
    }
}

We define equality over scalars as equality that ignores the nullability. This is useful because ideally, we want to say that a scalar that has type i32? (nullable integer) that has value 42 (a non-null value even though it is allowed to be null) is equal to the value 42, even if the type is non-nullable.

This logic can then be extended to arrays, where arrays should have the same semantics as a list of scalars.

I think this is an apt description, but other people might have different opinions. And it doesn't seem like we all agree that this is the correct semantics that we want.

cc @joseph-isaacs @robert3005