This can probably made much simpler, e.g.:

bool* out_value = &checked_cast<BooleanScalar*>(out)->value;
if (in.type->id() == Type::FLOAT) {
  *out_value = !options.nan_is_null || !std::isnan(internal::UnboxScalar<FloatType>::Unbox(in));
}
else if (in.type->id() == Type::DOUBLE) {
  *out_value = !options.nan_is_null || !std::isnan(internal::UnboxScalar<DoubleType>::Unbox(in));
}
else {
  *out_value = true;
}
return Status::OK();

I would add a check for validity prior to checking float type to prevent unnecessary checks when null bit is enabled. Nit: Also, I would change logic to to use && instead of ||, feels a bit more readable (and less characters).

    bool* out_value = &checked_cast<BooleanScalar*>(out)->value;
    if (in.is_valid) {
      switch (in.type->id()) {
        case Type::FLOAT:
          *out_value = options.nan_is_null && std::isnan(internal::UnboxScalar<FloatType>::Unbox(in));
        case Type::DOUBLE:
          *out_value = options.nan_is_null && std::isnan(internal::UnboxScalar<DoubleType>::Unbox(in));
        default:
          *out_value = false;
      }
    } else {
      *out_value = true;
    }

@pitrou I would be curious if it is worth it to have a specialized version for floating point types (similar to how arithmetic kernels are implemented), as the options.nan_is_null check does not applies to other data types.

This is the version for Scalar inputs. Micro-optimizing it is futile. The version for Array inputs may be more interesting ;-)

You can put this in the anonymous namespace above.

It was addressed on 2e772c1

This is comparing pointer values, which is incorrect, even though it will do what you expect most of the time.
(one could e.g. instantiate a separate instance using std::make_shared<FloatType>())

Remove the break statement for the default case.

Removed on 9a69e3e

Note the check for NaN values only applies to floating-point types and when nan_is_null is true, other types/cases can use the logic as it was.

In the case of ArrayData there are 3 scenarios:
a) arr.GetNullCount() == arr.length // All data is null
b) arr.GetNullCount() == 0 // No data is null
c) arr.MayHaveNulls() == true // Some data is null

IIUC, the null bitmap of the input ArrayData is not guaranteed to be consistent with the data (an ArrayData can be malformed bc buffer values can be modified directly). Scenarios (a)-(b) invoke arr.GetNullCount() which iterates through all the arr values and update the null count.

Given that scenarios (b)-(c) are the common case and the array data has to be traversed to identify the NaN values, (as an optimization) I suggest to not check the null count at all. Nevertheless, only check for NaNs in non-null indices.

I have pushed f404910 which contains the ArrayData validity for nulls (not sure if it's the best way to do).
There is a way to test those changes in R ? As the ticket is also related to R language.

No need to test kernel implementation in R (or Python), first bindings have to be put in place with the desired default option. The C++ kernels are invoked via the language bindings.

@Christian8491 when this PR is ready, the C++ code is all reviewed and approved, and the CI is all green, please let me know and I can push a commit that changes the R bindings to use this new option.

Sure @ianmcook I will.

@pitrou The is_null compute function most of the time will iterate through the bitmap of the input ArrayData and when nan_is_null option is set, it can increase the null count. Would it be a good idea to update the null count of the input array (arr.SetNullCount(...)) as a side-effect of invoking this compute function?

Why would you update the null count of the input array? I'm not sure I understand what you're proposing.

I see your point now, thanks! Modifying the input array does not makes sense because nan_is_null property applies only to the intermediate/output of the compute function. Please ignore these comments.

Rephrase to Output is true if the corresponding input element is null. NaN values can be considered as null via ...

Addressed on 915c868

I suggest renaming this NullOptions, in case other values than NaN need to be considered null at some point.

By convention, function descriptions are explicitly line-wrapped with \n characters. I suggest doing the same here.

I think the repetition can easily be avoided here, for example by making this a templated test.

I would make this argument keyword-only.

Same here.

We should try to maintain this alphabetically-ordered (though DayOfWeekOptions already violates it :-/).

These separate tests shouldn't be necessary, as CheckScalarUnary already tests Scalar values implicitly when Array values are given.