Investigate removing pandas dependency from Core Airflow

 I want to take a look at removing numpy/pyarrow from _core_ airflow. We don't really need it, and it would make installing easier.

After 2.0

_Originally posted by @ashb in https://github.com/apache/airflow/issues/11950#issuecomment-731115568_