Follow-up to #95

We should have a continuous (nightly) job that tests with presubmits. We usually have PRs open, but it'd be good to have timely notifications from flakey bot if something happens when we aren't making a PR.

Also, we should double-check that as we add pyarrow and such to required dependencies that we're still able to test with pre-release versions. It'd be good to ignore our pins so that we know when a dependency makes a breaking change.