Is your feature request related to a problem or challenge? Please describe what you are trying to do.

From security standpoint it would be great to have reproducible (byte-for-byte) release builds, however this issue is not about this (but is a prerequisite thereof).

Arrow downstream consumers consume Arrow source code, via crates or forking.

Im both these situations it would be great to guarantee that a given release actually builds, not only at the moment of the release.

Sometimes, a user needs to build an old version to reproduce a problem or fix a bug.

Describe the solution you'd like

If a commit passes a build on CI today, it should ideally pass the build on CI tomorrow.
Much like DataFusion did this in apache/datafusion#14479