reproducible historical feature queries #2980

Is your feature request related to a problem? Please describe.

I'd like a way to use Feast to run the exact same historical query and reproduce the same result the next day regardless of "late arriving" or "restating" data.

For example: If I called get_historical_features Tuesday at noon to run a model: On Wednesday I'd like to be able to run the run a new and improved model over the same data (to prove it really is a better model, not just benefiting from newer or better data). I don't see a way to pass created_timestamp=Tuesday@noon to get_historical_features to do that.

Describe the solution you'd like
I presume that a parameter like max_created_time would be passed to get_historical_features.

Describe alternatives you've considered
Some offline stores have a built in snapshot or time travel capabilities. These generally have limited retention windows and would still require changes to the queries used by feast to take advantage of.

Additional context
For context I'm working on scenarios where we are not the primary observer of the events in our database. We might learn from a 3rd party at time X that an event happened at time Y, or learn at time A that a prediction was made at time B for what would happen at future time C. We spend a lot of effort reasoning about which timestamps actually gives a sane ordering, and those different timestamps are sometimes relatively far apart.