Hi @quaquel

Do you prefer this approach or should we wait for RunConfiguration idea to crystallise?

Performance benchmarks:

Can you check the code coverage and if needed at tests? I'll merge later today or more likely tomorrow.

While writing tests, I found an interesting edge case in parquet backend, In old logic where we only pop from the in-memory buffer:

if is_overwrite:
    while buffer and buffer[-1]["time"] == time:
        buffer.pop()

If a file flush happens right before the overwrite (e.g., calling get_table_dataframe), the buffer is empty. The while loop skips, and it blindly appends the new row, missing the fact that the old row is already saved in the physical file and causing duplicate data.

It might be propagated or reflected in user's own custom backend as well. I am convinced that it is a delicate approach. Do you think these type of edge cases can be handled by run_configs easily?

If a file flush happens right before the overwrite (e.g., calling get_table_dataframe), the buffer is empty. The while loop skips, and it blindly appends the new row, missing the fact that the old row is already saved in the physical file and causing duplicate data.

It might be propagated or reflected in user's own custom backend as well. I am convinced that it is a delicate approach. Do you think these type of edge cases can be handled by run_configs easily?

That's an interesting edge case. In my current conception of run_config, we would not need the overwrite as used in this PR, so it would avoid the edge case. We would not need it because run_ended is only emitted once, and it would leave the model in a state where any subsequent run_x calls would fail.

On the edge case for the current approach, what about never flushing the last item in the buffer?

On the edge case for the current approach, what about never flushing the last item in the buffer?

Agree. However the trade-off for this approach, is that users would be strictly required to fetch their data through get_table_dataframe() to see the terminal state. If they bypass the recorder API and load the raw .parquet file directly from the disk, that final timestep will be missing

Agree. However the trade-off for this approach, is that users would be strictly required to fetch their data through get_table_dataframe() to see the terminal state. If they bypass the recorder API and load the raw .parquet file directly from the disk, that final timestep will be missing

Fair point. I am unfortunately not familiar with parquet, but it seems the real fix will require some check of the file and see if something needs to be overwritten. The other option is to at least document this for parquet.

I have already implemented following approach: if the in-memory buffer is empty, it falls back to opening the physical .parquet file, filters out the old timestep (df = df[df["time"] != time]), and rewrites the file before appending the new data.
Are you comfortable with this approach of explicitly manipulating the .parquet file for this edge case, or would you prefer we stick to the simpler buffer logic and just document the quirk?

As indicated, I have little knowledge regarding parquet, so I'll trust your judgment.