This probably explains why your users list was slow to load -- when you load the users list it makes a query to Event for each user, which can be very slow when loading hundreds of users.

Also, for various reasons (mostly performance), we don't query the events table.

If we want this feature, we will need to implement this in a different way (we can discuss, if needed).

Ah, that would explain it indeed. 😬

I think it would still make sense to implement this, maybe as a new column in the user table, which is updated every time users in a to-be-defined area of Redash (e.g. query page, dashboard view, etc)?

I guess updating it on every request would be a non-trivial amount of UPDATE queries, unless of course it would be done from a Celery task. But that seems like a good way to increase the risk of creating a trampling herd situation when lots of interactions happen and the Celery queues fills up without auto-scaling.

Alternatively we could use a Redis hash (user id -> last active date) and just update that on every request.

I agree that updating it on every request will be brutal, even through a Celery job.

I think a reasonable workflow can be:

1. Update Redis whenever the user interacts with the system.
2. Have a periodic job that syncs this timestamp to the database.

I wouldn't add a dedicated column for this, but rather add a generic JSON blob to the users table and store it there (like we do with many other models).

@jezdez From what I know about Redis it doesn't store information if the process stops. Can you confirm we're okay with that behavior (i.e. small % chance of data being lost)? Also for the periodic job that syncs Redis to database is 10 minutes okay? Or an hour?

@alison985 Ah interesting, according to Redis' Docker page uses an append only storage format only when passed the --appendonly yes option in Docker, so you're right about that being not ideal.

For Mozilla IIRC we're running on AWS' Elasticcache Redis instances, which persist by default, so it'd be not a huge deal on our side.

@arikfr Do you prefer switching to persistent storage in the Docker files (maybe not a bad idea any to be able to restore Celery queues on failure)?

Actually Redis does persist information to disk on some schedule so it's not really lost.

@arikfr Hm, are you sure the docker-compose will use the specified volume /data automatically? I understand the default (when not using appendonly) is to simply fsync things to disk. But inside a container that's still ephemeral if the volume isn't mounted during container start. Which is the case I believe for the docker-compose setup. Or am I missing something?

@jezdez you're right, it will be kept as long as the container lives but if the container gets rebuild it will be lost.

It does make sense to mount a volume to give the user a chance to back it up, but it's not a huge priority as the data there is (relatively) short lived anyway.

And to clarify: when the container restarts the data is kept. It's only when the container gets removed the data will be lost.

@arikfr Understood. I've filed #2860 to track this.