Aaron and I discussed this and outlined the following plan, in which we identified that there are three mechanisms or scenarios for fetching posts off disk and passing them to the browser:

1. Initial page load

We can GET /api/posts and that will retrieve the default PageSize number of posts.

1. Backscroll through timeline

The further back into the past we go, the less chance there is of old posts arriving late and being missed. Therefore I think the mechanism we have in place will work fine, simply asking for all posts before a low-water mark.

1. Stream of created and deleted posts

This is the regular running operation, whereby posts are created and removed by the sync application. For this, we want to use websockets and inotify. The Go package that implements a cross-platform inotify system is here. Websocket support is in a pull request that needs to be reviewed. Every time a filesystem event happens, and it's creating or removing a post, we load it and push it to the front end through the socket.

Some ideas: the Go backend could store a persistent (in memory) low water mark. Those are set when the initial GET /api/posts request comes through. So for example, it may deliver 50 posts, with IDs and timestamps like so:

O - 180
N - 179
M - 178
...
C - 121
B - 122
A - 123

It remembers that the low-water mark is 123. If an inotify event occurs for a new post with timestamp 120, it will not deliver this to the front end. The reason is that this old content is beyond the limit of what the user has in their browser. So when you follow a new user in sync app, and you get all 500 of their old posts in the folder, you won't receive these through the socket. You will receive all posts from that user that make sense in the your currently visible time period in the browser, so if this new user you're following is involved in the conversation, they should slot into place correctly.

Optimizations

Instead of unmarshaling all posts from all users into memory, we should only get the filenames, and unmarshal just the ones that we need to load. Since the timestamp is in the filename, we can sort using that instead of sorting on the created property of the json. This isn't a huge issue right now, but certainly could become one if the number of posts reaches the millions. Thankfully Go code is really efficient.