Update on recent site issues

We've had some issues with site availability and odd bugs like some people not being able to vote or being signed out unexpectedly. This is an update on what we know so far.

Summary

• We've been under a DDoS wave (whether targeted is unknown) for two weeks. Since it started, we've been in a loop of monitor, mitigate, see what new problems emerge.
• If you encounter problems, try logging out, deleting the codidact_acct cookie, and logging back in. This works for most people.
• Some of our mitigations have exposed cases where we, previously very infrequently, failed silently in various ways. We've improved error reporting in those cases, so sometimes you'll now see messages containing server response codes where previously things "just didn't work". We have changes almost ready to go to (we hope) fix most of these.
• Update 2025-09-19: We deployed changes on September 15 that helped with some issues, and are currently testing changes in how caching works. We hope to deploy those changes soon, and we hope that this will fix all remaining problems.
• Update 2025-09-20: We deployed those changes today and they seem to be making a difference. If you are still seeing problems, please report them on Meta.

Details

Around August 22 and then intensifying starting August 27, we began seeing a new pattern of service degradation. While we're used to blips where the site is slow for a few minutes (caused by bots and scrapers), we were now seeing sustained loads affecting us for hours at a time. Even though Cloudflare blocked the vast majority of unwanted traffic, enough was getting through to cause us problems.

We began to more aggressively cache assets (like images), so that repeated requests would be served by Cloudflare instead of by us. We also noticed and blocked some specific patterns of traffic from bad actors, and we added rate limits that are mainly hit by bots but can be hit by humans. If you've seen occasional "checking if you're a human" messages from Cloudflare recently, that's why. These rate limits helped some but did not solve the problem.

While we were reviewing our logs, we looked for pages that were getting hit a lot (including legitimate traffic), and reviewed the code for those pages. We found and fixed some performance issues, including the causes of some intermittent server (500) errors. We deployed the first batch of those code changes on September 1.

Then we started getting reports that some people couldn't use some features, like voting, but other people, or the same people on different devices, didn't see these problems. We found some issues with our use of CSRF tokens and mitigated. We advised affected people to log out, remove the codidact_acct (session) cookie, and log back in again, which fixed it for most people but not everybody.

While investigating that, we found that for people using 2FA, the session cookie is being flushed when the browser session ends. If you start your browser and restore your sessions and tabs you won't see the problem, but if you start fresh, you'll be logged out. We're not yet sure what's causing this (possibly something in a dependent library).

We deployed some improvements to both error-handling and error-reporting on September 9, and we have more fixes awaiting deployment.

Meanwhile, we're seeing some severe but short-lived load spikes a few times a day now -- not the sustained heavy loads that started a couple weeks ago, but we're not out of the woods yet either. We continue to investigate problems and improve code robustness in parallel. We're a small open-source project; if anyone reading this has relevant skills and wants to help, we're happy to have you.