Spotlight on: OpenEdition rolls out Anubis across its platforms

In response to an increasing number of automated requests, OpenEdition, like several of its partners, has rolled out the open-source software program Anubis across all its platforms. OpenEdition System Administration manager Bruno Cénou talks about the data protection issues affecting an open science infrastructure like OpenEdition.

• Anubis was rolled out following a cyber attack on the Calenda platform. What kind of attack was it? Is Calenda more exposed to this sort of attack than OpenEdition’s other platforms?

Using attack in the singular is an understatement for the wave of simultaneous requests sent from tens of thousands of different IP addresses, which ended up flooding our resources. The incident qualified as a DDoS (distributed denial of service attack). However, we weren’t able to find out why it took place or who was behind it.
Calenda is more vulnerable to this type of issue as its search engine offers a wealth of possible filter combinations, resulting in tens of thousands of URLs for bots to crawl through.

• Was it the first time this kind of attack took place? What are the risks associated with such attacks?

Over the last two years, we have regularly been targeted by similar attacks, but the intensity and frequency have significantly increased over the last six months.
The main risk is that platforms become unavailable because our system resources are saturated.

• What was the strategy applied previously to counter the rising number of illegitimate automated requests while ensuring continued open access to content?

Until recently, our stance had been not to regard any automated request as illegitimate. All our open access content was available to any type of device. The different cache levels and the gradual scaling up of our system resources allowed us to handle this type of usage by expanding at a relatively reasonable pace.
However, over the last few months, we have come up against a form of extractivism that completely ignores the viability of the resources exploited.
The difficulty lies in the fact that most of the abusive automated requests we deal with are browser requests – with random user agents selected from those most commonly used –  sent from tens of thousands of different IP addresses. It is therefore almost impossible to differentiate between this type of traffic and human user traffic.
Before all of that, we used to be able to identify abusive IP ranges impersonating human traffic and trace them back to companies such as copyright trolls, which we would then blacklist.
Also, there was once an understanding that bots were supposed to identify themselves as such, via their user agent, and take account of robots.txt files, but that no longer holds at all.
The bottom line is that we now have to deal with two main categories of abusive requests:

1. Requests disguised as human searches, which we think are probably coming from scrapers installed on infected personal computers, with IPs generally linked to home internet service providers.
2. Requests from AI scrapers, for training purposes, which identify themselves via their user agent and for which only part of the IP is available.

• What are the advantages of rolling out Anubis?

Anubis is rolled out as a reverse proxy on our platforms. It reacts to requests by submitting a computational puzzle to the client browser’s JavaScript engine. The puzzle takes a few seconds for regular browsers to solve and, if all goes well, the user barely notices the process. However, a bot that can’t execute JavaScript or doesn’t manage cookies will not pass the test and be prevented from accessing site content. This simple test denies entry to over 90% of automated requests disguised as human activity – it’s simple and effective.
As for long-standing indexing engines and trustworthy partners, they don’t have to pass the challenge. Filtering rules are clear and fully configurable to meet our needs.
The benefits are immediately apparent as the load imposed on our platforms has been reduced threefold.
We owe a big thank you to Anubis developer Xe Iaso, who generously explains how Anubis works.

• Is it possible to tell the difference between requests from malicious bots and those from legitimate bots used to analyse corpora automatically on our platforms?

I would call them pesky bots, as we don’t know the intentions and interests that lie behind them.
We can, of course, identify bots belonging to partners or indexing engines thanks to their IP addresses and/or user agents, and it’s something we already do.
There’s no denying that by setting up Anubis we’re potentially refusing entry to numerous legitimate bots we don’t know about, but for now, we deal with each case as and when.

Contact information

If you’re having difficulty accessing our platforms because of Anubis, please let the OpenEdition Team know: contact[at]openedition[dot]org.

Good news

You’ll soon be able to customize the page generated by Anubis when you connect to platform websites. We’re working on it right now!

---

The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.

---

---