Defend against AI crawlers & bots with server log analysis.
Presented at Blackhat USA 2025, this is the official repository for:
Protecting Small Organizations in the Era of AI Bots, R. Hoetzlein, 2025
Manual: Logrip Manual ver 1
Updated Nov 2025
Build with cmake is now simpler and faster.
Libmin dependent code is directly compiled into the project (no shared or static libs).
Steps:
- Clone this logrip repo
- Clone libmin as a sibling folder
\codes ├── \logrip └── \libmin
- Run cmake or cmake-gui on the logrip cmake.
See libmin for more details.
Running logrip requires two arguments as input:
> logrip {access_log} {config_file.conf}
The access_log must be either .txt or .log
The config_file must be .conf
An example log and config file are provided.
After installation you can quickly test logrip by doing: ./run.sh
Logrip takes a historic server access log as input.
To generate these you would typically use journalctl, or others server tools that output logs.
Here are examples of how to generate logs for apache2 or ruby-on-rails.
How to generate Apache2 logs:
> cd /var/log/apache2
> ls -l -a
> zcat access.log.*.gz > apache.log
> cat access.log.1 access.log >> apache.log
Now run logrip with the apache.log input file along with the apache2.conf config file.
How to generate Ruby-on-Rails logs:
> journalctl | grep {project} | grep 'Started GET' > ruby.log
Now run logrip with the ruby.log input file along with the ruby.conf config file.
A config file (.conf) controls the log parsing and the policy settings for blocking.
One would typically use, copy or modify an existing .conf file provided in /assets and then customize.
The full list of config settingscan be found in the manual here:
Manual: Logrip Manual ver 1.0
This is the complete example used in the video example at Blackhat 2025:
git clone https://github.com/quantasci/libmin
git clone https://github.com/quantasci/logrip
cd libmin
./build.sh
cd ..
cd logrip
./build.sh
cat assets/example_log.txt
cat assets/ruby.conf
../build/logrip/logrip example_log.txt ruby.conf
ls
xdg-open out_fig1.orig.png
xdg-open out_fig2_blocked.png
xdg-open out_fig3_filtered.png
cat out_ips.csv
Copyright (c) Quanta Sciences, 2024-2025
Rama Karl Hoetzlein, https://ramakarl.com
Apache 2.0 License
https://www.apache.org/licenses/LICENSE-2.0.txt