Paper title: Evaluating Google's Protected Audience Protocol

Description

This repository contains code for producing our simulation results and corresponding graphs (i.e. Figure 2, Table 1, and Figure 3).

Basic Requirements

Set up the environment

git clone https://github.com/Elena6918/PrAu-Simulation
python generate_artifacts.py

By running generate_artifacts.py, it will create a python virtual environment and run all three scripts (described in Experiments section) to produce Figure 2, 3 and Table 1 in the paper. Should take less a minute to run.

Experiments

If you want to run different scripts seperately, install all required packages by running pip install -r requirements.txt and continue reading the following section.

Experiment 1: Figure 2

python scenario_2_accuracy.py

The expected result is a pdf named "Accuracy", which match the Figure 2 of the paper. Should take less than a minute to run.

Experiment 2: Figure 3

python plot_fpr.py

The expected result is a pdf named "FPR_n20_top_k_extended.pdf", which match the Figure 3 of the paper. Should take less than a minute to run.

Experiment 3: Table

python simulate_colluder_num.py

The expected results should be a few lines printed to the console, showing the mean and variance of five simulations. Should take less than a minute to run.

Limitations

The code above only reproduce the graphs based on our provided simulation result data. The folder num_colluders contains the result of five simulations to obtain Number of Accusations. This data is generated by repeatedly running the smallest_number() function in simulate_colluder_num.py with adjusted parameters. Similarly, data to generate Figure 3 is provided in folder metrics_e1 and metrics_e10, where the dictionary key is the pool size, the first number represents the number of colluders, and the list contains the number of [true positives, true negatives, false positives, false negatives]. These metrics data is generated by running simulation_metrics.py repeatedly with different parameters. Since the simulation result includes randomness, every time the simulation code is ran, the result should be slightly different. Therefore, we record our simulation result and provide the code to generate graphs based on them, as they are the exact ones we present in the paper.

Re-run Simulations from Scratch

To run your own simulation data, run the simulation_metrics.py with customized parameters on command line. For example,

source venv/bin/activate
python simulation_metrics.py --epsilon 1 --num_accusations 1 --metrics_path "metrics_e1"

generates the metrics_e1_0.001K_accusations.json in the directory "metrics_e1". It takes around 6 hours to complete running this script. The default is to run 5 simulations with the same parameters, you may adjust the num_simulation variable according to your need. run_simulations.sh provides the parameters we used to generate simulations in the paper (running this script takes roughly two weeks).

Citation

For citation, please use

@inproceedings{long2024PrAu,
      title={Evaluating Google's Protected Audience Protocol},
      author={Long, Minjun and Evans, David},
      booktitle={Proceedings on Privacy Enhancing Technologies Symposium (PETS)},
      year={2024},
      month={july},
}