Cash registers at a festival that hold the wristband balances themselves. There is no server on site and none in the
cloud. How the ledger behaves normally, then two experiments, with the raw data for every run in results/ and scripts that recompute the numbers from it.
- Registers die in the middle of the rush: do the others keep selling, and is any money spent twice?
- The network splits in two: can both halves keep selling without spending the same money twice?
Every device keeps the whole ledger; no device is in charge, and none waits for an elected one.
python3 verify_normal.py recomputes this section from results/normal_*.
Throughput, three continents. One $5 server each in Frankfurt, Tokyo and Los Angeles (1 vCPU at 2.0 GHz), each also running the load generator that sends to its own device. Load goes up in 30-second steps:
| sent, all three together | records held per second, on every server | errors |
|---|---|---|
| 3,600 / s | 3,597–3,599 | 0 |
| 7,200 / s | 7,185 | 0 |
| 10,800 / s | 10,772–10,773 | 0 |
| 14,400 / s | 13,245–13,483 | Frankfurt refused 18 % of its own input (CPU 97 %) |
So about 10,800 records per second, held identically by all three, is the limit on this hardware. One run.
How long until a decision is final. A decision counts as final once every device holds it. How long that takes depends on how often the devices seal (a setting), not on having no leader. On one machine, three devices:
| seal every | median until final | 95th percentile | devices that decided differently |
|---|---|---|---|
| 250 ms | 0.48 s | 0.54 s | 0 of 116 |
| 100 ms | 0.18 s | 0.24 s | 0 of 263 |
| 50 ms | 0.10 s | 0.13 s | 0 of 385 |
| 20 ms | 0.05 s | 0.07 s | 0 of 548 |
This is a single machine. A real LAN adds its own delay and loss, and that has not been measured. The festival runs below seal every 250 ms.
The live three-continent network is configured to seal every 10 s. There a record is in the local ledger in
milliseconds but final after a median of ~15 s (p99 ~20 s). Under 7,200 records/s, Frankfurt's ledger fell 5.6 s
behind (FRA_7200tx.csv): a status script on that server was reading 216 MB of logs every 5 minutes on its single
core. Tokyo and Los Angeles stayed at 6 ms.
1,000 wristbands loaded with 30 EUR, a 40-second rush with 64 customers buying at once, and 50 "attacks" where the same wristband buys at two registers at the same instant. Halfway through, 3 registers are switched off. One machine.
| registers | purchases decided per second | median answer | registers disagree | wristbands below zero | attack approved twice | errors at live registers |
|---|---|---|---|---|---|---|
| 8 (3 switched off) | 94.8 | 0.51 s | 0 of 3,418 | 0 | 0 of 50 | 0 of 3,790 |
| 16 (3 switched off) | 107.1 | 0.52 s | 0 | 0 | 0 | 0 of 4,272 |
| 32 (3 switched off) | 56.9 | 1.04 s | 0 | 0 | 0 | 0 of 2,274 |
At 32 registers a single laptop runs 64 processes, so the rate measures the laptop as much as the registers. In these runs a purchase is decided only once every reachable register holds the same ledger. That is why an answer takes half a second, and why it is safe when registers die. When the network splits and both halves stay alive, the same rule lets both halves approve the same money (part 2, first row).
python3 verify_festival.py results/festival_8 8 5,6,7 recomputes the 8-register row from every register's decisions.
The exact moment the registers were switched off was not kept, so how fast the survivors went on (first answer
after 0.43 s) is not recomputed. For 16 and 32 registers only the customers' requests, the attacks and the final
status were kept, not each register's decisions.
We cut the network between four registers in two for 23 seconds while customers keep buying, and count what happens to the money.
Result: both halves keep selling (about 9 in 10 purchases approved), no wristband is ever charged past its balance,
and no money is lost or created. verify.py recomputes every number below from the raw data.
Same harness, same four registers (Frankfurt ×2 | Tokyo + Los Angeles), same 23-second cut, 40 "attacks". In each attack the same 30 EUR wristband tries to spend 20 EUR on both sides at the same instant.
| sold during the cut | attacks approved on both sides | wristbands below zero | median approval time | |
|---|---|---|---|---|
| sell independently (no coordination) | both sides | 40 / 40 | 69 | ~1–2 s |
| wait for a majority (quorum) | neither side | 0 | 0 | — (refused) |
| shares (this repo) | both sides: 335 / 377 and 358 / 398 | 0 | 0 | 0.16–0.21 s |
The first two rows come from earlier runs of the same harness on the same three servers (29–30 September 2026); their raw data is not in this repo. Approval times are not directly comparable: in those runs a purchase waited for the other registers to confirm, and in this one it does not.
Every wristband's balance is split into shares, one per register.
- A top-up made anywhere is split evenly across the registers.
- A register approves a purchase only if its own share covers it. It does not ask anyone whether it is allowed.
- If its share is short, it asks the registers it can reach to hand over part of theirs. A giver subtracts from its own share before sending, so the same euro is never in two shares.
- Registers on the other side of a cut simply do not answer, and the purchase is declined if what is reachable is not enough.
This is always on, not only during a split. There is no moment where the system switches mode, and so no window in which two registers both think they hold the same money.
Every decision is written to the register's own journal, and fsynced, before it leaves the register. A register that crashes rebuilds its share from that journal plus what the others tell it. A decision that was journaled is never undone: a timeout does not mean "refused", and undoing a hand-over that did happen would create money.
The idea is old (escrow). What is here is a measurement of it on registers that share no server.
| run | where | cut | sold during the cut | both-side attacks | below zero | money lost or created | every register knows every fact |
|---|---|---|---|---|---|---|---|
W1_wan_2x2 |
Frankfurt, Tokyo, Los Angeles | 2 + 2 | 335 / 377 · 358 / 398 | 0 | 0 | 0.00 EUR | yes (3,143 ×4) |
E4 |
one machine | 2 + 2 | 929 / 1,136 · 938 / 1,119 | 0 | 0 | 0.00 EUR | yes (8,417 ×4) |
E6_crash |
one machine | 2 + 2, register 2 killed with kill -9 5 s into the cut, restarted 7 s later |
926 / 1,132 · 816 / 1,118 | 0 | 0 | 0.00 EUR | yes (8,168 ×4) |
E7_3plus1 |
one machine | 3 + 1 | 1,105 / 1,119 · the lone register: 358 / 1,121 | 0 (exactly one approved, 40 / 40) | 0 | 0.00 EUR | yes (8,554 ×4) |
"Money lost or created" means: total loaded minus total approved, compared with what the shares hold at the end, to the
cent. After E6_crash's restart, register 2 rebuilt its share from 474 journal entries and approved 383 more purchases
with 0 errors. The only errors in any run (180) are requests sent to register 2 while it was dead.
- During a cut, some purchases are declined because the customer's money is on the other side. That was 82 of 775 purchases across three continents, and about 14 % on one machine. Before and after the cut: 0.
- A purchase larger than what one side holds cannot happen during a cut. With a 30 EUR wristband split in two, each side holds 15 EUR, so all 40 attempts to spend 20 EUR were declined on both sides. This follows directly from never spending the same money twice: if each side could spend more than half, both could.
- A register that is alone (3 + 1) sells only its own quarter.
- One run per setting. Four registers. No more than four has been measured; each short register asks every other one.
- In
E4,E6andE7the ordinary wristbands hold 200 EUR so that the money does not run out before the cut. Registers answer in ~1 ms, so the same simulated customers buy about 5× more often than they did with confirmation. The attack wristbands hold 30 EUR in every run. InW1, money was not exhausted. - A crash of the register's disk is not covered: the journal lives on it.
- Big festivals often keep the balance on the wristband's chip, which already prevents double spending offline. This matters for systems where the balance lives in an account: card-on-file wristbands, online top-ups, local currencies, and setups that run one central server today.
- The ledger engine the registers run on is not in this repository.
python3 verify.py results/W1_wan_2x2
Each run directory holds the customers' requests and answers (requests_A.csv, requests_B.csv), every register's
journal (opsK.txt), its ledger (ledger_nodeK.txt), its final status, and the cut and heal times. Python 3, standard
library only.
CC BY 4.0 (LICENSE): use, share and adapt freely, with attribution to Vasilis Nasopoulos.