python-tally-token

What is this?

tally-token is a Python library for split data into tokens with same length.

Tally is a historical object for prove something by splitting wood into tokens and matching tokens.

Medieval English split tally stick (front and reverse view). The stick is notched and inscribed to record a debt owed to the rural dean of Preston Candover, Hampshire, of a tithe of 20d each on 32 sheep, amounting to a total sum of £2 13s. 4d.

Changelog

See CHANGELOG.md for release notes and upgrade notes.

Usage

Install

$ pip install tally-token

CLI Usage

$ tally-token --help
usage: tally-token [-h] {split,merge} ...

positional arguments:
  {split,merge}  Commands: split: split a file into multiple files merge: merge multiple files into a fileExample: tally-token split example.bin
                 example.bin.1 example.bin.2 example.bin.3 tally-token merge example.bin.1 example.bin.2 example.bin.3 --output example-merged.bin

options:
  -h, --help     show this help message and exit

split

You can use split to split a file into multiple files.

$ tally-token split something.bin split-1.bin split-2.bin split-3.bin

merge

You can use merge to merge multiple files into a file.

$ tally-token merge split-1.bin split-2.bin split-3.bin --output merged.bin

Large files

Nothing special. You can split and merge large file.

$ dd if=/dev/urandom of=original.1g.bin bs=1G count=1
$ tally-token split original.1g.bin split-1.bin split-2.bin split-3.bin
$ shasum -a 256 original.1g.bin
> 736a344d99d27e2dcdab8bc37ca94c83eda26f812a3dee87ac98989f89b3f965 original.1g.bin
$ tally-token merge split-1.bin split-2.bin split-3.bin --output recovery.1g.bin
$ shasum -a 256 recovery.1g.bin
> 736a344d99d27e2dcdab8bc37ca94c83eda26f812a3dee87ac98989f89b3f965 recovery.1g.bin

Example

Split and merge text

from tally_token import split_text, merge_text

tokens = split_text("Hello, World!", into=5)
assert merge_text(tokens) == "Hello, World!"

Tokens are random opaque bytes. For a custom encoding, pass the same encoding to both operations; the tokens do not record it:

tokens = split_text("こんにちは", encoding="CP932")
assert merge_text(tokens, encoding="CP932") == "こんにちは"

Bytes interface

from tally_token import split_bytes_into, merge_bytes_into

tokens = split_bytes_into(b"Hello, World!", into=5)
assert merge_bytes_into(tokens) == b"Hello, World!"

Migrating tokens created with 0.6.3 or earlier

The token format changed: new tokens contain a shared 16-byte session ID. Existing raw tokens cannot be merged by the new version. Keep every old token and recover the original using version 0.6.3 before upgrading. Run these commands outside any tally-token source checkout, for example:

python -m venv /path/to/legacy-tally-env
/path/to/legacy-tally-env/bin/python -m pip install tally-token==0.6.3
/path/to/legacy-tally-env/bin/python -m tally_token merge recovered.bin old-1 old-2

Verify the recovered data, then use the new version to split it again. Protect and securely remove the recovered plaintext according to your threat model. Retain the original tokens until the replacement tokens have been verified. The split_bytes_into keyword argument formerly named n is now into.

Security and Logging

Security model

This library implements an n-of-n XOR split. All tokens produced by the same split operation are required to recover the original bytes. It is not a k-of-n threshold scheme such as Shamir's Secret Sharing, so any missing token prevents recovery.

Tokens include a shared session ID so merge_bytes_into and merge_io can detect different session IDs. Different splits can coincidentally have matching IDs, so this does not guarantee that mixed tokens are rejected. The check is not a message authentication or tamper-detection mechanism: if a token from the correct session is modified, merging can still return modified bytes without raising an error.

Calling split_bytes_into(secret, 1) or split_text(secret, into=1) does not protect confidentiality. The single returned token contains the original secret payload after the session ID, so use at least two tokens when the goal is to split secret material.

Report vulnerabilities through the private channel described in SECURITY.md.

Logging policy

This library handles secret material (token bytes) and follows a log-nothing policy: no secret data should appear in log output.

• The library installs a NullHandler on its root logger so log records are silently discarded unless the application configures a handler. This is the standard practice for libraries.
• Do not log token bytes or plain-text secrets in your application code. Functions such as split_bytes_into, _split1, write_split_tokens, and merge_bytes_into operate directly on secret material.
• If you add debug logging to code that calls this library, ensure no token content is captured in log records or exception messages.

Reference

• Tally stick https://en.wikipedia.org/wiki/Tally_stick
• Tessera or Symbolum (Hospitium token) https://en.wikipedia.org/wiki/Hospitium
• 割符 https://ja.wikipedia.org/wiki/%E5%89%B2%E7%AC%A6
• One-time pad https://en.wikipedia.org/wiki/One-time_pad

Development

This repository uses lefthook to run the same checks as CI locally, so problems surface before they reach CI.

# Install dependencies
uv sync

# Install the Git hooks (once; requires lefthook on your PATH)
lefthook install

Once installed, the hooks run automatically:

• pre-commit: uv run poe check
• pre-push: uv run poe check and uv run poe test

You can also run the checks manually:

uv run poe check
uv run poe test

CI still runs the full matrix (see .github/workflows/); the hooks only bring that feedback earlier on your machine.

LICENSE

BSD 3-Clause License