tape-dataset-opendal exports standard Bub tapes to a backend-agnostic dataset layout through Apache OpenDAL.
The package is intentionally narrow:
- source side: any standard
TapeStoreorAsyncTapeStore - sink side: any OpenDAL backend
- export shape: tape-native records plus tape-level and anchor-derived views
- Bub integration: a thin CLI layer over the same exporter
It does not redefine tape semantics or impose an application-specific session schema.
uv pip install "git+https://github.com/bubbuild/bub-contrib.git#subdirectory=packages/tape-dataset-opendal"You can also install it with Bub:
bub install tape-dataset-opendal@mainUse this package when you need one of these outcomes:
- export the active Bub tape store through a standard CLI path
- export any standard tape store from Python without tying the export to a specific backend
- shape exports with CEL before publishing or downstream consumption
- stage a dataset locally for scanning and review before uploading it elsewhere
This package keeps three boundaries explicit:
- source boundary: the exporter reads only through the standard tape store interface
- transport boundary: the exporter writes only through OpenDAL
- data boundary: the output remains tape-native, with derived summaries and anchor slices layered on top
That means:
- SQLite, SQLAlchemy, Redis, or another tape-store implementation can be the source as long as it follows the standard interface
- local filesystem, S3, MinIO, GCS, AzBlob, and other OpenDAL targets are all just sinks
- the export format is not constrained by another application-specific session schema
The package exposes one Bub plugin entry point:
- plugin entry point:
tape-dataset-opendal - command:
bub tape-export
The command acquires the active tape store inside BubFramework.running() and releases lifecycle-managed resources after export.
Minimal example:
bub tape-export \
--scheme fs \
--config root=/tmp/tape-dataset \
--root snapshot-2026-04-23S3-compatible example:
bub tape-export \
--scheme s3 \
--config bucket=my-bucket \
--config endpoint=http://127.0.0.1:9000 \
--config region=us-east-1 \
--config access_key_id=minioadmin \
--config secret_access_key=minioadmin \
--config enable_virtual_host_style=false \
--root snapshotUseful flags:
--filter: repeatable CEL expression--filter-file: file containing one CEL expression per line--no-segments: skipsegments.jsonl--no-raw: skipraw/*.jsonl
from opendal import Operator
from bub.core.tape import InMemoryTapeStore, TapeEntry
from tape_dataset_opendal import ExportableTapeStore
inner = InMemoryTapeStore()
inner.append("ops__incident", TapeEntry.anchor("triage"))
inner.append("ops__incident", TapeEntry.message({"role": "user", "content": "DB timeout"}))
store = ExportableTapeStore(inner)
report = store.export_dataset(Operator("fs", root="/tmp/tape-dataset"))
print(report.entry_count)
print(report.manifest_path)import asyncio
import opendal
from bub.core.tape import AsyncTapeStoreAdapter, InMemoryTapeStore, TapeEntry
from tape_dataset_opendal import AsyncExportableTapeStore
async def main() -> None:
inner = AsyncTapeStoreAdapter(InMemoryTapeStore())
store = AsyncExportableTapeStore(inner)
await store.append("agent__session", TapeEntry.anchor("task"))
await store.append("agent__session", TapeEntry.message({"role": "assistant", "content": "Done"}))
report = await store.export_dataset_async(
opendal.AsyncOperator("fs", root="/tmp/tape-dataset"),
)
print(report.segment_count)
asyncio.run(main())Filtering uses cel-python. The package does not define a custom filter DSL.
bub tape-export \
--scheme fs \
--config root=/tmp/tape-dataset \
--root filtered \
--filter 'kind == "message"' \
--filter 'payload.role == "user" || text.contains("timeout")'When the filter set is large:
bub tape-export \
--scheme fs \
--config root=/tmp/tape-dataset \
--root filtered \
--filter-file ./filters.celFilter files use one expression per line. Lines beginning with # are ignored.
Available CEL variables per entry:
tapekinddatepayloadmetatextjsonentry
Multiple filters are combined with logical AND.
Each export writes:
manifest.json: export metadata, counts, paths, and active filterstapes.jsonl: one summary record per tapeentries.jsonl: one normalized record per retained entrysegments.jsonl: anchor-derived slices when segment export is enabledraw/<encoded-tape>.jsonl: canonical tape-native JSONL per tape when raw export is enabled
Segment rules follow standard tape semantics:
- if a tape contains anchors, each segment starts at one anchor and ends before the next anchor
- if a tape contains no anchors, the whole tape is exported as one
full_tapesegment
For transport or integration testing, direct export to remote object storage is enough.
For public-sharing workflows, use this order:
- export to local filesystem staging
- apply CEL filters or manual edits
- run TruffleHog on the staged output
- run a final LLM review on the staged dataset
- publish only after those checks pass
The skill at src/skills/tape-dataset-share-review/SKILL.md exists to guide that workflow.
The package includes a focused verification recipe for the SQLite through SQLAlchemy to MinIO path:
That guide verifies two things together:
- tapes are generated through the real Bub runtime before export
- a real
bub-tapestore-sqlalchemySQLite source can be exported through the Bub CLI path to an S3-compatible backend - one tapestore containing multiple tapes is exported as multiple dataset members
Apache-2.0
The release-review workflow in this package was informed in part by the ideas in
pi-share-hf.