PocketBook Dictionary Toolkit (pbdt)

This toolkit allows you to create XDXF dictionaries from pre-processed Kaikki.org Wiktionary data (in JSONL format), and converts dictionaries between XDXF, StarDict, and PocketBook .dic formats.

It includes a powerful command-line tool pbdt as well as a browser-based WASM UI.

Usage

Command-Line Interface (pbdt)

Building from source

To build the tool from source, you need Go installed. Run the following command:

go build -o pbdt ./main.go

Running the toolkit

Run ./pbdt --help to see the available commands and flags:

• create: Create a Pocketbook dictionary from Kaikki.org JSONL
• convert: Convert a dictionary between XDXF, StarDict and Pocketbook format
• merge: Merge multiple dictionaries into one
• extract-meta: Extract metadata from a Pocketbook dictionary
• lookup: Lookup a word in a Pocketbook or StarDict dictionary
• show: Show dictionary contents

Usage Examples

Create dictionary from Kaikki JSONL

# XDXF
./pbdt create --from=en --to=en -i raw-wiktextract-data.jsonl -o output.xdxf -t examples/entry.tpl --name "EN -> EN"

# DIC
./pbdt create --from=en --to=en -i raw-wiktextract-data.jsonl -o output.dic -t examples/entry.tpl --name "EN -> EN" --meta-dir language_folder/

# With custom merge method (useful for separating translations and definitions)
./pbdt create --from=es --to=es -i raw-wiktextract-data.jsonl -o output.xdxf -t examples/translations.tpl --merge-method first

Merge method flags

The create command supports the following flags for controlling how duplicate words are merged:

• --merge-method: How to handle duplicate words. Options are:
  • join: Concatenate definitions from multiple sources (default)
  • first: Keep only the first definition encountered
• --merge-sep: Separator to use when merge-method=join. Default is <br/> (HTML line break)

Note on --meta-dir: To create PocketBook .dic files, you need language configuration files (collate.txt, keyboard.txt, morphems.txt). You can find them for various languages in the LanguageFilesPocketbookConverter repository.

Convert between formats

# XDXF to DIC
./pbdt convert input.xdxf language_folder/
# or
./pbdt convert input.xdxf output.dic --meta-dir language_folder/

# DIC to XDXF
./pbdt convert input.dic output.xdxf

# XDXF to StarDict
./pbdt convert input.xdxf output-stardict/

Merge dictionaries

# Merge two XDXF files with 'join' method
./pbdt merge source1.xdxf source2.xdxf output.xdxf --method join

# Merge with 'first' method (keeps first definition, ignores duplicates)
./pbdt merge source1.xdxf source2.xdxf output.xdxf --method first

# Merge to DIC format with custom separator
./pbdt merge source1.xdxf source2.xdxf output.dic --method join --separator "<br>" --meta-dir language_folder/

Merge command flags:

• --method: How to handle duplicate words. Options are first (keep first definition only) or join (concatenate definitions, default)
• --separator: String to insert between merged definitions when using --method join. Use \n for newline (default), or HTML tags like <br> for visual separation
• --meta-dir: Required for .dic output; directory containing language metadata files

Format notes

The toolkit is HTML-native: definitions are stored and operated on as HTML fragments. The .dic (PocketBook) and StarDict formats are transparent containers — they store the HTML verbatim. XDXF is different: it is XML-based, so there is a HTML ↔ XML normalisation step on every read and write. This produces a few gotchas worth knowing if you use XDXF for anything beyond quick visual inspection.

Data changed when writing XDXF (any format → XDXF)

HTML-only entities are decoded to literal UTF-8. XML defines only five built-in entities (&, <, >, ", '). All other entities are HTML-specific and are not valid XML. When writing XDXF, pbdt decodes them to their Unicode equivalents:

In HTML definition	In XDXF output
 	literal U+00A0 (non-breaking space)
©	©
—	—
<, >, &	<, >, & (unchanged)

The semantic meaning is preserved (both represent the same character), but the representation changes. After one round-trip through XDXF, these entities are permanent UTF-8 literals in subsequent files.

Note on reading source XDXF files containing   etc.: if an XDXF file was hand-edited and contains HTML-only entities like   (technically invalid XML), pbdt's XDXF reader treats them as opaque text and passes them through intact rather than failing. This means XDXF → DIC preserves   as  , but the next DIC → XDXF pass will convert it to a literal non-breaking space as described above.

<br> normalises to <br/> and back. XML requires self-closing syntax for void elements. pbdt converts <br> → <br/> when writing XDXF and back to <br> when reading. No data is lost.

Round-trip summary

Path	Attributes	HTML entities (  etc.)
DIC → XDXF → DIC	preserved	converted to literal UTF-8
StarDict → XDXF → StarDict	preserved	converted to literal UTF-8
DIC ↔ StarDict (direct)	preserved	preserved

A direct DIC ↔ StarDict conversion is fully lossless — definitions are stored as raw bytes in both formats without any transformation.

StarDict sametypesequence

pbdt writes StarDict files with sametypesequence=h (HTML). Third-party StarDict readers (e.g. GoldenDict, Stardict) that support this type will render definitions as HTML.

PocketBook literal newlines

Literal newlines are removed during DIC conversion. The PocketBook .dic reader interprets actual newline characters (0x0A) in definition text as line breaks and renders them as <br/> tags. To preserve the visual structure as intended, pbdt's DIC writer removes literal newlines and relies on HTML break tags (<br/>) instead. This means definitions should use <br/> or <br> tags for line breaks rather than actual newlines.

---

Downloading the data

You can download the raw Wiktionary data from the Kaikki.org website. You will need to download the "raw data file" for the language you are interested in. For Spanish, this is https://kaikki.org/eswiktionary/raw-wiktextract-data.jsonl.gz (decompress with gunzip).

This will give you a raw-wiktextract-data.jsonl file, which you can then use as input for this tool.

Advanced: Two-Template Pipeline for Multi-Sense Entries

When a word has multiple parts of speech (e.g., "vestido" as both noun and adjective), you may want translations to appear only once at the top, followed by all definitions grouped by part of speech. This can be achieved using a two-template pipeline:

Step 1: Create translations-only dictionary

Use a template that outputs only translations (e.g., examples/translations.tpl):

./pbdt create -t examples/translations.tpl -o translations.xdxf --merge-method first

The --merge-method first flag ensures each word keeps only its first translation entry.

Step 2: Create definitions-only dictionary

Use a template that outputs only definitions without translations (e.g., examples/es.tpl):

./pbdt create -t examples/es.tpl -o definitions.xdxf

Step 3: Merge with a visual separator

Merge both dictionaries, placing translations first:

./pbdt merge --method join --separator "<br>" translations.xdxf definitions.xdxf final.xdxf

The result will be a dictionary where translations appear once at the top, followed by all definitions:

<ar><k>vestido</k><def>
  <div><b>EN:</b> clothes • dress</div>
  <br>
  <ul>
    <li><i>sust.</i> – La ropa, lo que se usa para vestirse...</li>
    <li><i>sust.</i> – Prenda de vestir femenina...</li>
  </ul>
  <ul>
    <li><i>adj.</i> – Que lleva ropas...</li>
  </ul>
</def></ar>

WASM UI

The toolkit also features a browser-based UI for converting dictionaries without a command line.

Building and Running

You need tinygo and python3 installed.

cd wasm
make build
make serve

Then visit http://localhost:8088 in your browser.

Customizing the template

You can customize the output of the dictionary by creating your own template file for Kaikki JSONL creation. The template is a standard Go template. You can use the provided examples/entry.tpl as a starting point.

The following data is available in the template:

• .Word: The word for the dictionary entry.
• .Entries: A slice of kaikki.Entry structs that match the word.

The kaikki.Entry struct has the following fields:

• Word: The word itself.
• Pos: The part of speech (e.g., "noun", "verb"), localized based on target language.
• RawPos: The original part of speech before localization (e.g., "noun", "verb").
• Senses: A slice of kaikki.Sense structs, each representing a different sense of the word.
• LangCode: The language code of the word (e.g., "es").
• Tags: A slice of tags for the entry (form/gender tags are not included in senses).
• Translations: A slice of translations.
• Synonyms: A slice of synonyms.
• Antonyms: A slice of antonyms.
• Gender: The grammatical gender extracted from the first sense's tags (e.g., "m", "f", "n", "c").
• Plural: The plural form extracted from the forms array (first occurrence with "plural" tag).
• Forms: A slice of kaikki.Form structs representing word forms (plural, gender variants, conjugations).
• EtymologyText: Etymology information for the word.

The kaikki.Sense struct has the following fields:

• Glosses: A slice of strings, each being a definition of the sense.
• Tags: A slice of tags for the sense. Form/gender tags (feminine, masculine, plural, etc.) are automatically trimmed from senses, keeping only usage/register tags.
• Synonyms: A slice of synonyms for the sense.
• Antonyms: A slice of antonyms for the sense.

The kaikki.Form struct has the following fields:

• Form: The word form string (e.g., "aguas" for plural of "agua").
• Tags: A slice of tags for this form (e.g., "plural", "feminine").
• ShortTag: A human-readable tag generated from form tags (e.g., "f", "pl", "f pl", "m pl").

Template Functions

The following functions are available in the template:

• join: Joins a slice of strings with a separator.
• getTranslations: Takes a kaikki.Entry and a language code, and returns a slice of translations for that language.
• getSynonyms: Takes a kaikki.Entry and a kaikki.Sense and returns a slice of synonyms for that sense.
• getAntonyms: Takes a kaikki.Entry and a kaikki.Sense and returns a slice of antonyms for that sense.

Whitespace and Newlines in Go Templates

Go templates can be tricky when it comes to controlling whitespace and newlines. If you are unfamiliar with them, your best bet is to feed your template into an LLM and ask it to improve it.