LLMify Prototype

LLMify is a small local Streamlit experiment for testing whether alternative documentation representations can reduce context tokens while preserving answer quality.

Upload one document, generate a compressed context, and ask the same evaluation questions against the original and compressed versions. The app compares context size, answer size, and model-judged answer quality, then exports the results as CSV or Markdown.

What This Is / What This Is Not

This is a focused prototype for comparing structured Markdown profiles, controlled line-grammar profiles, and LLMLingua-2 prompt compression. It is useful for exploring compression tradeoffs and reviewing where operational details are lost.

It is not a production benchmark, a generic chatbot, or a full retrieval-augmented generation (RAG) system. It has no authentication, persistent storage, or document retrieval layer.

Quick Start

Requirements: Python 3.10 or newer, make, and an OpenAI API key.

git clone <repository-url> llmify-prototype
cd llmify-prototype
python -m venv .venv
source .venv/bin/activate
make install
export OPENAI_API_KEY="your_api_key"
make run

Open the local URL printed by Streamlit, choose a compression method, upload a document, and select Generate compressed context. Add questions manually or generate samples, then select Run eval. You can skip the export command and enter the API key in the masked sidebar field instead.

For a runtime-only install, use python -m pip install -r requirements.txt instead of make install.

To test LLMLingua-2, install its optional local-model dependencies before starting the app:

make install-llmlingua

The first LLMLingua run downloads the configured Hugging Face classifier model. Subsequent Streamlit reruns reuse the cached model.

Sample Documents

The repository includes sample_docs/vespergate-admission-locks.md, a long fictional operator manual for an invented technology. It is useful for testing whether compression and evaluation rely on the uploaded document instead of model prior knowledge.

Compression Methods

All methods first run deterministic cleanup that removes front matter, HTML comments, presentation-only Hugo shortcodes, figures, and embedded table markup without removing the table text.

• LLM profile (Markdown): generates compact human-reviewable Markdown through the OpenAI API.
• LLM profile (line grammar): generates typed, line-oriented records with counted positive and negative example sets.
• LLMLingua-2 prompt compression: locally prunes source tokens using Microsoft's LLMLingua-2 classifier. It produces a compressed prompt context, not a durable semantic profile. See the official LLMLingua repository.

Evaluation questions and judge reference contexts always use the untouched extracted source, so cleanup and compression omissions remain measurable.

For a reproducible, full-context comparison of all three aggressive modes on the Kubernetes and fictional VesperGate documents, follow BENCHMARK.md. The batch runner does not set application-level output limits or truncate the benchmark contexts.

Configuration

Environment variables are documented in .env.example.

Variable	Required	Default
OPENAI_API_KEY	No	Sidebar API key
LLMIFY_MODEL	No	gpt-5.5
LLMIFY_JUDGE_MODEL	No	gpt-5.5
LLMIFY_TOKEN_ENCODING	No	o200k_base
LLMIFY_LLMLINGUA_MODEL	No	microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank

An API key is required for model calls, either through the masked sidebar field or OPENAI_API_KEY. A sidebar key overrides the environment variable for the current browser session. The generation and judge models can also be changed in the sidebar and must be available to the configured API account.

Select Check model access to load the text models visible to the current API project. The generation and judge fields then become dropdowns restricted to those model IDs.

Supported Inputs

• Plain text: .txt
• Markdown: .md, .markdown
• HTML: .html, .htm
• PDF: .pdf
• Word: .docx

PDF and DOCX extraction is intentionally basic. Scanned PDFs require OCR before upload.

Privacy and Secrets

Uploaded files stay in Streamlit memory. LLM profile generation and all evaluation calls send document text to the configured OpenAI API. Deterministic cleanup and LLMLingua compression run locally. Do not upload sensitive material unless each selected operation is permitted by your data-handling requirements.

The sidebar key remains in Streamlit session memory and is not written to disk by this app. Keep OPENAI_API_KEY in your shell environment or an untracked local environment file. Never commit .env, .streamlit/secrets.toml, uploaded documents, generated profiles, or evaluation reports.

Known Limitations

• This is not full RAG. Large documents are truncated to a configurable context limit instead of being chunked and retrieved.
• The judge is another LLM, so its scores are directional rather than ground truth.
• Generated questions can favor information the generation model finds salient.
• Token counts estimate encoded text only; they do not include every provider-side token or cost.
• Extraction does not preserve every layout, image, table, or scanned page accurately.
• LLMLingua-2 can remove or fragment exact commands, identifiers, and examples; it is included as an experimental baseline.
• LLMLingua-2 requires PyTorch, Transformers, local memory, and a one-time model download.
• An API or model failure aborts the current operation; evaluation does not retry or resume partial runs.

Development

make lint
make test
python -m py_compile app.py

Tests cover pure token, parsing, normalization, and report-generation behavior and do not require an API key.

License

MIT. See LICENSE.

Roadmap

• Add semantic chunking for large documents.
• Compare retrieval over original, cleaned, and LLMified representations.
• Add gold expected facts for deterministic quality checks.
• Check answer citations and source provenance.
• Run fixed evaluation suites in CI.