OrgFlow — Communication Transcripts for Process Discovery and Workflow Automation

Overview

OrgFlow is a research pipeline that converts formal business process models into realistic multi-actor communication transcripts (emails, SMS, call transcripts), then uses those transcripts to drive two downstream tasks:

Process discovery — a ReAct LLM agent rediscovers the underlying business processes from raw transcripts and renders them as BPMN models.
Workflow automation — a code-interpreter agent converts discovered (or ground-truth) BPMN models into executable n8n workflows, deploys them, and validates execution traces.

The corpus covers 55 business processes and 243 scenarios from the PMo Dataset, each with a fully annotated JSON transcript linking every message back to a BPMN task.

Repository Structure

.
├── data/
│   ├── pmo-dataset/              # Source dataset (55 process models)
│   │   ├── descriptions/         # Natural-language process descriptions (*.txt)
│   │   ├── bpmn/                 # Ground-truth BPMN files
│   │   ├── pme/                  # Process Model Elements JSON (used by automation agent)
│   │   ├── scenarios/            # Enumerated execution paths (XX_scenarios.json × 55)
│   │   └── actors.json           # Actor classification (201 actors across 55 processes)
│   ├── transcripts/              # 243 annotated transcripts (one folder per process)
│   ├── few_shot/                 # Gold few-shot examples for generation prompt
│   ├── react_store/              # Embedded message store + evaluation artifacts
│   │   ├── message_store.jsonl   # All messages (stripped of BPMN annotations)
│   │   ├── embeddings.npy        # Sentence embeddings (all-MiniLM-L6-v2)
│   │   ├── ground_truth_labels.json
│   │   ├── name_mappings.json
│   │   ├── bpmn_matches.csv      # Embedding-similarity matches (GT vs discovered)
│   │   └── llm_judge_results.csv # LLM-judge scores for best run
│   ├── n8n_workflows/            # n8n node catalog, spec, and generated workflows
│   ├── eval/                     # Per-run discovery evaluation outputs
│   ├── archive_react_v5/         # Agent run, seed 42
│   ├── archive_react_v5b/        # Agent run, seed 1
│   ├── archive_react_v5c/        # Agent run, seed 7 (best run — used in paper)
│   ├── baseline_oneshot_flash_v2/# One-shot Gemini Flash baseline
│   ├── discovery_clustering_messages/ # K-Means baseline
│   ├── end_to_end/               # Full end-to-end pipeline run (all 5 stages)
│   └── best_discovery/           # End-to-end pipeline seeded with best agent run
├── src/
│   ├── data_generation/          # Corpus generation (Phases 0–4)
│   │   ├── build_actors.py       # Phase 0 — classify actors (human vs. system)
│   │   ├── path_enumerator.py    # Phase 1 — enumerate execution paths via DFS
│   │   ├── channel_assigner.py   # Phase 2 — assign communication channels per task
│   │   ├── prompt_builder.py     # Phase 2 — assemble LLM prompts
│   │   ├── validate.py           # Phase 2 — parse, validate, compute coverage_index
│   │   ├── generate.py           # Phase 3/4 — LLM driver; --pilot / --all
│   │   └── pilot_check.py        # Post-generation quality checks
│   ├── process_discovery/        # Process discovery pipeline
│   │   ├── prepare_store.py      # Build embedded message store from transcripts
│   │   ├── react_agent.py        # ReAct tool-using agent (Gemini / Anthropic)
│   │   ├── bpmn_generator.py     # Discovered JSON → BPMN via ProMoAI
│   │   ├── bpmn_to_pme.py        # BPMN XML → PME JSON (for automation pipeline)
│   │   ├── eval_discovery_ongoing.py # Per-process + domain-family F1 evaluation
│   │   ├── baseline_oneshot.py   # One-shot LLM baseline
│   │   ├── cluster_baseline.py   # K-Means clustering baseline
│   │   └── tools.py              # Agent tools (search, filter, save)
│   └── automation/               # BPMN → n8n workflow automation (self-contained uv project)
│       ├── agent.py              # convert_pme_to_n8n() — core conversion pipeline
│       ├── pipeline.py           # Deploy, simulate, validate, feedback loop (55 workflows)
│       ├── eval.py               # Parallel corpus eval harness
│       ├── prompts.py            # Mode-aware system/user prompts + tool definitions
│       ├── validate.py           # Pydantic schema for n8n WorkflowCreate
│       ├── code_sandbox.py       # Jupyter kernel wrapper for code_interpreter mode
│       ├── compare_bpmn.py       # BPMN structure comparison utilities
│       ├── bpmn_to_pme.py        # (copy) BPMN XML → PME JSON
│       └── ...                   # llm_client, n8n_client, config, tests, scripts
├── end_to_end.py                 # Orchestrates all 5 stages in one run
├── scripts/
│   ├── draw_pipeline.py          # Renders figures/pipeline.pdf
│   ├── llm_judge.py              # LLM-as-judge evaluation over matched pairs
│   └── match_bpmns.py            # Embedding similarity matching (GT vs discovered)
├── notes/
│   ├── Worklog.md                # Session-by-session decisions and findings
│   ├── Plan.md                   # Original high-level plan (Plan A / Plan B)
│   └── Findings.md               # Dataset selection rationale
└── figures/
    ├── pipeline.pdf              # Full pipeline diagram
    └── pipeline.png

Dataset: PMo Dataset

The PMo Dataset (data/pmo-dataset/) aggregates 55 hand-crafted, expert-validated process models drawn from four published sources. Each process is represented in 9 formats; the BPMN file is the ground truth.

Stat	Value
Total processes	55
Total scenarios (execution paths)	243
Actors per process	2–8 (avg ~3.7)
Total actors	201 (175 human, 26 system)
Processes with system actors	19

See data/pmo-dataset/README.md for full documentation and citations.

Part I — Corpus Generation

Pipeline (Phases 0–4)

Phase 0 — Actor classification
  build_actors.py  →  data/pmo-dataset/actors.json

Phase 1 — Path enumeration
  path_enumerator.py  →  data/pmo-dataset/scenarios/XX_scenarios.json
  DFS over BPMN sequence flows; XOR branches produce separate paths;
  AND/parallel branches are flattened; capped at 32 paths per process.

Phase 2 — Prompt infrastructure
  channel_assigner.py  — keyword rule engine assigns email/sms/call_transcript per task
  prompt_builder.py    — builds system prompt (rules + schema + 2 gold shots) + user prompt
  validate.py          — parses LLM JSON, checks all fields, computes coverage_index

Phase 3 — Pilot generation (15 scenarios: processes 01, 17, 21, 23, 51)
  python src/data_generation/generate.py --pilot

Phase 4 — Bulk generation (all 243 scenarios)
  python src/data_generation/generate.py --all

Channel assignment rules

Task semantic type	Channel
Formal decision, approval, rejection, submission	`email`
Notification, confirmation, acknowledgment	`email`
Automated system notification	`email` or `sms`
Discussion, negotiation, consultation, review	`call_transcript`
Quick status update, reminder, alert	`sms`

One deliberate channel override per path (rule_override) is injected for dataset diversity. System actors never appear in call_transcript threads.

Transcript format

Each transcript is at data/transcripts/XX/XX_PYYY.json:

Field	Description
`process_id`	Two-digit process number (e.g. `"01"`)
`scenario_id`	Unique scenario ID (e.g. `"01_P000"`)
`scenario`	`scenario_label`, `scenario_description`, `path_tasks`
`actors`	List of actors with `actor_id`, `name`, `actor_type`
`threads`	Communication threads with `channel`, participants, and annotated `messages`
`tasks_omitted`	Tasks handled silently by system actors
`coverage_index`	`task_coverage_ratio`, counts, `full_coverage` flag

All 243 scenarios passed automated quality checks: 100% task coverage, no system actors in calls, timestamps non-decreasing, no generation errors.

Quick start — corpus generation

# Install dependencies (main venv)
uv sync

# Set API keys
cp .env.example .env   # fill in ANTHROPIC_API_KEY (or OPENAI_API_KEY)

# Dry run (no API calls)
uv run python src/data_generation/generate.py --pilot --dry-run

# Pilot (5 processes)
uv run python src/data_generation/generate.py --pilot

# Full bulk generation
uv run python src/data_generation/generate.py --all

# Quality checks
uv run python src/data_generation/pilot_check.py
uv run python src/data_generation/pilot_check.py --process 17

Part II — Process Discovery

Given the transcripts, a ReAct LLM agent iteratively searches the message store and reconstructs business processes as BPMN models.

transcripts/  ──►  message_store      ──►  react_agent   ──►  discovered JSONs  ──►  BPMN files
               (prepare_store.py)      (react_agent.py)                           (bpmn_generator.py)

prepare_store.py strips BPMN annotations, anonymises actor names, adds timestamps, and embeds all messages. Writes to data/react_store/.
react_agent.py is a tool-using LLM agent (Gemini or Anthropic) that searches and filters the message store and saves coherent processes as JSON.
bpmn_generator.py calls ProMoAI (vendored at third_party/ProMoAI/) to convert each discovered JSON into a BPMN XML file.

Setup

Two environments are required because ProMoAI's deps (pm4py, powl, torch) conflict with Python 3.13 on macOS arm64:

# Main venv (agent + baselines + eval)
uv sync

# ProMoAI venv (bpmn_generator.py only)
python3.12 -m venv .venv-promoai
source .venv-promoai/bin/activate
pip install -r third_party/ProMoAI/requirements.txt
deactivate

Put GEMINI_API_KEY (or GOOGLE_API_KEY) and ANTHROPIC_API_KEY in .env.

Results (threshold τ=0.5, 12-domain groupings)

System	Directory	n	covGT	Lenient F1	Strict F1	Hungarian F1	Dom-G F1	Dom-H F1
K-Means (k=55)	`data/discovery_clustering_messages/`	55	20	0.537	0.364	0.364	0.719	0.519
One-shot Flash	`data/baseline_oneshot_flash_v2/`	40	31	0.745	0.653	0.716	0.950	0.895
Agent, seed 1	`data/archive_react_v5b/`	30	20	0.571	0.471	0.541	0.912	0.852
Agent, seed 42	`data/archive_react_v5/`	51	37	0.810	0.698	0.755	0.959	0.879
Agent, seed 7 (best)	`data/archive_react_v5c/`	67	42	0.865	0.689	0.754	0.953	0.814

The reported paper numbers are the median across seeds 1, 42, 7. Seed 7 (archive_react_v5c) is the best single run.

Reproducing results

1. Evaluate an existing run:

uv run python src/process_discovery/eval_discovery_ongoing.py \
    --store-dir data/archive_react_v5c \
    --families-file data/eval/react_store/gt_domains.json \
    --output-file data/archive_react_v5c/eval_results_domains.json

Swap --store-dir for any of the other systems above.

2. Generate BPMN from the best archived run:

.venv-promoai/bin/python src/process_discovery/bpmn_generator.py \
    --process-dir data/archive_react_v5c/discovered_processes \
    --output-dir  data/archive_react_v5c/discovered_processes_bpmn

3. Re-run the agent from scratch (~7 min, ~$3 on Gemini Flash):

uv run python src/process_discovery/react_agent.py \
    --provider gemini --model gemini-2.5-flash \
    --max-steps 150 --seed 42 \
    --store-dir data/react_store_new_run

BPMN generation runs automatically at the end of each agent run.

4. Re-run the baselines:

# One-shot Flash
uv run python src/process_discovery/baseline_oneshot.py \
    --store-dir data/baseline_oneshot_new \
    --output-dir data/baseline_oneshot_new \
    --model gemini-2.5-flash --no-structured

# K-Means
uv run python src/process_discovery/cluster_baseline.py \
    --store-dir data/react_store \
    --output-dir data/kmeans_new \
    --n-clusters 55 \
    --provider gemini --model gemini-2.5-flash

5. Prepare the message store from scratch (only needed after regenerating transcripts):

uv run python src/process_discovery/prepare_store.py

Part III — Workflow Automation

Given a BPMN process model (in PME JSON format), an LLM agent generates a validated n8n workflow and optionally deploys and simulates it.

PME JSON  ──►  agent.py  ──►  validated n8n JSON  ──►  pipeline.py  ──►  simulation results

The automation pipeline is a self-contained uv project at src/automation/. All commands below assume you are inside that directory.

Conversion modes

Mode	How it works	Recommendation
`single`	LLM emits the entire workflow in one tool call	Small BPMNs (< 20 elements)
`multi_turn`	Phase 1: nodes. Phase 2: nested connections.	Medium BPMNs (20–40 elements)
`multi_turn_edges`	Phase 1: nodes. Phase 2: flat edge list.	Large BPMNs (40+ elements)
`code_interpreter`	LLM writes Python using helper functions; executed in a Jupyter kernel.	Default — 100% pass@3 on full corpus

Setup

cd src/automation
cp .env.example .env   # fill in FPT_AI_API_KEY and N8N_API_KEY
uv sync

# Verify connection
uv run python scripts/check_fpt.py
uv run python scripts/check_fpt_tools.py

Key environment variables:

Variable	Default	Purpose
`FPT_AI_API_KEY`	(required)	FPT AI Factory bearer token
`FPT_AI_MODEL`	`Kimi-K2.5`	LLM for workflow generation
`FPT_AI_FEEDBACK_MODEL`	`Qwen3-32B`	LLM for simulation feedback
`N8N_BASE_URL`	`http://localhost:5678`	Local n8n instance
`N8N_API_KEY`	(optional)	From n8n Settings > API

Run the full corpus eval

cd src/automation

# All 55 models, code_interpreter mode, 24 parallel workers
uv run python eval.py --mode code_interpreter --workers 24

# Specific models
uv run python eval.py --mode code_interpreter --models 1,17,18

# Resume interrupted run
uv run python eval.py --run-dir data/eval_runs/EXISTING_DIR --skip-existing

Outputs: data/n8n_workflows/generated/pmo-NN.json (workflow) and pmo-NN.meta.json (metadata).

Simulation results (ground-truth BPMNs)

Results from running pipeline.py against all 55 generated workflows (run date: 2026-04-11):

Metric	Value
Total workflows	55
Pass rate (any attempt)	9/55 (16%)
Pass rate among acyclic BPMNs	9/14 (64%)
Mean F1 on passing workflows	0.810

Failure breakdown:

Failure source	Count	Notes
`n8n_limitation`	41 (75%)	BPMN loops → n8n merge node deadlock
`translation_error`	5 (9%)	Gateway routing issues
`bpmn_quality`	2 (4%)	Ambiguous source BPMN
pass	9 (16%)	—

The 75% failure rate is due to looping exclusive gateways in the PMo BPMNs, which n8n cannot represent. Among the 14 acyclic BPMNs, 9 execute successfully (64%).

Tests

cd src/automation
uv run pytest tests/ -v

End-to-End Pipeline

end_to_end.py chains all five stages in a single run:

Stage 1  prepare_store.py    — flatten annotated transcripts into message store
Stage 2  react_agent.py      — process discovery → discovered BPMNs
Stage 3  bpmn_to_pme.py      — BPMN XML → PME JSON
Stage 4  agent.py            — PME JSON → validated n8n workflow JSON
Stage 5  pipeline.py         — deploy, execute, validate, feedback loop

# Full run from raw transcripts
python end_to_end.py \
    --transcripts-dir  data/transcripts \
    --store-dir        data/my_run/message_store \
    --bpmn-dir         data/my_run/bpmn \
    --pme-dir          data/my_run/pme \
    --n8n-dir          data/my_run/n8n_workflows \
    --results-dir      data/my_run/simulation_results \
    --provider         gemini \
    --max-steps 150    --max-retries 2

# Skip to a later stage (e.g. you already have BPMNs)
python end_to_end.py --start-stage 3 --bpmn-dir data/my_run/bpmn ...

Archived end-to-end runs:

Directory	Description
`data/end_to_end/`	Full 5-stage run (discovery eval included)
`data/best_discovery/`	Stages 4–5 seeded with best agent run (`archive_react_v5c`)

End-to-end results summary (data/end_to_end/):

Stage	Key result
Discovery (Stage 2)	48 processes discovered; Lenient F1=0.732, Hungarian F1=0.699, 12/12 domains covered
Simulation (Stage 5)	35/57 attempts pass (41 unique models); mean F1=0.905 on passing

Best-discovery simulation (data/best_discovery/):

Stage	Key result
Translation (Stage 4)	67 discovered processes translated to n8n
Simulation (Stage 5)	50/102 attempts pass; mean F1=0.907 on passing

Current Status

Phase	Status	Output
0 — Actor classification	✅ done	`data/pmo-dataset/actors.json`
1 — Path enumeration	✅ done	`data/pmo-dataset/scenarios/` (243 paths)
2 — Prompt infrastructure	✅ done	`src/data_generation/{channel_assigner, prompt_builder, validate, generate}.py`
3 — Pilot generation	✅ done	`data/transcripts/` (15 scenarios)
4 — Bulk generation	✅ done	`data/transcripts/` (all 243 scenarios)
5 — Process discovery	✅ done	`data/archive_react_v5c/` (best run) + baselines
6 — Workflow automation	✅ done	`data/n8n_workflows/generated/` (55 workflows)
7 — End-to-end pipeline	✅ done	`data/end_to_end/`, `data/best_discovery/`
LLM judge evaluation	✅ done	`data/react_store/llm_judge_results.csv`

Notes

notes/Worklog.md — full session-by-session log of every design decision, bug found, and fix applied. Read this first if you are joining mid-project.
notes/Plan.md — original high-level plan (Plan A: BPMN→DCR; Plan B: use BPMN as ground truth).
notes/Findings.md — why the PDC dataset was rejected in favour of PMo.
src/automation/README.md — detailed documentation for the n8n automation pipeline.

Name		Name	Last commit message	Last commit date
Latest commit History 79 Commits
data		data
notebook		notebook
notes		notes
results		results
src		src
third_party/ProMoAI		third_party/ProMoAI
.DS_Store		.DS_Store
.gitignore		.gitignore
.python-version		.python-version
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

OrgFlow — Communication Transcripts for Process Discovery and Workflow Automation

Overview

Repository Structure

Dataset: PMo Dataset

Part I — Corpus Generation

Pipeline (Phases 0–4)

Channel assignment rules

Transcript format

Quick start — corpus generation

Part II — Process Discovery

Setup

Results (threshold τ=0.5, 12-domain groupings)

Reproducing results

Part III — Workflow Automation

Conversion modes

Setup

Run the full corpus eval

Simulation results (ground-truth BPMNs)

Tests

End-to-End Pipeline

Current Status

Notes

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

OrgFlow — Communication Transcripts for Process Discovery and Workflow Automation

Overview

Repository Structure

Dataset: PMo Dataset

Part I — Corpus Generation

Pipeline (Phases 0–4)

Channel assignment rules

Transcript format

Quick start — corpus generation

Part II — Process Discovery

Setup

Results (threshold τ=0.5, 12-domain groupings)

Reproducing results

Part III — Workflow Automation

Conversion modes

Setup

Run the full corpus eval

Simulation results (ground-truth BPMNs)

Tests

End-to-End Pipeline

Current Status

Notes

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages