Visible Multi-Persona Reasoning Model + Benchmark Evidence
Society of Thought is a public preview of a Qwen adapter that reasons through a visible multi-persona debate rather than a single inner voice.
On the released medium comparison, the same trained model scores 0.732 in debate mode and 0.197 in monologue mode across 40 examples, a gap of +0.535. Task score improves by +0.200, and disagreement quality improves by +0.560.
This is evidence on the benchmark designed to test that behavior. It is not yet a claim about broader transfer outside this setup.
- Live demo: scasella91/society-of-thought-bench-demo
- Adapter page: scasella91/society-of-thought-qwen3-30b-paper-faithful-adapter
- Evidence: Debate vs Monologue Summary
- Audit materials: artifact_hardening/README.md
- a released model can carry out a visible multi-persona debate inside its reasoning trace
- on the benchmark built to test that behavior, debate mode beats monologue mode for the same trained model
- the traces are inspectable through the demo, raw samples, and audit pack
- broader gains outside this benchmark
- stronger reliability on harder branching and reconciliation
- replication beyond the current Qwen-centered setup
- the Society of Thought adapter for
Qwen/Qwen3-30B-A3B - a live demo for inspecting raw reasoning traces
society-of-thought-bench, the benchmark and evidence layer for the release- result files, audit materials, and helper scripts for inspection and evaluation
Start with the released comparison, then read the model overview and the raw sample, then move to the audit pack.
After that, use the live demo or local scripts for exploratory prompting. Benchmark-style prompts are still the most faithful setting for the released behavior.
The benchmark is the evidence layer for this release. It is designed to test whether a visible multi-persona reasoning trace is more than a stylistic wrapper.
The benchmark enforces a simple separation:
- the reasoning trace stays inside
<think>...</think> - the visible answer stays outside it
It then measures whether the debate is grounded, interactive, and useful rather than merely present.
Primary evidence:
<think>
<cast_of_characters>
<persona1>Role: ... Personality: ... Expertise: ... Style: ...</persona1>
<persona2>...</persona2>
...
</cast_of_characters>
<conversation>
<think1>...</think1>
<think2>...</think2>
...
</conversation>
<group_solution>...</group_solution>
</think>
<answer>...</answer>
<support>...</support> # evidence tasks only
<think>single-voice reasoning</think>
<answer>...</answer>
<support>...</support> # evidence tasks only
Within that format, the benchmark rewards:
- distinct personas with distinct roles
- challenge rather than shallow agreement
- grounded back-and-forth
- alternative paths on harder tasks
- reconciliation into one final answer
Final task accuracy still matters, but only as a grounding term in the default profile.
A procedural arithmetic target task that checks whether the discussion stays grounded in the numbered inputs.
- inputs are numbered
N1..Nkplus targetT - the answer is a final arithmetic expression inside
<answer> - scoring checks safe expression use, number use, and target closeness
A synthetic evidence-reconciliation task that checks whether the discussion stays grounded in the cited evidence snippets.
- evidence snippets are numbered
E1..En - the answer is
TRUE,FALSE, orINSUFFICIENTinside<answer> - decisive evidence IDs go in
<support> - scoring checks verdict correctness and support quality
- GitHub repo: scasella/society-of-thought-bench
- Hugging Face adapter repo: scasella91/society-of-thought-qwen3-30b-paper-faithful-adapter
- Hugging Face demo: scasella91/society-of-thought-bench-demo
- base model:
Qwen/Qwen3-30B-A3B
The published adapter is the model used for the released comparison and the live demo.
From this directory:
uv pip install -e .
uv run pytest -qFrom the workspace root:
prime env install society-of-thought-bench -p environments
uv run python -c 'import verifiers as vf; vf.load_environment("society-of-thought-bench", family="all", difficulty="medium", num_train=8, num_eval=8)'To try it in a browser:
To see the raw visible reasoning trace:
uv run python scripts/try_tinker_checkpoint.py --family countdown --difficulty medium --show-raw-responseTo chat with the checkpoint directly from the terminal:
uv run python scripts/chat_tinker_checkpoint.py --show-raw-responseUse the chat path for exploration. Use the benchmark examples and audit materials for the clearest view of the released behavior.
- Model Overview
- Findings
- Results
- Raw Sample
- Limitations
- Additional Validation and Audit Materials
- HF Model Card Source
- GitHub Announcement Copy
- Hugging Face Announcement Copy
- Release Notes
society_of_thought_bench/benchmark implementationscripts/data, training, evaluation, and live-try helperstests/local verificationrelease_preview/public release materials
For the full statement, see release_preview/LIMITATIONS.md.
- This is an experimental research preview, not a final benchmark release.
- The strongest evidence comes from the released debate-versus-monologue comparison within this benchmark.
- Medium difficulty is currently the most reliable setting.
- Harder traces still need richer branching and stronger reconciliation.
This preview package is released under the MIT License.