Social Story Frames

SocialStoryFrames is a formalism for distilling contextually plausible inferences about reader response to storytelling in online communities.

This repo contains code and data for replicating the SocialStoryFrames paper: https://arxiv.org/abs/2512.15925.

Key Components

• SSF-Taxonomy: A two-tier taxonomy of 10 dimensions of reader response, grounded in narrative theory, linguistic pragmatics, and psychology.
• SSF-Generator: model for generating structured inferences about reader response, given a story and its conversational contexts
• SSF-Classifier: multi-label classifier for mapping inferences onto SSF-Taxonomy subdimensions
• SSF-Corpus: Dataset of 6,240 Reddit stories + conversational contexts, annotated with plausible reader response inferences and subdimension labels
• SSF-Sim: A relative metric for comparing online communities in terms of storytelling practices and reader response patterns

Quick Start

Install Requirements

pip install -r requirements.txt

Demos

• demos/main_demo.ipynb: run SSF-Generator and SSF-Classifier on SSF-Corpus stories
• demos/ssf_sim_demo.ipynb: compare storytelling practices of subreddits with SSF-Sim metric

Repo Structure

• configs/: YAML configuration files for the replication pipeline. Controls which stages to (re)run, model settings, hyperparameters, and directory paths.

• data/: Contains the SSF-Taxonomy definitions and SSF-Corpus data. The taxonomy/ subdirectory has dimension definitions and prompt templates. The replication/ subdirectory stores pipeline outputs (e.g., corpus, model checkpoints, inferences, classifications, annotations, ablations, etc).

• src/ssf/: Core framework implementation. Orchestrates SSF-Corpus creation, inference and classification tasks, and metrics (including ssf-sim).

• pipelines/: Main replication script (replicate.py) that orchestrates the full pipeline from corpus preprocessing through model training to evaluation.

• scripts/: Training and evaluation scripts for fine-tuning SSF-Generator and SSF-Classifier models, plus utilities for uploading to HuggingFace.

• analysis/: Jupyter notebooks and Python scripts for generating paper results, figures, and tables. Includes analyses of model performance and exploratory descriptive analyses of storytelling practices across Reddit. Also handles inter-annotator agreement, ablation studies, error analyses, etc.

• demos/: Interactive notebooks demonstrating how to download and use SSF-Corpus, SSF-Generator and SSF-Classifier.

• results/: Model outputs and evaluation metrics from replication runs.

Replication

We provide a couple of paper replication paths below with varying levels of effort/completeness. can rehydrate the raw texts to enable

[Quick] Analysis Replication

Due to privacy concerns (among others), we cannot republish the raw posts/comments from ConvoKit reddit-corpus-small corpus on which SSF-Corpus is based. However, we have staged the intermediary result files generated by replication.py, minus the raw storytelling texts, in data/replication/corpus/text_masked/. Run the following to rehydrate the raw texts:

cd scripts && python rehydrate_corpus.py

After running this step, you can proceed with the analysis notebooks (see below).

Full Replication

This will take at least 72 hours and will require some occasional babysitting due to how the code handles the OpenAI batch API. The code caches progress across runs, so you will not lose all progress for unexpected program termination. A rough cost estimate for the OpenAI context summarization and reference data generation for SSF-Generator and SSF-Classifier is $75-$100. We developed and ran this code with access to three NVIDIA RTX A6000 GPUs.

pipelines/replication.py orchestrates corpus creation and model reference data generation, training, and inference for both SSF-Generator and SSF-Classifier:

cd pipelines && python replication.py -c ../configs/replication.yaml

Running the Analysis

After completing either replication method above, run the notebooks in the analysis dir to replicate our main analyses (given the cached model outputs). The primary analysis notebook covering most of the main results in the paper is analysis/main_analysis.ipynb.

Questions

Please open an issue or contact Joel Mire with any questions.

Citation

@ARTICLE{Joel2025-od,
  title         = "Social story frames: Contextual reasoning about narrative
                   intent and reception",
  author        = "Joel, Mire and Maria, Antoniak and Steven, R Wilson and
                   Zexin, Ma and Achyutarama, R Ganti and Andrew, Piper and
                   Maarten, Sap",
  journal       = "arXiv [cs.CL]",
  month         =  dec,
  year          =  2025,
  archivePrefix = "arXiv",
  primaryClass  = "cs.CL"
}