Model Card: ViStoryBench

Dataset Description

ViStoryBench is a comprehensive benchmark dataset for story visualization. It aims to thoroughly evaluate and advance the performance of story visualization models by providing diverse story types, artistic styles, and detailed annotations. The goal of story visualization is to generate a sequence of visually coherent and content-accurate images based on a given narrative text and character reference images.

Key features of this dataset include:

• Diversity: Encompasses various story types (e.g., comedy, horror, fairy tale) and visual aesthetics (e.g., anime, 3D rendering, realistic).
• Character Consistency: Features stories with single and multiple protagonists to test a model's ability to maintain character appearance consistency.
• Complex Narratives: Includes intricate plots and detailed world-building to challenge models in generating accurate visual content.
• Bilingual Support: All test-related texts (e.g., story scripts, shot descriptions) are provided in both English and Chinese.
• Detailed Annotations: Each story segment includes detailed scene descriptions, plot correspondences, characters appearing, static shot descriptions, and shot perspective designs.

ViStoryBench originates from the WildStoryBench benchmark project, designed to provide researchers with a structured, multifaceted framework to thoroughly identify the strengths and weaknesses of different story visualization models, thereby fostering advancements in the field.

Dataset Curators

Cailin Zhuang, Ailin Huang, Wei Cheng, Jingwei Wu, Yaoqi Hu, Jiaqi Liao, Zhewei Huang, Hongyuan Wang, Xinyao Liao, Weiwei Cai, Hengyuan Xu, Xuanyang Zhang, Xianfang Zeng, Gang Yu Chi Zhang.

Versions

• Dataset Version: 1.0.0
• Configurations:
  • full: The complete dataset containing all 80 stories.
  • lite: A subset containing 20 carefully selected stories, designed to maintain a similar distribution of text styles and character reference image styles as the full dataset.

Dataset Sources and Construction

• Story Sources: The dataset comprises 80 story segments manually collected from various sources, including film and television scripts, literary masterpieces, legends from around the world, novels, and picture books. Overly long stories were summarized with the assistance of Large Language Models (LLMs) to ensure each story is approximately 1000 words.
• Story Type Distribution: Includes 13 folktales, 10 love stories, 4 suspense crime stories, 3 horror stories, 6 historical stories, 10 fantasy stories, 7 science fiction stories, 3 war stories, 10 stories about social life, 3 adventure survival stories, and 11 fairy tales.
• Storyboard Generation: The collected story segments were adapted into storyboards using a Large Language Model (Step-1V). The entire dataset comprises 1317 storyboards, with each story containing between 4 and 30 storyboards, averaging 16.5 storyboards per story.
• Character Reference Images: Images aligning with character descriptions were manually collected from the internet, ensuring consistent character image styles within the same story. For well-known stories, character reference images are sourced from relevant visual works. For the remaining stories, screenshots were either retrieved from films or TV shows with similar settings (for 16 stories) or animation character images were generated by SDXL (for 7 stories).
• Character Statistics:
  • A total of 344 characters, including 190 realistic humans, 135 non-realistic humans, and 19 non-humans.
  • Gender distribution: 210 males, 108 females, and 26 characters with no gender or non-binary gender.
  • Each character has between 1 and up to 10 reference images, with 89 characters having more than one image.
  • The entire dataset consists of 509 reference images.
• Story Classification: All 80 stories are categorized into two types based on the image category of the main characters: 39 realistic stories and 41 unrealistic stories.

Dataset Structure

The dataset is structured at the shot level, meaning each row in the dataset represents a single shot from a story. The fields for each shot are defined in vistorybench.py and are as follows:

• story_id: (String) Unique identifier for the story (e.g., "01", "23").
• story_type_en: (String) English description of the story type.
• story_type_ch: (String) Chinese description of the story type.
• shot_id_in_story: (Integer) Original index of the shot within its story.
• shot_plot_correspondence_en: (String) English description of the plot correspondence for the shot.
• shot_plot_correspondence_ch: (String) Chinese description of the plot correspondence for the shot.
• shot_setting_description_en: (String) English description of the setting for the shot.
• shot_setting_description_ch: (String) Chinese description of the setting for the shot.
• shot_perspective_design_en: (String) English description of the shot perspective design.
• shot_perspective_design_ch: (String) Chinese description of the shot perspective design.
• shot_static_shot_description_en: (String) English static description of the shot.
• shot_static_shot_description_ch: (String) Chinese static description of the shot.
• shot_characters_appearing_names_en: (Sequence of Strings) List of English names of characters appearing in this specific shot.
• shot_characters_appearing_names_ch: (Sequence of Strings) List of Chinese names of characters appearing in this specific shot.
• shot_character_prompts_en: (Sequence of Strings) List of English prompts for the characters appearing in this shot.
• shot_character_prompts_ch: (Sequence of Strings) List of Chinese prompts for the characters appearing in this shot.
• shot_character_image_paths: (Sequence of Strings) List of relative paths to reference images for characters appearing in this shot. The paths are relative to the root of the dataset directory (e.g., "01/image/David/00.jpg").
• story_all_characters_info: (Sequence of Objects) List containing information for all characters defined within the story. Each object in the sequence has the following fields:
  • id: (String) The internal ID or key of the character (e.g., "David").
  • tag: (String) Character tag (e.g., "unrealistic_human").
  • name_ch: (String) Chinese name of the character.
  • name_en: (String) English name of the character.
  • prompt_ch: (String) Chinese description prompt for the character.
  • prompt_en: (String) English description prompt for the character.
  • num_of_appearances_in_story: (Integer) Number of times the character appears in the story.
  • story_character_all_image_paths: (Sequence of Strings) List of relative paths to all reference images for this character in the story.

Dataset File Structure

The raw data for ViStoryBench is organized as follows. The vistorybench.py script processes this structure to generate the dataset.

ViStoryBench/
├── 01/
│   ├── image/
│   │   ├── CharacterName1/  (e.g., David)
│   │   │   ├── 00.jpg
│   │   │   ├── 01.png
│   │   │   └── ...
│   │   ├── CharacterName2/
│   │   │   ├── 00.jpg
│   │   │   └── ...
│   │   └── ...
│   └── story.json
├── 02/
│   ├── image/
│   │   └── ...
│   └── story.json
├── ...
└── 80/
    ├── image/
    │   └── ...
    └── story.json

Each numbered directory (e.g., 01, 02) represents a story.

• Inside each story directory:
  • The image/ subdirectory contains character reference images, organized into subdirectories named after the character (e.g., David). Each character's subdirectory holds their reference images (e.g., 00.jpg, 01.png).
  • story.json contains the textual data for the story, including story-level information, character details, and descriptions for each shot. An example of the story.json structure is:

{
    "Story_type": {
        "en": "Children's Picture Books",
        "ch": "儿童绘本"
    },
    "Characters": {
        "David": {
            "tag": "unrealistic_human",
            "name_ch": "大卫",
            "name_en": "David",
            "prompt_ch": "一个金发碧眼的白种人活泼小男孩...",
            "prompt_en": "A lively Caucasian boy with blond hair and blue eyes...",
            "num_of_appearances": 9
        },
        "David's Mom": {
            // ... details for David's Mom
        }
    },
    "Shots": [
        {
            "index": 2,
            "Plot Correspondence": {
                "ch": "有一天，大卫站在凳子上...",
                "en": "One day, David stood on a stool..."
            },
            "Setting Description": {
                "ch": "白天，厨房...",
                "en": "Daytime, kitchen..."
            },
            "Shot Perspective Design": {
                "ch": "中景，低角度镜头",
                "en": "Medium shot, low-angle shot"
            },
            "Characters Appearing": {
                "ch": ["大卫"],
                "en": ["David"]
            },
            "Static Shot Description": {
                "ch": "大卫站在凳子上，伸手去够糖果盒...",
                "en": "David stands on the stool, reaching for the candy box..."
            }
        },
        // ... more shots
    ]
}

Intended Uses

ViStoryBench is primarily intended for training and evaluating story visualization models. Its detailed annotations and diverse content enable research in areas such as:

• Text-to-image sequence generation.
• Maintaining visual coherence between images.
• Preserving character identity consistency across frames.
• Accurately understanding and visualizing complex scenes and plots.
• Evaluating the quality, text-image alignment, and narrative engagement of generated images.

Limitations and Biases

• Data Sourcing: Some reference images in the dataset are sourced from popular movies, TV shows, animations, etc. This could potentially lead to certain metrics being overfitted to this specific data, creating a possibility for those metrics to be manipulated or "hacked."
• Language Differences: The dataset includes both Chinese and English versions. Although the appropriate language will be chosen for each model, the quality of the generated results may vary due to differences in the language of the instructions.
• Content Policy: When testing with commercial software, content policy restrictions might prevent certain stories from generating results normally, introducing discrepancies in the evaluation of some models.

Licensing Information

MIT License

Citation Information

@misc{ViStoryBench: Comprehensive Benchmark Suite forStory Visualization,
  author    = {Cailin Zhuang, Ailin Huang, Wei Cheng, Jingwei Wu, Yaoqi Hu, Jiaqi Liao, Zhewei Huang, Hongyuan Wang, Xinyao Liao, Weiwei Cai, Hengyuan Xu, Xuanyang Zhang, Xianfang Zeng, Gang Yu Chi Zhang},
  title     = {ViStoryBench: Comprehensive Benchmark Suite forStory Visualization},
  year      = {2025},
  url       = {https://huggingface.co/datasets/ViStoryBench/ViStoryBench}
}

Dataset Homepage

https://huggingface.co/datasets/ViStoryBench/ViStoryBench