Awesome Multimodal LLMs for Time Series Analysis

Time series, traditionally represented as a temporally ordered sequence of numbers, can be flexibly expressed across diverse modalities, including text, images, graphs, audio, and tables

👂TL;DR

• 🙋‍♂ This paper list compiles time series research in various modalities and representative MLLMs compatible with those modelities

Table of Contents

• 💡 About
• 📑 Papers
  • 📈 Time Series Modalities
    • Numbers
    • Text
    • Images
    • Graphs
    • Audio
    • Tables
  • 🤖 Representative Multimodal LLMs
    • Text & Image - compatible LLMs
    • Graph - compatible LLMs
    • Audio - compatible LLMs
    • Table - compatible LLMs
• 🕑 Future Directions

About

• We survey existing work from data and model perspectives: time series modalities and multimodal LLMs.
• From the data perspective, we emphasize that time series, traditionally represented as a temporally ordered sequence of numbers, can be flexibly expressed across diverse modalities, including text, images, graphs, audio, and tables
• From the model perspective, we introduce representative MLLMs that are either currently applicable or hold potential for specific time series modalities.

Papers

Time Series Modalities

Numbers

• TimeGPT-1
  Azul Garza, Cristian Challu, Max Mergenthaler-Canseco. [paper] [code]
• Lag-llama: Towards foundation models for time series forecasting
  Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Hena Ghonia, Rishika Bhagwatkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopoulos, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zantedeschi, Yuriy Nevmyvaka, Irina Rish. [paper] [code]
• A decoder-only foundation model for time-series forecasting
  Abhimanyu Das, Weihao Kong, Rajat Sen, Yichen Zhou. [paper] [code]
• Timer: Generative Pre-trained Transformers Are Large Time Series Models
  Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, Mingsheng Long. [paper] [code]
• Unified Training of Universal Time Series Forecasting Transformers
  Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, Doyen Sahoo. [paper] [code]
• MOMENT: A Family of Open Time-series Foundation Models
  Azul Garza, Cristian Challu, Max Mergenthaler-Canseco. [paper] [code]
• Chronos: Learning the language of time series
  Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, Yuyang Wang. [paper] [code]
• Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series
  Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam H. Nguyen, Wesley M. Gifford, Chandra Reddy, Jayant Kalagnanam. [paper] [code]
• Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
  Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, Ming Jin. [paper] [code]
• Pre-training Time Series Models with Stock Data Customization
  Mengyu Wang, Tiejun Ma, Shay B. Cohen. [paper] [code]

Text

• PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting
  Hao Xue, Flora D. Salim. [paper] [code]
• One Fits All:Power General Time Series Analysis by Pretrained LM
  Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, Rong Jin. [paper] [code]
• LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
  Ching Chang, Wei-Yao Wang, Wen-Chih Peng, Tien-Fu Chen. [paper]
• TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series
  Chenxi Sun, Hongyan Li, Yaliang Li, Shenda Hong. [paper] [code]
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
  Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, Qingsong Wen. [paper] [code]
• TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting
  Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, Yan Liu. [paper] [code]
• Large Language Models Are Zero-Shot Time Series Forecasters
  Nate Gruver, Marc Finzi, Shikai Qiu, Andrew Gordon Wilson. [paper] [code]
• UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting
  Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, Roger Zimmermann. [paper] [code]
• LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting
  Haoxin Liu, Zhiyuan Zhao, Jindong Wang, Harshavardhan Kamarthi, B. Aditya Prakash. [paper] [code]
• S^2IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting
  Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, Dongjin Song. [paper] [code]
• Advancing Time Series Classification with Multimodal Language Modeling
  Mingyue Cheng, Yiheng Chen, Qi Liu, Zhiding Liu, Yucong Luo. [paper] [code]
• GPT4MTS: Prompt-based Large Language Model for Multimodal Time-series Forecasting
  Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, Yan Liu. [paper]
• TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment
  Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, Rui Zhao. [paper] [code]
• ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data
  Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, Jianxin Liao. [paper] [code]
• Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series
  Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, Yuntian Chen. [paper]
• Random Initialization Can’t Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting
  Roland Riachi, Kashif Rasul, Arjun Ashok, Prateek Humane, Alexis Roger, Andrew R. Williams, Yuriy Nevmyvaka, Irina Rish. [paper]
• EventTSF: Event-Aware Non-Stationary Time Series Forecasting
  Yunfeng Ge, Ming Jin, Yiji Zhao, Hongyan Li, Bo Du, Chang Xu, Shirui Pan. [paper]
• Leveraging Language Foundation Models for Human Mobility Forecasting
  Hao Xue, Bhanu Prakash Voutharoja, Flora D. Salim. [paper] [code]
• Where Would I Go Next? Large Language Models as Human Mobility Predictors
  Xinglei Wang, Meng Fang, Zichao Zeng, Tao Cheng. [paper] [code]
• Towards Explainable Traffic Flow Prediction with Large Language Models
  Xusen Guo, Qiming Zhang, Junyue Jiang, Mingxing Peng, Meixin Zhu, Hao (Frank) Yang. [paper] [code]
• The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges
  Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, Jimin Huang. [paper]
• Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models
  Alejandro Lopez-Lira, Yuehua Tang. [paper]
• Temporal Data Meets LLM -- Explainable Financial Time Series Forecasting
  Xinli Yu, Zheng Chen, Yuan Ling, Shujing Dong, Zongyi Liu, Yanbin Lu. [paper]
• Frozen Language Model Helps ECG Zero-Shot Learning
  Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, Shenda Hong. [paper]
• Large Language Models are Few-Shot Health Learners
  Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, Shwetak Patel. [paper]
• MedualTime: A Dual-Adapter Language Model for Medical Time Series-Text Multimodal Learning
  Jiexia Ye, Weiqi Zhang, Ziyue Li, Jia Li, Meng Zhao, Fugee Tsung. [paper] [code]
• MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis
  Nimeesha Chan, Felix Parker, William Bennett, Tianyi Wu, Mung Yao Jia, James Fackler, Kimia Ghobadi. [paper] [code]

Images

• Insight Miner: A Time Series Analysis Dataset for Cross-Domain Alignment with Natural Language
  Yunkai Zhang, Yawen Zhang, Ming Zheng, Kezhen Chen, Chongyang Gao, Ruian Ge, Siyuan Teng, Amine Jelloul, Jinmeng Rao, Xiaoyuan Guo, Chiang-Wei Fang, Zeyu Zheng, Jie Yang. [paper] [dataset]
• TimeSeriesExam: A time series understanding exam
  Yifu Cai, Arjun Choudhry, Mononito Goswami, Artur Dubrawski. [paper] [code]
• Plots Unlock Time-Series Understanding in Multimodal Models
  Mayank Daswani, Mathias M.J. Bellaiche, Marc Wilson, Desislav Ivanov, Mikhail Papkov, Eva Schnider, Jing Tang, Kay Lamerigts, Gabriela Botea, Michael A. Sanchez, Yojan Patel, Shruthi Prabhakara, Shravya Shetty, Umesh Telang. [paper]
• Can LLMs Understand Time Series Anomalies?
  Zihao Zhou, Rose Yu. [paper] [code]
• See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzers
  Jiaxin Zhuang, Leon Yan, Zhenwei Zhang, Ruiqi Wang, Jiawei Zhang, Yuantao Gu. [paper]
• A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization
  Haoxin Liu, Chenghao Liu, B. Aditya Prakash. [paper]
• On the Feasibility of Vision-Language Models for Time-Series Classification
  Vinay Prithyani, Mohsin Mohammed, Richa Gadgil, Ricardo Buitrago, Vinija Jain, Aman Chadha. [paper] [code]
• Can Multimodal LLMs Perform Time Series Anomaly Detection?
  Xiongxiao Xu, Haoran Wang, Yueqing Liang, Philip S. Yu, Yue Zhao, Kai Shu. [paper] [code]
• Time Series as Images: Vision Transformer for Irregularly Sampled Time Series
  Zekun Li, Shiyang Li, Xifeng Yan. [paper] [code]
• Multimodal LLMs for health grounded in individual-specific data
  Anastasiya Belyaeva, Justin Cosentino, Farhad Hormozdiari, Krish Eswaran, Shravya Shetty, Greg Corrado, Andrew Carroll, Cory Y. McLean, Nicholas A. Furlotte. [paper]
• Harnessing Vision-Language Models for Time Series Anomaly Detection
  Zelin He, Sarah Alnegheimish, Matthew Reimherr. [paper] [code]

Graphs

• GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputation
  Yakun Chen, Xianzhi Wang, Guandong Xu. [paper]
• Spatio-Temporal Graph Learning with Large Language Model
  Qianru Zhang, Xubin Ren, Lianghao Xia, Siu Ming Yiu, Chao Huang. [paper]
• Spatio-Temporal Graph Convolutional Network Combined Large Language Model: A Deep Learning Framework for Bike Demand Forecasting
  Peisen Li, Yizhe Pang, Junyu Ren. [paper]
• Strada-LLM: Graph LLM for traffic prediction
  Seyed Mohamad Moghadas, Yangxintong Lyu, Bruno Cornelis, Alexandre Alahi, Adrian Munteanu. [paper]
• LLM-based Online Prediction of Time-varying Graph Signals
  Dayu Qin, Yi Yan, Ercan Engin Kuruoglu. [paper]
• ChatGPT Informed Graph Neural Network for Stock Movement Prediction
  Zihan Chen, Lei Nico Zheng, Cheng Lu, Jialu Yuan, Di Zhu. [paper] [code]
• Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognition
  Haojun Xu, Yan Gao, Zheng Hui, Jie Li, Xinbo Gao. [paper] [code]

Audio

• Voice2Series: Reprogramming Acoustic Models for Time Series Classification
  Chao-Han Huck Yang, Yun-Yun Tsai, Pin-Yu Chen. [paper] [code]

Tables

• TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models
  Jiahao Wang, Mingyue Cheng, Qingyang Mao, Yitong Zhou, Feiyang Xu, Xin Li. [paper] [code]
• The Tabular Foundation Model TabPFN Outperforms Specialized Time Series Forecasting Models Based on Simple Features
  Shi Bin Hoo, Samuel Müller, David Salinas, Frank Hutter. [paper] [code]

Representative Multimodal LLMs

Text & Image - compatible LLMs

• GPT-4 Technical Report
  OpenAI. [paper]
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  Gemini Team Google. [paper]
• Claude 3 Haiku: our fastest model yet
  Anthropic. [blog]
• The Llama 3 Herd of Models
  Meta AI Llama Team. [paper] [code]
• Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  Qwen Team Alibaba Group. [paper] [code]
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang. [paper] [code]

Graph - compatible LLMs

• How Can Large Language Models Understand Spatial-Temporal Data?
  Lei Liu, Shuo Yu, Runze Wang, Zhenxun Ma, Yanming Shen. [paper]
• UrbanGPT: Spatio-Temporal Large Language Models
  Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, Chao Huang. [paper] [code]
• STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM
  YiHeng Huang, Xiaowei Mao, Shengnan Guo, Yubin Chen, Junfeng Shen, Tiankuo Li, Youfang Lin, Huaiyu Wan. [paper]

Audio - compatible LLMs

• SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
  Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, Xipeng Qiu. [paper] [code]
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
  Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, Shinji Watanabe. [paper] [code]
• MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
  Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, Jinren Zhou. [paper]

Table - compatible LLMs

• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
  Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter. [paper] [code]
• TableLlama: Towards Open Large Generalist Models for Tables
  Tianshu Zhang, Xiang Yue, Yifei Li, Huan Sun. [paper] [code]
• TableGPT2: A Large Multimodal Model with Tabular Data Integration
  Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Gang Chen, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Junlin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long, Liyao Li, Pengzuo Wu, Qi Zhang, Qingyi Huang, Saisai Yang, Tao Zhang, Wentao Ye, Wufang Zhu, Xiaomeng Hu, Xijun Gu, Xinjie Sun, Xiang Li, Yuhang Yang, Zhiqing Xiao. [paper] [code]

Future Directions

Video

• Deep Video Prediction for Time Series Forecasting
  Zhen Zeng, Tucker Balch, Manuela Veloso. [paper]

Reasoning

• Language Models Still Struggle to Zero-shot Reason about Time Series
  Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, Tim Althoff. [paper] [code]
• Towards Time Series Reasoning with LLMs
  Winnie Chow, Lauren Gardiner, Haraldur T. Hallgrímsson, Maxwell A. Xu, Shirley You Ren. [paper]
• Position: Empowering Time Series Reasoning with Multimodal LLMs
  Yaxuan Kong, Yiyuan Yang, Shiyu Wang, Chenghao Liu, Yuxuan Liang, Ming Jin, Stefan Zohren, Dan Pei, Yan Liu, Qingsong Wen. [paper]
• Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time Series Forecasting: A Benchmark and Insights
  Haoxin Liu, Zhiyuan Zhao, Shiduo Li, B. Aditya Prakash. [paper] [code]
• Time-R1: Towards Comprehensive Temporal Reasoning in LLMs
  Zijia Liu, Peixuan Han, Haofei Yu, Haoru Li, Jiaxuan You. [paper] [code]
• Time Series Forecasting as Reasoning: A Slow-Thinking Approach with Reinforced LLMs
  Yucong Luo, Yitong Zhou, Mingyue Cheng, Jiahao Wang, Daoyu Wang, Tingyue Pan, Jintao Zhang. [paper] [code]
• Can Large Language Models Adequately Perform Symbolic Reasoning Over Time Series?
  Zewen Liu, Juntong Ni, Xianfeng Tang, Max S.Y. Lau, Wenpeng Yin, Wei Jin. [paper]

Agents

• Agentic Retrieval-Augmented Generation for Time Series Analysis
  Chidaksh Ravuru, Sagar Srinivas Sakhinana, Venkataramana Runkana. [paper]
• Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models
  Yile Gu, Yifan Xiong, Jonathan Mace, Yuting Jiang, Yigong Hu, Baris Kasikci, Peng Cheng. [paper]
• LLM Knows Geometry Better than Algebra: Numerical Understanding of LLM-Based Agents in A Trading Arena
  Tianmi Ma, Jiawei Du, Wenxin Huang, Wenjie Wang, Liang Xie, Xian Zhong, Joey Tianyi Zhou. [paper] [code]
• FinArena: A Human-Agent Collaboration Framework for Financial Market Analysis and Forecasting
  Congluo Xu, Zhaobin Liu, Ziyang Li. [paper]

Interpretability and Hallucination

• Large Language Models can Deliver Accurate and Interpretable Time Series Anomaly Detection
  Jun Liu, Chaoyun Zhang, Jiaxu Qian, Minghua Ma, Si Qin, Chetan Bansal, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang. [paper] [code]
• Can LLMs Serve As Time Series Anomaly Detectors?
  Manqing Dong, Hao Huang, Longbing Cao. [paper]

👏 Contributing to this paper list

There may be cases where we overlook important work in this rapidly evolving field. Please feel free to share your awesome work or relevant contributions here! Thanks in advance for your efforts!

📝 Citation

If you find our work useful, please cite the below paper:

@article{xu2025beyond,
  title={Beyond Numbers: A Survey of Time Series Analysis in the Era of Multimodal LLMs},
  author={Xu, Xiongxiao and Zhao, Yue and Philip, S Yu and Shu, Kai},
  journal={Authorea Preprints},
  year={2025},
  publisher={Authorea}
}