Skip to content

About

AI multimodal application platform integrating 9 modules (chat, RAG, image, video, TTS, ASR, Excel) on Streamlit + Qwen/Ali Bailian

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

🤖 AI Multimodal Platform

One Streamlit app, nine AI modules — chat, RAG, text-to-image, image understanding, image-to-video, TTS, ASR and an Excel data assistant, all powered by Alibaba Bailian (DashScope / OpenAI-compatible)

Python Last Commit Repo Size License


An end-to-end multimodal AI application built with Streamlit multi-page architecture on top of Alibaba Bailian models — qwen-plus for chat, qwen-vl-plus for vision, text-embedding-v3 for retrieval, Tongyi Wanxiang for image/video generation, CosyVoice for speech synthesis and Paraformer for speech recognition. One config page manages API keys, models and prompt templates globally.

📝 Note: the app UI is in Chinese; this README documents the project in English.

🖼️ Preview

App home: model metrics and feature navigation
Home page: active model metrics + nine-module navigation sidebar

✨ Highlights

📚 Real RAG, not a demo

Upload PDF / TXT / DOCX / MD / CSV / XLSX, auto-chunk with LangChain, embed with text-embedding-v3, persist in ChromaDB — answers come back with source citations.

🎨 Full multimodal pipeline

Text → image (Tongyi Wanxiang, styles / sizes / batch / negative prompts), image → 5s video (wanx2.1-i2v), text ↔ speech (CosyVoice × 6 voices, Paraformer-v2 with sentence-level timestamps).

💬 Production-style chat

Streaming multi-turn dialogue over the OpenAI-compatible endpoint, with model switching and a managed prompt-template library (10 built-in templates, editable).

📊 Excel data assistant

Upload a spreadsheet, then preview / filter / sort / chart it with Plotly — and ask the LLM for written insights on the data.

🧩 Feature Matrix

# Module Key capabilities
1 💬 Chat Streaming output · multi-model switching · prompt management
2 📚 RAG Knowledge Base 6 file formats · auto chunking · ChromaDB persistence · source citations
3 🎨 Text-to-Image Tongyi Wanxiang · multi-style · multi-size · batch · negative prompts
4 🔍 Image Understanding qwen-vl-plus templates: description / OCR / composition / emotion / product
5 🎬 Image-to-Video wanx2.1-i2v — turn a still image into a ~5s clip
6 🗣️ TTS CosyVoice-v1 · 6 voices · adjustable speech rate & volume
7 🎧 ASR Paraformer-v2 · mixed Chinese/English · sentence-level timestamps
8 📊 Excel Assistant Preview · filter · sort · Plotly charts · AI insights
9 ⚙️ Settings Global API Key / base URL / model selection / prompt templates

🏗️ Architecture

flowchart TD
    UI["Streamlit UI<br/>app.py + 9 pages"] --> CORE["core/ service layer"]
    CORE --> CFG["config.py<br/>global config + prompts"]
    CORE --> LLM["llm.py<br/>OpenAI-compatible client"]
    CORE --> RAG["rag.py<br/>LangChain + ChromaDB"]
    CORE --> IMG["image.py / video.py"]
    CORE --> AUD["tts.py / asr.py"]
    CORE --> VIS["vision.py"]
    CORE --> XLS["excel.py<br/>pandas + Plotly"]
    LLM --> OC["Aliyun OpenAI-compatible endpoint<br/>qwen-plus · qwen-vl-plus · text-embedding-v3"]
    RAG --> OC
    VIS --> OC
    IMG --> DS["DashScope native SDK<br/>wanx2.1-t2i · wanx2.1-i2v"]
    AUD --> DS
    RAG --> CHROMA[("ChromaDB<br/>data/chroma")]
Loading

Two API channels in one app: text/vision/embedding go through the OpenAI-compatible endpoint, while image, video, TTS and ASR use the native DashScope SDK.

🚀 Quick Start

# 1) Install dependencies (Python 3, pip)
pip install -r requirements.txt

# 2) Set your API keys (or skip and enter them in the ⚙️ Settings page)
export OPENAI_API_KEY="sk-..."        # OpenAI-compatible endpoint key
export DASHSCOPE_API_KEY="sk-..."     # DashScope key for image/video/TTS/ASR

# 3) Run (default port 8501)
streamlit run app.py
# or the one-shot launcher:
bash run.sh            # macOS / Linux — auto-installs deps if missing
# on macOS you can also double-click 启动.command

Then open http://127.0.0.1:8501.

📁 Project Structure

├── app.py                    # Streamlit entry point (home page)
├── pages/                    # Multi-page modules
│   ├── 1_💬_对话.py           # Chat
│   ├── 2_📚_RAG知识库.py      # RAG knowledge base
│   ├── 3_🎨_文生图.py         # Text-to-image
│   ├── 4_🔍_图片理解.py       # Image understanding
│   ├── 5_🎬_图生视频.py       # Image-to-video
│   ├── 6_🗣️_语音合成.py       # TTS
│   ├── 7_🎧_语音识别.py       # ASR
│   ├── 8_📊_Excel助手.py      # Excel assistant
│   ├── 9_⚙️_配置.py           # Settings
│   └── _common.py            # Shared page utilities
├── core/                     # Service layer (one module per capability)
│   ├── config.py             # Global config + prompt template management
│   ├── llm.py                # OpenAI-compatible LLM client
│   ├── rag.py                # LangChain + ChromaDB RAG
│   ├── image.py / video.py   # Tongyi Wanxiang t2i / i2v
│   ├── vision.py             # qwen-vl image understanding
│   ├── tts.py / asr.py       # CosyVoice / Paraformer
│   └── excel.py              # Excel data assistant
├── data/                     # Runtime data (chroma / uploads / outputs / prompts.json)
├── assets/                   # Screenshots used by docs
├── requirements.txt
├── run.sh / 启动.command      # Launchers
└── 开发文档.md                # Full development log (zh, PDF version included)

⚙️ Configuration

config.json is gitignored (it holds your API keys) and is created on first run with these defaults; everything can also be edited visually on the ⚙️ Settings page:

{
  "openai_compatible": {
    "base_url": "https://llm-8mft2kgjzzzbyc6v.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
    "api_key": "<OPENAI_API_KEY>",
    "default_chat_model": "qwen-plus",
    "default_vl_model": "qwen-vl-plus",
    "default_embedding_model": "text-embedding-v3"
  },
  "dashscope": {
    "base_url": "https://dashscope.aliyuncs.com/api/v1",
    "api_key": "<DASHSCOPE_API_KEY>",
    "tts_model": "cosyvoice-v1",
    "asr_model": "paraformer-v2",
    "image_model": "wanx2.1-t2i-turbo",
    "video_model": "wanx2.1-i2v-turbo"
  }
}

🔧 Tech Stack

  • Frontend: Streamlit (multi-page)
  • LLM / VL / Embedding: Alibaba Bailian qwen-plus · qwen-vl-plus · text-embedding-v3 (OpenAI-compatible)
  • RAG: LangChain + ChromaDB + DashScope embeddings
  • Multimodal: dashscope SDK (ImageSynthesis / VideoSynthesis / SpeechSynthesizer / Recognition)
  • Data & charts: pandas · Plotly

📝 Documentation

Full development process, AI-tool usage and pitfalls: 开发文档.md (also available as PDF and DOCX), plus the standalone showcase page 项目说明.html.

⚠️ Limitations

  • Requires valid Alibaba Bailian / DashScope API keys — no offline or open-source model fallback.
  • The UI language is Chinese; page file names are Chinese as well.
  • Image-to-video generation is asynchronous and limited to short (~5s) clips.
  • Single-user local deployment — no auth, no multi-tenant session isolation.

Built with Streamlit + LangChain + DashScope | Course project: AI Application Development
Licensed under MIT

About

AI multimodal application platform integrating 9 modules (chat, RAG, image, video, TTS, ASR, Excel) on Streamlit + Qwen/Ali Bailian

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages