Note
This is a case study repository. The source code is proprietary and not publicly available.
An AI-powered voice assistant for HIV pre-screening health education — a university-industry collaboration between National Yunlin University of Science and Technology (YunTech) and St. Joseph Hospital (天主教若瑟醫療財團法人若瑟醫院).
A voice-based AI assistant that guides patients through a structured 6-stage HIV pre-screening education flow. Deployed as both a clinic kiosk with an animated character and a LINE Bot for remote access. The system conducts personalized conversations, assesses risk factors, retrieves medical knowledge via RAG, and notifies nursing staff upon completion.
Built as a university-industry collaboration project between YunTech and St. Joseph Hospital — 2-person development team.
- Function calling stage transitions — Each stage defines its own allowed tool set. The LLM triggers stage-specific function calls (e.g.
stage1_complete,stage4_complete) to collect structured data and advance the state machine — no hardcoded transition logic, fully driven by OpenAI function calling. - 6-stage state machine — Identity verification, screening motivation, test history, risk assessment, personalized education, and satisfaction survey. Stage prompts, allowed tools, and conversation context are dynamically swapped per stage.
- Real-time voice conversation — Full-duplex voice via OpenAI Realtime API over WebSocket, with built-in STT, TTS, and VAD. Audio streamed as PCM16 from sounddevice, responses rendered with character-level subtitle synchronization.
- RAG medical knowledge retrieval — FAISS cosine similarity search over OpenAI Embeddings of a curated HIV Q&A knowledge base. Retrieved context is injected into the LLM prompt during Stage 5, with dynamic forgetting to manage token limits.
- Animated character kiosk — Tkinter full-screen app with VLC-driven character animations (listening, explaining, idle states) synchronized to conversation events via a pub/sub EventBus.
- LINE Bot remote access — Flask webhook with LINE Bot SDK, Flex message cards, and automatic Cloudflare Tunnel deployment with LINE webhook URL auto-update.
- Nurse notification system — Subscribed nursing staff receive LINE push notifications with education summaries and structured patient data export (openpyxl).
- Personalized risk context — Stage 4 risk assessment data (behavior type, timing, window period calculation) dynamically shapes Stage 5 education prompts and RAG query context.
- Kiosk App — Tkinter full-screen desktop app with voice conversation, VLC animated character, and embedded subtitles.
- LINE Bot — Flask-based LINE integration with Cloudflare Tunnel auto-deployment and nurse notification.
- Core Engine — 6-stage state machine, RAG knowledge retrieval, and OpenAI Realtime API agent with tool calls.
- OpenAI Realtime API — WebSocket-based real-time voice conversation with function calling support.
| Category | Technologies |
|---|---|
| AI / LLM | OpenAI Realtime API, Function Calling |
| RAG | FAISS, OpenAI Embeddings |
| Voice | sounddevice (recording), OpenAI TTS/STT |
| Desktop | Tkinter, VLC (python-vlc) |
| Bot | Flask, LINE Bot SDK, Cloudflare Tunnel |
| Database | SQLite |
| Language | Python 3.12 |
Developer — 2-person team, university-industry collaboration.
- Developed the 6-stage state machine and agent tool call pipeline
- Created the animated character visuals and kiosk UI
- Implemented RAG knowledge retrieval with FAISS vector search
- Built the LINE Bot integration with Cloudflare Tunnel auto-deployment
- Implemented nurse notification and patient data export
Feel free to reach out via email for details.