🤔Assuming a large language model application takes 5 GPUs to serve, does it require 50 GPUs to serve 10 different LLM apps?
🌟Out latest research project, Punica, enables serving multiple LoRA finetuned LLMs at the cost of one!
github.com/punica-ai/puni…
We are building our in-house LLM inference stack. Join us if this excites you!
And, I have a more in-depth tutorial about achieving 3200 Gbps here: le.qun.ch/en/blog/2024/1…
Using a custom RDMA-based networking library, we've been able to achieve 3200 Gbps GPU memory transfers, bypassing NCCL limits for 97.1% theoretical bandwidth efficiency.
Our latest blog shares our journey of building a custom high-performance networking solution on AWS.
10x faster than PyTorch All-to-All. 2x faster than DeepEP on single node. Although 2x slower than DeepEP on 128 GPUs, our impl is less picky about hardware and software. Make your MoE go brrr github.com/ppl-ai/pplx-ke…
We've built custom NVSHMEM-based kernels for Mixture-of-Experts (MoE) models that deliver up to 10x faster communication than standard all-to-all operations.
Our approach balances performance with adaptability across different hardware configurations.
DeepSeek-V3/R1 contains 671B total parameters but activates only 37B per token.
Testing shows EP128 configurations deliver up to 5x higher throughput at equivalent output speeds compared to single-node deployments. Higher EP values assign fewer experts per GPU, reducing memory
How? We developed a CUDA kernel, called SGMV, that efficiency runs different LoRA models in a batch. SGMV enables strong batching effect.
🔥Increasing batch size does not increase latency significantly. You can run multiple LoRA models at the cost of one.
(1/4) Announcing FlashInfer, a kernel library that provides state-of-the-art kernel implementations for LLM Inference/Serving.
FlashInfer's unique features include:
- Comprehensive Attention Kernels: covering prefill/decode/append attention for various KV-Cache formats (Page
Really good observation from @tianle_cai and @junrushao . I did a quick sanity check. Delta between Mixtral 8x7B MoE and Mistral 7B is NOT low-rank. SGMV is not applicable here. We need new research :)
Workload-wise, the dynamic routing operator per se in MoE is pretty similar to the SGMV kernel in multi-LoRA serving as introduced in Pinuca @abcdabcd987@ye_combinator :))
I prefer this UI (win2003 even better) to today's UI. Today's UI feels inconsistent, whitespace is too big, info is hidden in nested menus. Screen and resolution gets bigger and bigger, but information density gets lower and lower.
