Anyscale (@anyscalecompute) / X

Anyscale

1,925 posts

Anyscale

@anyscalecompute

The best way to run @raydistributed, the distributed compute framework for scaling AI workloads.

anyscale.com

Joined September 2019

Following

13K

Followers

Anyscale
@anyscalecompute
Dec 13, 2023
We’re excited to announce the official @MistralAI Mixtral 8x7B model on Anyscale Endpoints, offering the best price on the market with an OpenAI compatible API. 💸 Pricing: $0.5 / million tokens 📆 Coming soon: JSON mode and function calling Try out Mixtral on Anyscale
547K
Anyscale
@anyscalecompute
Dec 21, 2023
📈We’re excited to introduce the LLMPerf leaderboard: the first public and open source leaderboard for benchmarking performance of various LLM inference providers in the market. Our goal with this leaderboard is to equip users and developers with a clear understanding of the
376K
Anyscale
@anyscalecompute
Dec 12, 2023
We're announcing new features and models today. 🔵 JSON mode ⚫️ function calling Try them out with our API.
Blog | Anyscale
From anyscale.com
174K
Anyscale
@anyscalecompute
Jul 1, 2024
1/ 🚀 Introducing RouteLLM: a routing framework based on human preference data for routing queries between powerful proprietary LLMs and cost-effective LLMs, developed in collaboration with @lmsysorg . By intelligently selecting the best model for each query, our router models
20K
Anyscale
@anyscalecompute
Jul 16, 2020
Anyscale is honored to have been recognized by Forbes as one of America’s Most Promising Artificial Intelligence Companies anyscale.com/blog/forbes-ai…
Anyscale
@anyscalecompute
Jul 10, 2024
We’ve recently contributed FP8 support to the @vllm_project in collaboration with @neuralmagic. With this feature, you can see up to a 1.8x reduction in inter-token latency, with >99% accuracy preservation! 1/n
34K
Anyscale
@anyscalecompute
Dec 22, 2023
🔥 Mixtral-8x7B JSON Mode and Function Calling API is now available on Anyscale Endpoints! Empirically, we observed noticeable improvements in response to tool messages by Mixtral MoE, compared @MistralAI 7B. 🚀 👇 Try it out: app.endpoints.anyscale.com
29K
Anyscale
@anyscalecompute
May 28, 2024
Recently, we’ve contributed chunked prefill to @vllm_project, leading to up to 2x speedup for higher QPS regimes! In vLLM, prefilling, which fills the KV cache, and decoding, which outputs new tokens, can interfere with each other, resulting in latency degradation. 1/n
33K
Anyscale
@anyscalecompute
Dec 7, 2021
🎉 We’re thrilled to announce our $100M Series C led by @a16z & Addition + general availability of the Anyscale managed @raydistributed platform! forbes.com/sites/kenrickc… Both are big steps forward in our mission to accelerate the scaling and productionization of #AI apps.
Anyscale
@anyscalecompute
Oct 22, 2025
Today we’re donating Ray to The Linux Foundation under the PyTorch Foundation with PyTorch + vLLM, strengthening the open compute fabric for AI. Ensures long-term neutrality, open governance, and ecosystem alignment. Blog: na2.hubs.ly/H01JydX0 Ray Summit (Nov 3–5, SF):
22K
Anyscale
@anyscalecompute
Jan 16, 2024
Producing ~1B embeddings can take weeks and cost tens of thousands of dollars ($60K with OpenAI in the example below). We are thrilled to partner with @pinecone on the launch of their new serverless offering! Anyscale + Pinecone reduce the cost of computing these embeddings by
18K
Anyscale
@anyscalecompute
Jan 30, 2024
Ready to hear from #RAG experts at @langchain @vercel @pinecone @anyscalecompute and get hands-on with intensive guided trainings? The 2-day RAG Developer Bootcamp is for you! Learn more & register now 👉hubs.ly/Q02j9QmW0 #llm #ml #rag #ai #vectordatabase #ray #pinecone
57K
Anyscale
@anyscalecompute
Apr 18, 2024
🦙 We're excited to host @Meta Llama-3 8b and 70b on Anyscale Endpoints! ➕ Fine-tuning, JSON mode and function calling support coming soon as well! Pricing: - 8B: $0.15 / Million tokens - 70B: $1.00 / Million tokens
10K
Anyscale
@anyscalecompute
May 31, 2023
Our Eng team has been learning a lot about managing and scaling open source LLMs, using some great models from @huggingface . We packaged up what we've learned to help you build/deploy your own! Announcing Aviary: Open Source Multi-LLM Serving
Blog | Anyscale
From anyscale.com
12K