Anyscale

ByteScale is a new LLM training framework from ByteDance - Evaluated on 7B to 141B param models - 256K to 2048K context lengths - 12,000 GPUs - Optimized for mixed long and short sequences The crux of it is a much more dynamic parallelism strategy (as opposed to a static mesh) to account for heterogeneity in sequence length. They call this strategy Hybrid Data Parallelism (HDP), which combines regular data parallelism with context parallelism in a dynamic manner. Their data loading strategy is very network and CPU-memory intensive and requires global coordination across workers (as opposed to each worker doing its own thing). They use Ray actor for this coordination. There are - Servers to fetch and preprocess raw data from HDFS and generate metadata - A scheduler to collect global metadata from all servers, figure out the the loading plan, and broadcast the plan to clients - Clients (on GPUs), which read the partial data from servers based on the loading plan

Attentive is creating a better, personalized shopping experience for consumers – built on a foundation of scalable ML infrastructure. ?? Check out how they're taking personalized marketing to the next level with Anyscale ?? https://lnkd.in/eqnWB3_A "Before Anyscale, we didn’t have the ability to consolidate our data into a single model because we simply couldn’t process it all. With Anyscale, we were able to unify it into one model and reduce the cost by 99% while increasing the data volume for it by 12X." -Christian Stano, Engineering Manager for the ML Platform team at Attentive

We are excited to invite you to our upcoming meet-up on March 27th, co- hosted with Daft, where we’ll be collaborating on scaling data processing and ML training with Daft + Ray. ?? Hear from Daft Software Engineer, Desmond Cheong and Anyscale Product Manager, Ricardo Decal on developing AI applications from local production, connect with the AI community, and grab some pizza ???? Don’t miss it! Sign up now ?? https://lu.ma/u5p41kte?

Anyscale's Co-founder Robert Nishihara will be speaking at the Human[X] Conference in Las Vegas on Monday, 3/10, with other experts discussing the future of AI infrastructure--breakthroughs in custom hardware, scalable systems, and real-time data processing that are key to unlocking AI’s full potential and keeping businesses competitive. ?? There's still time to get your tickets: https://lnkd.in/gJdDqTEf

Exciting to see DeepSeek AI using Ray and Arrow in their smallpond release today! - Smallpond targets high performance data processing - It provides a high-level dataframe API - Targets petabyte-level scaling Data processing is essential for training data prep. Typically includes a lot of filtering (to increase data quality) and annotation (to extract structured information) as well as deduplication (there is lots of redundancy on the internet, and you want to control the data mixture and not leave it to chance). The challenges around training data prep only grow when you include multimodal data, e.g., images, video, audio, and start running a ton of transcription, captioning, not to mention a lot of synthetic data workloads. https://lnkd.in/gGsKMjwJ

uv is one of the most important projects that's been created recently, and the team at Astral is very very good. It's hard to overstate how important getting dependency management right is for iteration velocity. My favorite thing about uv is that when installing a new Python dependency, instead of modifying the overall global Python environment, you can treat the dependency as scoped locally just to the execution of a particular script. It's very clean.

Python dependency management has been a longstanding challenge facing AI teams. The uv package manager, built by the team at Astral, goes a long long way toward putting that problem to rest, at least for code running on a single machine. The challenge is even bigger in the cluster setting, where developers must iterate rapidly on code and dependencies while keeping everything synced across a potentially autoscaling pool of compute. This integration with Ray extends the benefits of uv to distributed Python applications running on a cluster. It's quite simple. If main.py is your Ray driver script, you can manage the dependencies for the entire Ray application (the driver and all of the Ray worker processes) by running: "uv run <args> main.py". https://lnkd.in/g2-FadxB

Awesome turnout for Anyscale's Cody Yu presentation at the vLLM meetup—nearly 300 people joined to hear about the vLLM roadmap and our team's release of new LLM APIs in Ray Data and Ray Serve. The new batch inference APIs seamlessly integrate vLLM, improving both speed and scalability. See the APIs here: Ray Data + LLMs- https://lnkd.in/gJ_Ucc4W Ray Serve for LLMs- https://lnkd.in/gi2TVSAz

We love seeing the work Cambridge Mobile Telematics is doing with Ray.?With sensor data from millions of IoT devices, they scale deep learning training workloads using large datasets with Distributed Data Parallelism (DDP).?To scale up to even larger models, they rely on Ray for Full Sharded Data Parallel (FSDP) training.?All of this runs on their Amazon EKS cluster. Read on to learn about EKSRay??? https://lnkd.in/eSby4GQ7

?Anyscale Webinar Alert! - THIS WEDNESDAY? Join us tomorrow, Wed Feb 26 for an Introduction to Anyscale and Ray AI Libraries, a webinar designed to help you get started with Ray’s AI libraries for distributed machine learning. Our training instructors will walk you through: ?? How Anyscale addresses today’s AI challenges ?? An overview of Ray AI Libraries (Data, Train, Tune, Serve) ?? A full ML workflow covering all MLOps stages Don't miss this chance to learn directly from experts and see how Ray AI Libraries can supercharge your ML workflows. ?? Register now: https://lnkd.in/dbK8PFw5