Youssef Mrini的动态

Kubeflow, Ray, KubeRay,mlflow,Flyte,Argo,JupyterHub,Kserve and many more…. As I am exploring the landscape for OpenSource MLops and ML Lifecycle management, distributed computing and s more… the list just gets longer and the data is showing that different people build different platforms and combination of tools based on very specific needs. If you’re running on OSS, What are your tools ? What is your sauce for running your ML Data Process ? I am curious to know .. #mlops #oss #genai #aiml

In case you missed Jeff Chou's talk at Databricks Data + AI conference, check it out below. This talk covers best practices to managing jobs at scale while lowering compute costs . It highlights common pitfalls and effective strategies for managing Databricks Jobs clusters, SQL warehouses, and ML workloads. ?? Compare the cost and performance of serverless vs. optimized classic compute clusters. ?? Get tried and true optimization tactics based on years of research into high-frequency computing? ?? See real-world examples of companies saving up to 65% on their Databricks compute spend Watch it now and let us know what you think in the comments ?? https://lnkd.in/g8YMZYFQ

Pipeline Optimization: Lessons You Only Learn by Doing When I started, I thought a working pipeline was all that mattered. But after nearly two years in the field, I learned that performance is just as important. I had a pipeline pulling from multiple sources, and it took forever to run. That’s when I dug into optimization. One key technique? Parallelism and partitioning. Tuning parallel copies and splitting large datasets effectively cut the runtime in half. It took some time to get right, but the payoff was huge. If your pipeline’s dragging, ask yourself: can I optimize the data partitions? #DataEngineering #Azure #Optimization

Prototype to Production: Crafting Applications with #AzureCosmosDB - Part 1 This is part one of the three-episode series that dives deep into the challenges faced when moving applications from prototype to production. https://lnkd.in/ePkypP6N

If you need to efficiently process huge datasets for HPC, machine learning or other applications, watch this video to see how FSx for Lustre (FSxL) can save you time and money by seamlessly integrating S3 object storage with a high performance filesystem. #AWS #machinelearning #bigdata #HPC https://buff.ly/4d9XhN7 Marcos Perez Seoane Laura Shepard Brendan Bouffler

Lots of interesting announcements on re:invent this year for data & AI engineering, with SageMaker and Iceberg on the spotlight. I've been following SageMaker for 4+ years now. It was lackluster for a long time, but lately grown to be actually enjoyable: much nicer IDE, well integrated, easy accessibility etc. It makes sense that AWS is betting on SM to take on Databricks, Snowflake, Fabric. I wonder why it took them so long, but with native Iceberg in S3 now, they may be able to attract the engineers. If they get pricing right also, this might just put AWS back on the data platform map.

Watching Gradient drop a Fortune 100's Databricks jobs costs by >50% in a few days is crazy. Everyone was shocked. ?? ?? Best part was - it was all automatic. ?? ?? While everyone was in meetings, building other pipelines, or just grabbing lunch - Gradient was at work dropping costs with no overhead at all. ?? Gradient literally prints money for our customers ? What's even crazier is Gradient isn't even done yet, it's still early on its journey in optimization The power of AI/ML in the cloud infrastructure world is, in my opinion, the killer app. And it's amazing to see this new technology in action helping real people. But don't take my word for it. Try it out for yourself. Book a demo here if you want help automating and managing your Databricks jobs Sync Computing

Prototype to Production: Crafting Applications with #AzureCosmosDB - Part 1 Watch part one of the three-episode series that dives deep into the challenges faced when moving applications from prototype to production. https://lnkd.in/eStUYRN4

I'll be at AWS re:Invent 2024! Would love to connect with developers, ML practitioners or anyone else interested in: - Vector Quantization - Hybrid search architectures - Embedding model capabilities - Search evaluation metrics - Configuring HNSW indexes ?? DM me if you're attending and want to dive into the technical nitty-gritty of vector search systems! #AWSreinvent #MachineLearning #MLOps #VectorSearch #AIEngineering"

Great close out summary from Andy! Lots of excitement and pride in the advancements in Sagemaker AI, Q Transformations (aka TDK or "Tech Debt Killer", so cool) and its expanded capabilities around Documentation (old, dead code, what does it do?), Unit Tests (create unit tests for end to end projects) and Code Review (utilizing agents and automation as a first line of defense for dev teams).