Comet

"LLMs and RAGs are powerful, but without a strong evaluation pipeline, you're flying blind." Here at Comet, our team has been obsessing over what it takes to build, test, and ship a GenAI application you can truly rely on. Check out these six critical LLM evaluation steps from founder and CEO?Gideon Mendels, and try them for yourself with our open source eval tool, Opik: https://lnkd.in/dW4D6xMt

Here is a guide on how to evaluate your LLM's performance using OpenAI's Python API. The guide uses Opik, an open-source platform for evaluating, testing, and monitoring LLM applications. Guide: https://lnkd.in/eSKdK884 Use this to: ? Detect hallucinations ? Evaluate RAG applications ? Determine answer relevance ? Measure context recall ? Create and store test cases ? Integrate it with your CI/CD pipeline using Pytest

We're excited to connect with the SF developer community next month alongside Yujian Tang, Pinecone & Komodo AI. RSVP ? https://lu.ma/x8lcld46

?????? Today, our Head of Research, Douglas Blank, (virtually) joined Swarthmore College students to discuss the impact of open-source software, sharing some behind-the-scenes stories from projects like Comet’s Opik and Kangas. Always exciting to see the next generation of OSS developers in the making!

?? Building an LLM application is tough - especially when you don't have a way to confidently test its performance. In this code tutorial, you'll learn how you can leverage Opik’s native OpenAI integration to have more visibility into your LLM workflows: ?? Log traces and spans ?? Manually annotate LLM responses ?? Automatically score responses using Opik's out-of-box evaluation metrics

???? We’ll be joining Intel Software at their Seattle Developer Meetup this Thursday, November 14th! It's shaping up to be a great opportunity to connect with fellow AI developers and participate in workshops focused on AI PCs and MLOps best practices. We'll also be chatting about #Opik, our open-source LLM eval framework that's quickly gaining momentum in the #GenAI space ?? If you're in the area, we’d love for you to join us. You can register here: https://intel.ly/3YDmoCJ #AI #MLOps #LLMEval

?? Opik Weekly Changelog ?? Highlight of the Week: We just launched a Prompt Library! ?? Now you can easily create and manage your prompts in the Opik platform. But there a twist ! We heard time and time again that folks preferred to have their prompts stored in code so built functionality to automatically sync the prompts stored in code with the Opik library, just wrap the prompt with the `Prompt` class and your done. We also released: ? Bedrock integration ? SDK method to search and return spans

? Opik has officially hit 2,000 GitHub Stars! ? For the last several months, the Comet team has been working on our open source LLM evaluation framework, Opik. And since releasing it in September, we’ve been absolutely blown away by the support the ML community has sent our way. Since launching, we’ve had: ?? Dozens of feedback sessions, feature requests, opened issues, and PRs from individual practitioners ?? Integrations from our friends in the open source community, including LiteLLM (YC W23) and Ragas ?? Tutorials, projects, and even an entire course (by the amazing Elvis S., no less) dedicated to running LLM evaluations with Opik And all of this has culminated in 2,039 GitHub Stars, 530 closed PRs, and thousands of users—from hobbyists to researchers to engineers at Fortune 500 companies—using Opik to build their LLM evaluation pipelines. Thank you all so much for your support so far. We're so excited to continue building with you ??

??????Join us and Intel Corporation on November 14th for a hands-on developer workshop in #Seattle! ? Dive into AI application development and MLOps best practices ? Connect with fellow developers over drinks ? Learn about #opik, the open-source LLM eval framework trusted by top GenAI teams ?? Grab your spot: https://lu.ma/er85bzb7

?? Exciting to see that the LLM’s Engineer’s Handbook is now live! This essential guide equips developers with a full framework for building #LLM systems, and we’re proud to see #Opik included as a tool in their LLMOps tutorials.? Big congrats to Paul Iusztin and Maxime Labonne for delivering this incredible resource. It couldn’t have come at a better time for the #GenAI community. ?? Find limited-time discount links in the comments??!