Langfuse is the ???????? ?????????????? ???????? ???????????? ???????????? ????????????????. It helps teams collaboratively develop, monitor, evaluate, and debug AI applications. Langfuse can be ????????-???????????? in minutes and is battle-tested and used in production by thousands of users from YC startups to large companies like Khan Academy or Twilio. Langfuse builds on a proven track record of reliability and performance. Developers can trace any Large Language model or framework using our SDKs for Python and JS/TS, our open API or our native integrations (OpenAI, Langchain, Llama-Index, Vercel AI SDK). Beyond tracing, developers use ???????????????? ???????????? ????????????????????, ?????? ???????? ????????, ?????? ?????????????? ?????? ???????????????????? ?????????????????? to improve the quality of their applications. Product managers can ??????????????, ????????????????, ?????? ?????????? ???? ???????????????? by accessing detailed metrics on costs, latencies, and user feedback in the Langfuse Dashboard. They can bring ???????????? ???? ?????? ???????? by setting up annotation workflows for human labelers to score their application. Langfuse can also be used to ?????????????? ???????????????? ?????????? through security framework and evaluation pipelines. Langfuse enables ??????-?????????????????? ???????? ?????????????? to iterate on prompts and model configurations directly within the Langfuse UI or use the Langfuse Playground for fast prompt testing. Langfuse is ???????? ???????????? and we are proud to have a fantastic community on Github and Discord that provides help and feedback. Do get in touch with us!
Langfuse (YC W23)的外部链接
?? Langfuse Community Hour next week Tuesday!
? Reminder: Langfuse (YC W23) Community Hour on Tuesday! We will hang out together, talk AI, and also go deep on the new features we launched last week. ?? Tuesday, Dec 3rd; 9-10am PT; 6-7pm CEST ?? Sign-up link in comments.
Recap of our second Launch Week ??
?? Quick recap of new features in Langfuse (YC W23): Last week, we released some amazing new features as part of Launch Week #2, focusing on supporting next-generation models and integrating Langfuse deep into your development loop: ?? Full Multi-Modal Support: Langfuse now supports multi-modal traces, including images, audio, and attachments for end-to-end debugging of multi-modal LLM applications. ?? Prompt Experiments: Test prompt versions on datasets with expected outputs and compare results side by side to speed up your feedback loop. ?? LLM-as-a-Judge for Dataset Experiments: We now offer managed LLM-as-a-Judge evaluators to automatically score outputs based on your criteria, catching issues before they reach production. ?? Dataset Experiment Run Comparison View: Our new comparison view allows you to analyze multiple experiment runs side by side to compare performance metrics like latency and cost. ?? New Datasets and Evaluations Documentation: We revamped our documentation to be more thorough and user-friendly, including core data models and end-to-end examples. ?? Prompt Management for Vercel AI SDK: We introduced native integration of Langfuse Prompt Management with the Vercel AI SDK, enabling prompt versioning, usage, and metric monitoring. We're excited to hear your questions and feedback and will continue to ship more features in the coming weeks. Stay tuned!
?? My take on LLM proxies for observability LLM proxies offer useful features like caching, rate limiting, and routing for LLM requests. But: They can introduce a single point of failure in your system and are generally less comprehensive. For production-grade applications, these drawbacks might outweigh the benefits. ??♂? We designed Langfuse (YC W23) to not act as an LLM proxy but as an ???????????????????????? ?????????????????????????? ??????????. This approach not only avoids potential bottlenecks but also lets you track non-LLM-related processes (through application traces). ? There are cases in which using a proxy is advantageous. If you find that a proxy fits your needs, consider using amazing open-source tools like LiteLLM (YC W23), which can be self-hosted and offer a first-class integration with Langfuse. Ultimately, it's essential to weigh the pros and cons carefully. Your application's reliability and scalability depend on it. Choose wisely! ?? Check out our blog post, where Marc Klingen and I explain how LLM proxies work and when they make sense. [?? link in comments:) ]
?? Langfuse (YC W23) is the #1 Open Source LLMOps Tool. ?? We have collected different metrics and are excited to share that Langfuse (YC W23) is the most used open-source LLMOps product out there. From our early days in Y Combinator, we've grown to support thousands of teams across startups, tech companies, and enterprises. A massive thank you to our incredible community—your support and engagement helps drive the overall roadmap and success of Langfuse. Benchmarking Metrics: ? Downloads: Langfuse is the most downloaded OSS LLMOps tool across Python SDK, JS/TS SDK, and Docker images. ? GitHub Stars: We're the most starred open-source LLMOps tool on GitHub and the fastest-growing since our launch in June 2023. ? Community: Proud to have the most active community on GitHub and a thriving Discord server. What does this mean? ?? Large Community: Numerous actively maintained integrations with the latest LLM frameworks (like LangChain, LlamaIndex, and OpenAI SDK). ?? Battle-Tested: Langfuse is used in production by thousands of users and has a proven track record of reliability and performance. ?? Check out our blog post to see the detailed metric comparison (link in comments).
Last day of our Launch Week #2. We released some incredible features in the last couple of days. Here is the full overview:
?? New feature in Langfuse (YC W23): Prompt Experimentation! See them in action here langfuse.com/ph We believe this will have a big impact on how you develop LLM applications with Langfuse. Here's what you can look forward to: ? Test and Evaluate Simultaneously: Experiment with different prompt versions and models on hundreds of dataset items at once. ? Live LLM-as-a-Judge Evaluations: Perform real-time evaluations to see how your prompts and models stack up. ? New Dataset Comparison View: Compare results side-by-side to optimize prompts for your specific use case. ?? Check out our Product Hunt page to learn how you can incorporate Langfuse Prompt Experiments into your development workflow: langfuse.com/ph ?? A huge thank you to Marlies Mayerhofer for this!
Langfuse (YC W23) (OSS LLM Observability/Evals) is built at this speed and team size only because of Supabase, PostHog, ClickHouse, and Cursor. → Langfuse (YC W23) is live on producthunt.com!
?? Introducing Langfuse (YC W23) Prompt Experiments – Move fast on prompts without breaking things! ?? Support our launch on Product Hunt today: langfuse.com/ph Watch the video for an end-to-end walkthrough of how this works
Introducing our native integration with Langfuse (YC W23) for comprehensive LLM observability, tracing, and evaluation. https://lnkd.in/gbYexrA6
Day 4 of Langfuse Launch Week #2 -> All New Documentation for Datasets, Experiments, and Evals
?? Day 4 of Langfuse (YC W23) Launch Week #2: Elevating Developer Experience with New Documentation! At Langfuse, we believe that documentation is product. As part of Day 4 of our Launch Week, we're shining a spotlight on an often overlooked but critical element of great developer experience: documentation. We've completely rebuilt many of our docs to be more thorough and user-friendly than ever before, helping teams accelerate the development of their LLM applications. ?? What's New in Our Documentation? ? Guidance on using Datasets and Evals: Dive deep into the effective evaluation of your LLM applications during development. ? Introduction to Core Data Models: Get acquainted with our foundational data structures—datasets, experiment runs, and scores. ? End-to-End Examples: Common workflows with our Jupyter Notebook examples and see Langfuse in action. ? Visuals and Explanations: Enjoy more GIFs and interactive elements throughout the docs for an engaging learning experience. As our community continues to grow, best-in-class documentation has become essential for teams adopting Langfuse. To celebrate Launch Week #2, we've also summarized all the documentation improvements we've made over the past year. We think it's an interesting read and welcome any feedback you may have! ?? Fun Fact: This update marks the 1000th PR to the Langfuse Docs! Dive into the new documentation and let us know what you think [link in comments]
We are very excited to release multimodality in Langfuse as this has been one of the top requests from the community. ?? Collaborate with us on our public Langfuse roadmap:
?? Launch Week Day 3: Full multi-modal support in Langfuse (YC W23), including images, audio files, and attachments ?????? Back in August, we took the first step by enabling multi-modal traces that reference external files. Since then, expanding this support to include base64 encoded images, audio files, and attachments has been one of the top requests from our community. We heard you loud and clear, and we're excited to finally release this. What's new? ? Enhanced Media Support: Integrate images (PNG, JPG, WEBP), audio files (MPEG, MP3, WAV), and other attachments (PDF, plain text) directly into your Langfuse traces. ? In-UI rendering: Multi-modal content is now rendered inline in the Langfuse UI for images and audio, providing a richer, more interactive experience. ? Custom attachments: Have specific needs? Upload arbitrary media attachments using the new LangfuseMedia class in our SDKs. Check out our changelog post to see how this works under to hood and to get started [link in comments] s/o to Hassieb Pakzad for shipping this in no time! ??