Langfuse (YC W23)

?? New feature in Langfuse (YC W23): Prompt Experimentation! See them in action here langfuse.com/ph We believe this will have a big impact on how you develop LLM applications with Langfuse. Here's what you can look forward to: ? Test and Evaluate Simultaneously: Experiment with different prompt versions and models on hundreds of dataset items at once. ? Live LLM-as-a-Judge Evaluations: Perform real-time evaluations to see how your prompts and models stack up. ? New Dataset Comparison View: Compare results side-by-side to optimize prompts for your specific use case. ?? Check out our Product Hunt page to learn how you can incorporate Langfuse Prompt Experiments into your development workflow: langfuse.com/ph ?? A huge thank you to Marlies Mayerhofer for this!

Langfuse (YC W23) (OSS LLM Observability/Evals) is built at this speed and team size only because of Supabase, PostHog, ClickHouse, and Cursor. → Langfuse (YC W23) is live on producthunt.com!

?? Day 4 of Langfuse (YC W23) Launch Week #2: Elevating Developer Experience with New Documentation! At Langfuse, we believe that documentation is product. As part of Day 4 of our Launch Week, we're shining a spotlight on an often overlooked but critical element of great developer experience: documentation. We've completely rebuilt many of our docs to be more thorough and user-friendly than ever before, helping teams accelerate the development of their LLM applications. ?? What's New in Our Documentation? ? Guidance on using Datasets and Evals: Dive deep into the effective evaluation of your LLM applications during development. ? Introduction to Core Data Models: Get acquainted with our foundational data structures—datasets, experiment runs, and scores. ? End-to-End Examples: Common workflows with our Jupyter Notebook examples and see Langfuse in action. ? Visuals and Explanations: Enjoy more GIFs and interactive elements throughout the docs for an engaging learning experience. As our community continues to grow, best-in-class documentation has become essential for teams adopting Langfuse. To celebrate Launch Week #2, we've also summarized all the documentation improvements we've made over the past year. We think it's an interesting read and welcome any feedback you may have! ?? Fun Fact: This update marks the 1000th PR to the Langfuse Docs! Dive into the new documentation and let us know what you think [link in comments]

?? Launch Week Day 3: Full multi-modal support in Langfuse (YC W23), including images, audio files, and attachments ?????? Back in August, we took the first step by enabling multi-modal traces that reference external files. Since then, expanding this support to include base64 encoded images, audio files, and attachments has been one of the top requests from our community. We heard you loud and clear, and we're excited to finally release this. What's new? ? Enhanced Media Support: Integrate images (PNG, JPG, WEBP), audio files (MPEG, MP3, WAV), and other attachments (PDF, plain text) directly into your Langfuse traces. ? In-UI rendering: Multi-modal content is now rendered inline in the Langfuse UI for images and audio, providing a richer, more interactive experience. ? Custom attachments: Have specific needs? Upload arbitrary media attachments using the new LangfuseMedia class in our SDKs. Check out our changelog post to see how this works under to hood and to get started [link in comments] s/o to Hassieb Pakzad for shipping this in no time! ??

AI Fireside Chat #1 with Marc Klingen from Langfuse (YC W23) Monday evening {Tech: Berlin} & LLM Studios hosted AI Fireside Chat #1 at Merantix AI Campus in Berlin in front of 80+ attendees. ??? As we were not able to fit everyone, we decided to publish the chat as a Podcast. You can listen to it via the link in the comments. Give it a follow, as we now turn this into a monthly series, with AI Fireside Chat #2 already in planning for early December. ?? Sign-Up link is in the comments. Leonard Off Leo Schoberwalter Matteo von Haxthausen Francesco Ricciuti Let's build a proper tech ecosystem in Berlin! ?? Call for partners: If you would like to partner for an upcoming edition, please reach out, it is not possible to set these events up without a great partner network (This time it was Runa Capital ??)

?? Launch Week Day 2: LLM-as-a-judge evaluators for Dataset Experiments! Super excited that Langfuse (YC W23) now helps run model-based evaluations on Dataset Experiments. Building reliable AI applications can feel like playing whack-a-mole with bugs and regressions, all while having way too many ideas of how they could potentially be solved. Our new LLM-as-a-judge evaluators for Datasets helps turn this uncertainty into a more structured engineering process. How this helps: ? Measure Impact Before Deployment: Assess changes before they reach production. ? Detect Regressions Early: Spot and fix issues early in development. ? Compare Dataset Items Across Runs: Track specific items over time with reliable scoring. ? Strengthen Your Test Datasets: Identify gaps between testing and production to make your datasets more robust. ? Create Reliable Feedback Loops: Continuously improve your AI applications with structured feedback. More on this feature in the comments ?? P.S.: Check out our new Dataset Comparison View, for improved eval visibility [link in comments] ?? Kudos to Marlies Mayerhofer for shipping this.

?? Day 1 of Langfuse (YC W23) Launch Week #2: We're adding a side-by-side comparison view for dataset experiments When building LLM applications, structured datasets and evaluations are key to continuously iterate on prompts, model configurations, or other elements of the LLM application. Today, we launch an intuitive way for both ?????????????????? and ??????-?????????????????? users to compare dataset experiment runs side-by-side. ?? What's New? ? Overviews: Get a clear picture of each item in your dataset. ? Summaries: Dive into metrics like latency, cost, and scores, and see the application's output response for each dataset item across selected experiment runs. This means you can compare how different prompts, models, or parameters perform against the same dataset. ??? How to Get Started? 1?? Create a Dataset and populate with Items (e.g. combine sanitized production data and synthetic data) 2?? Try different prompts or application configurations, run them against the dataset as an experiment 3?? Analyze your Dataset experiment runs directly in the Langfuse UI and share the results with your team For code snippets and detailed guides, check out our docs and changelog [links in comments] ?? Thank you, Marlies Mayerhofer, for shipping this in no time ?? P.S. don't miss our Langfuse Town Hall meeting on Wednesday!

Just in time for launch week, we've updated the most important page of every devtool: `/docs` It now features a wholistic/concise/visual overview of the most important Langfuse (YC W23) features. [Link in comments]