Guardrails AI

gRPC and/or websocket implementations will become really important to agentic workflows once LLMs get quick enough. Bidirectional communication between agents will become a core assumption for agent-based workflows. We've started working on a protobuf endpoint for streamed validation. When looking around, we've found that most of the main model providers don't have gRPC endpoints. Our first cut will work as a parallel to the HTTP endpoints. It'll be interesting to see how openai etc will eventually deal with gRPC and inform the broader adopted protobuf design. Example use case: Imagine a multi-agent system where a research agent and a writing agent collaborate in real-time. gRPC would allow for efficient, typed data exchange, enabling the research agent to stream relevant info to the writer as it's discovered. Another example: In a customer service scenario, WebSockets could enable an AI agent to provide real-time updates to the user while simultaneously querying multiple knowledge bases and APIs, creating a more responsive experience. bidi comms have their challenges too. Increased complexity in system design, potential compatibility issues with existing infrastructure, and the need for specialized knowledge among developers are hurdles to consider. The last is likely why the major providers haven’t invested heavily in this yet. There's also the question of standardization. As more providers potentially adopt gRPC, ensuring interoperability between different systems could become a significant challenge for the industry.

Validating LLM responses may seem simple on the surface, but there's a world of complexity lurking beneath. While a basic implementation might seem straightforward, each aspect - from reusability to performance, from streaming to structured data - adds layers of consideration. We've tackled these challenges head-on. Our open-source framework offers a standard for validation, supports all major LLMs, enables real-time fixes, and even provides monitoring. It's not just about checking responses; it's about elevating your entire AI app. https://lnkd.in/d2m4DR8Q

Release day! Guardrails v0.5.12 is out! This one’s mainly bug fixes and usability improvements. If all goes well, this will be the last 0.5.x update, and we’ll be moving over to 0.6.0 next week. The main usability updates center on presenting helpful warnings more often and guard initialization patterns. As far as bug fixes go, we’ve taken a deeper look at async and solved a high prio bug with the server. Our wonderful discord community was quick to find and point out these bugs. See release notes here https://lnkd.in/g59xu253

The vast majority of Guards either log, reask, or implement automatic fixes when validators fail. But there are a growing number of Guards that use custom on_fail actions. custom on_fail actions are useful when you want to modify some of the output text. We wrote a guide for using on_fail actions, and custom on_fail actions specifically, here ???? https://lnkd.in/gmr6Qe3N

We’re renaming and deprecating a few fields: from_pydantic → for_pydantic from_string → for_string from_rail → for_rail from_rail_string → for_rail_string Why? from_ really does not describe well what these functions do, and really, the guard is created FOR validating those different structures.

We see a ton of custom LLMs being used. A double-digit percentage of calls passed through guardrails use some kind of custom LLM or wrapper. Our current approach is lacking. We expect a function that takes a string and returns a string response While at first glance this might look fine, it’s missing a few key features which end up being wired through at different stages. For example, we miss out on temperature, message history, reask prompt, and more. This is really the same problem as we originally had with all callables and finding a unified interface that’s easy to operate but still feature rich. It’s also why we began using LiteLLM to internally handle LLM requests. We’re going to bring custom callable interfaces as close to the standard LiteLLM interface as possible. For a quick way to uplevel your custom callable, simply add a few params to the header that you can expect a guard to pass through - prompt - instruction - msg_history - temperature ???? Full guide on writing custom LLM wrappers https://lnkd.in/gDVxX7Ru

Guardrails v0.5.11 is out! This week, we added information to spans that show up in telemetry. We also solved a few high prio bugs, including the handling of certain custom validators. Full changelog here: https://lnkd.in/gbciudQk

Hosting validation models opens up interesting usecases, like streaming validated LLM outputs. These models run on GPU based instances that perform much better than running stuff on our M2 macs. This opens up possibilities, like creating validated, streamed chat experiences. In our video we stream back results from an LLM request validate live (while processing the stream) to scrub out PII and profanity. While a little latency is added, it’s still really quick. Streaming validations is difficult to do, we wrote a small paper on it a bit ago: https://lnkd.in/gzk2gea9 These are the validators that we currently host, check them out on https://lnkd.in/gxYvyDfm - Toxic Language - Detect PII - NSFW Text - Competitor Check - Restrict to Topic Let us know if you’d like to see more hosted validators!

Huge updates to concurrency in Guardrails! We’re able to save tons of latency by switching over to async actions all through our validation loop. This allows us to default to concurrent execution of validators within a guard execution. Take a look at the before and after latencies, we’ve saved hundreds of ms! The important thing to notice is that the 3 validators all fire off at the same time! You will notice something weird here. Shorter running validators appear to take longer. That’s not totally true, it’s a limitation of the level at which we’ve implemented spans. We’ll fix that soon. You can see the code we used to generate these numbers. The long term here is to switch to a true event driven architecture that can scale across workers horizontally. Learn more about our approach here. https://lnkd.in/gviX27mA

I’ve been chatting with enterprises building apps around LLMs, and I believe there are five core principles that can make or break the user interface experience — especially if you’re seeing user drop-off. 1?? Interactive – LLMs won’t always get it right on the first try. Your app should be designed for an interactive, two-way dialogue to refine responses and build user trust. 2?? Onboarding – Users need a clear starting point. Offering guided workflows or prompts can help them explore what the app can do without feeling overwhelmed. 3?? Context – LLMs need context and access to the right data sources to provide meaningful, accurate responses. Context is key to relevance. 4?? Response Structure – Asking LLMs to return information in specific formats ensures results are clear, actionable, and tailored to your users’ needs. 5?? Scope – Narrowing the scope of interactions helps build trust. Focusing on tasks where the LLM excels will create a more reliable user experience. Finally, systematic, iterative testing—both broad and deep—is critical to identify where the app does well and where improvements are needed. What other principles have you found essential when designing user interfaces for LLM-powered applications?