Haize Labs

haize labs joins ai grant batch 4. https://lnkd.in/dA4KwT-c

Two new jailbreaking techniques highlight how fragile state-of-the-art LLMs like GPT-4 are. The first from Haize Labs introduces a new attack method called Bijection Learning. The irony? The more advanced the underlying model is, the more successful the attack is. Bijection Learning uses custom-encoded languages to trick models into unsafe responses. Unlike previous jailbreak methods, it dynamically adjusts complexity to exploit small and large models alike without manual intervention. In their tests, even Claude 3.5 Sonnet, a model heavily fine-tuned for safety, was compromised with a staggering 86.3% attack success rate on a challenging dataset (HarmBench). It works by generating a random mapping between characters (a “bijection language”) and training the model to respond in this language. By adjusting the complexity of this mapping—such as changing how many characters map to themselves or using unfamiliar tokens—researchers can fine-tune the attack to bypass safety measures, making it effective even against advanced models. Full post https://lnkd.in/gtRysbTt The second method, by researchers at EPFL, addresses refusal training. The researchers discovered that simply rephrasing harmful requests in the past tense can often bypass safety mechanisms, resulting in an alarmingly high jailbreak success rate. For instance, rephrasing a harmful query in the past tense boosts the success rate to 88% on leading models, including GPT, Claude, and Llama 3. This mainly happens because supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) don’t always generalize well to subtle linguistic changes like tense modification. Neither of these techniques consistently equips the models to handle adversarial or unexpected reformulations, such as rephrasing harmful queries into the past tense. These studies highlight an alarming trend: as AI models become more capable, they also become more vulnerable to sophisticated jailbreaks. Attack #1: Bijection Learning https://lnkd.in/gtRysbTt Attack #2: Refusal training generalization to past tense https://lnkd.in/ggxnNGQ2 — Join thousands of world-class researchers and engineers from Google, Stanford, OpenAI, and Meta staying ahead on AI https://aitidbits.ai

couldn't be more ecstatic to have Constantin Weisser, PhD on the team!

Excited to share a deep dive of the red-teaming research we've been doing for OpenAI at Haize Labs ?? In month before release before the new o1 series, we've been rigorously haizing (stress-testing) the safety and adversarial robustness of their models. Many thanks to Nathan Labenz for having us on the Cognitive Revolution podcast to chat about this work! Listen here for the details about our research and engineering intuitions for automated red-teaming of frontier AI systems. Shoutout especially to Brian Huang and Aidan Ewart for their amazing automated red-teaming efforts ??

??????????????????: Business Insider just dropped a list of 85 of the most promising startups to watch for 2024… ??Some of the largest outcomes (Amazon, Airbnb, Stripe, Uber etc) we’ve seen in ?????????????? ?????????????? have come on the back of hard times. ?? Only the strongest & most resilient teams survive. ?? Business insider asked top venture capitalists at firms including Accel, GV, Founders Fund, Greylock, Khosla Ventures & IVP to name the startups they’re most excited by. ?? Flagging 4 companies from the list I work with that are worth having on the radar?? —— 1?? Gamma - An AI-powered content creation tool for enterprise customers. The tool enables users to create presentations, websites, and documents quickly. ?? Founders: Grant Lee, Jon Noronha, James Fox ?? Funding: $21.5Mn ?? Investors: Accel, LocalGlobe, Script Capital ?? Why it’s on the list? Gamma has already amassed 20Mn+ users with 60M+ gammas created, and is profitable. It is reinventing creation of websites, presentations & documents. —— 2?? Sema4.ai - It is building enterprise AI agents that can reason, collaborate, and act. ?? Founders: Antti Karjalainen, Paul Codding, Sudhir Menon, Ram Venkatesh ?? Funding: $54Mn ?? Investors: Benchmark, Mayfield Fund, Canvas Ventures ?? Why it’s on the list? The company is developing AI agents that can move beyond simple repetitive tasks and actually solve real-world problems, taking into account the unique context of the organization and working seamlessly with existing teams. —— 3?? Mutiny - An account-based AI platform that helps companies unify sales and marketing to generate pipeline and revenue from their target accounts at scale. ?? Founders: Jaleh Rezaei, Nikhil Mathew ?? Funding: $72Mn ?? Investors: Sequoia, Tiger, Insight, Cowboy Ventures ?? Why it’s on the list? Mutiny leverages AI to help B2B companies generate pipeline & revenue from their target accounts through AI-powered personalized experiences, 1:1 microsites & account intelligence - more important than ever in the current software consolidation cycle / market & budget environment. —— 4?? Haize Labs - Automatic stress testing of large language models. ?? Founders: Leonard Tang, Steve Li ?? Why it’s on the list? Haize promises to "robustify" any large language model through automated red-teaming that continuously stress tests and identifies vulnerabilities. As models evolve, the question of how to make sure they're secure becomes increasingly difficult to answer. —— ?? The biggest themes on the list this year? ?? Data infrastructure ?? Security ?? Personalised agents —— ??It’s helpful for ???????????? to keep on top of lists like these to track & source new opportunities. ?? to the FULL list in comments. ?? Any companies not on the list that should be? Tag them in comments. ?? PS- If you enjoyed this & found it valuable, ????follow me Akhil Paul for more! #startups #venturecapital #investing #technology #angelinvesting

A consistent theme in discussions I've had with AI Application founders is a request for red teaming solutions. Combined with regulatory frameworks like the EU AI Act adding further pressure, there's more attention going towards both human and model-based ways of scaling red teaming of LLMs where the immense combinatorial space is difficult to address. Protect AI acquired SydeLabs to expand their AI security suite last week, but there are others like Haize Labs, Promptfoo and more devising innovative new ways of scaling automated red teaming without the associated trade off in model performance. https://lnkd.in/en5_9jMP

Haize Labs Introduced Sphynx: A Cutting-Edge Solution for AI Hallucination Detection with Dynamic Testing and Fuzzing Techniques Haize Labs has recently introduced?Sphynx, an innovative tool designed to address the persistent challenge of hallucination in AI models. In this context, hallucinations refer to instances where language models generate incorrect or nonsensical outputs, which can be problematic in various applications. The introduction of Sphynx aims to enhance the robustness and reliability of hallucination detection models through dynamic testing and fuzzing techniques. Hallucinations represent a significant issue in large language models (LLMs). These models can sometimes produce inaccurate or irrelevant outputs despite their impressive capabilities. This undermines their utility and poses risks in critical applications where accuracy is paramount. Traditional approaches to mitigate this problem have involved training separate LLMs to detect hallucinations. However, these detection models are not immune to the issue they are meant to resolve. This paradox raises crucial questions about their reliability and the necessity for more robust testing methods. Haize Labs proposes a novel “haizing” approach involving fuzz-testing hallucination detection models to uncover their vulnerabilities. The idea is to intentionally induce conditions that might lead these models to fail, thereby identifying their weak points. This method ensures that detection models are theoretically sound and practically robust against various adversarial scenarios..... Read our full take on Sphynx: https://lnkd.in/gQjNJ8Ww Check out their GitHub Page: https://lnkd.in/gZ_fwhEe Haize Labs Leonard Tang

introducing Sphynx - the leading hallucination haizing algorithm????? - breaks SOTA hallucination detection models (HDM) - open source, open data - surfaces critical hallucinations in high-stakes domains - enables adversarial training for more robust hallucination detection the failure cases that Sphynx produces can be used as adversarial training data to make HDMs more robust. rather than wait for HDMs to fail in the wild, Sphynx forces these failures to happen in development. only by hazing can you achieve truly reliable HDMs and AI systems check out Sphynx here: https://lnkd.in/gZ_fwhEe twitter thread:?https://lnkd.in/gXi5kRkN

thanks for believing in us Ben ??

Today, we emerged out of stealth to tackle the longstanding problem of AI reliability. In the era of grossly excessive AI hype and demoware, it is high time that someone recalibrated and revisited the difficult, unsexy, underlying problem that everybody is avoiding -- the AI reliability and safety problem. We have never really believed in LLMs, or neural nets, or deep learning, favoring PPLs, graphical models, and interpretable architectures that could admit true reasoning. In the murky depths of LLM land, we lose structure and control. And this is how you end up with scenarios like Google AI search telling you to eat rocks, or the Chai chatbot inducing a man's suicide, or the Air Canada lawsuit, just to name a few... This is why we need to haize models. that is -- we need to rigorously test them to discover all of their vulnerabilities, failure modes, and "gotchas" before they get deployed in production. Today we showcased one particular application of haizing to safety red-teaming (https://lnkd.in/eGWnevbb); that is, eliciting harmful behaviors from safety-trained models. This is just a start (albeit a very illustrative start). Preventing misuse of AI systems is an absolute must given the pace at which models are accelerating. But we're not stopping here. In the very near future, will be the underlying robustness and safety layer for any model, in any setting, for any use case. Hope you are excited join us for the long and exciting journey ahead.