Sentius

Our latest research just landed on arXiv: "Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity" Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, Mikhail Burtsev https://lnkd.in/eM358KDb In the rapidly evolving world of LLMs, one persistent challenge has been to compress long token sequences into compact real-valued vectors efficiently. In our work on RMT (Recurrent Memory Transformer) we post-train a transformer to filter out relevant facts and store them in general-purpose [mem] tokens to solve QA tasks over 10 million tokens. But how much text can actually be stored in a single vector? In this new study we explored compression capabilities of LLMs?with a per-sample optimization procedure and found that vectors with compression ratios up to 1500x are possible—without any loss of information. This compression ratio is x100 to previous work and still about 10-20% of the theoretical capacity. Our findings show that compression limits are determined not by text length, but by the inherent uncertainty (as measured by cross-entropy loss). Some questions that our work raises: - How can we use this untapped capacity to improve reasoning and long-context processing? - What are the computational trade-offs of leveraging per-sample optimization at scale? - In what ways can these insights inform the development of memory-augmented architectures? - What further improvements can be made to bridge the gap between theoretical and practical performance? #MachineLearning #DeepLearning #LanguageModels #AIResearch #LLMs

Nice to see our research on sharing memory between agents to be #1 Paper of the day at Huggingface papers ?? https://lnkd.in/eY7GwjYR

A few weeks ago, we've published a link to our updated website and demo video of Sentius. Today we are thrilled to announce that we've just published our benchmark results compared to Google Project Mariner, H, Emergence, and Anthropic computer use: we reached 94% - more than anyone else in the market! Learn more about Sentius and what we're doing in a short announcement in the Cerebral Valley newsletter. Sign up for our private beta using the link in the overview below! https://lnkd.in/e8k57U8t Warm thanks to our advisors Eugene Izhikevich, Brett Brewer, Bill MacCartney, Anastasia Paushkina, our investors at Techstars and Techstars Columbus Powered by The Ohio State University, to Eric Horvitz, David Vaskevitch, Mary Czerwinski, Yann LeCun and Subbarao Kambhampati for inspiration, to AGI House for hosting us and helping us to unleash our product vision at numerous hackathons in 2024, and to our founders Mikhail Burtsev, Dilyara Zharikova (Baymurzina), and Daniel Kornev for making this breakthrough and more on our road to building Agentic Enterprise. #AI #AIAgents #GenerativeAI #EnterpriseSoftware