AI Newsletter

Another week - another cool updates in the world of AI!

?? Grok-2 Release

xAI has released Grok-2, a large language model built into X, for premium subscribers. Grok-2 has been tested against top models like GPT-4 Turbo and Claude 3.5, demonstrating impressive capabilities in text and image generation, with the added distinction of being completely uncensored. This release has sparked significant discussion, particularly around its affordability at $8 per month and its innovative use of the Flux 1 model for image generation. Сompared to other premium AI art generators, Grok-2 offers a cost-effective alternative, delivering high-quality images for X premium members. While it still faces competition from established models like MidJourney and DALL-E 3, Grok-2’s uncensored and versatile features are gaining attention.

?? Anthropic Introduces Prompt Caching with Claude

Anthropic has shown a new prompt caching feature for Claude, promising faster and more cost-effective AI interactions. This update is particularly beneficial for developers, with potential savings of up to 90% on inference costs and a 79% improvement in response times.

?? Google Pixel 9: AI at the Core of New Features

Google’s recent event showcased the AI-driven capabilities of its new Pixel 9 phone. The Pixel 9 debuts Gemini Nano, a large language model optimized for mobile, offering users faster and smarter AI interactions. The phone also includes AI-enhanced photography features, such as advanced zoom improvements and a new “Add Me” feature that seamlessly integrates the photographer into group shots.

?? Google AI Updates: Enhanced Search Overviews and Imagen 3 Access

Google continues to enhance its AI offerings, with updates to AI overviews in search and the release of Imagen 3, a new image generation model. The AI overviews now include a ‘simpler’ button for more digestible information, while Imagen 3 offers U.S. users high-quality image generation through Google’s Image FX platform.

?? Free DALL-E 3 Image Generation for ChatGPT Users

OpenAI has made DALL-E 3 image generation available to users on the free plan of ChatGPT, although limited to two images per day. This move is part of OpenAI’s strategy to democratize access to powerful AI tools, allowing a broader audience to experience and experiment with AI-driven creativity. The introduction of free DALL-E 3 access adds significant value to ChatGPT.

?? OpenAI's ChatGPT-4o Reclaims #1 Spot in Chatbot Arena

OpenAI's latest ChatGPT-4o (20240808) has reclaimed the top position in the Chatbot Arena, surpassing Google’s Gemini-1.5-Pro-Exp with a score of 1314. The model excels in technical areas like Coding, gaining 30+ points over its predecessor, and dominates categories such as Instruction-Following and Hard Prompts.

Top Rankings:

• Overall: #1
• Coding: #1
• Hard Prompts: #1
• Instruction-Following: #1

?? OpenAI’s SWE-Bench Benchmark Raises the Bar for AI Evaluation

In response to the rapid advancements in large language models, OpenAI has introduced a new benchmark called SWE-Bench. This benchmark is designed to provide a more accurate and future-proof method of evaluating AI performance, addressing the limitations of older benchmarks that could no longer distinguish between top-performing models. SWE-Bench aims to set a new standard in the AI industry, ensuring that as models continue to evolve, their capabilities are rigorously and fairly assessed.

?? Hermes 3: An Open-Source Challenger in the LLM Space

Noose Research has announced Hermes 3, an open-source large language model available in both 870 and 405 billion parameter versions. Designed to be less censored and more customizable than its competitors, Hermes 3 is positioned as a serious alternative to models like LLaMA 3.1

?? SAG-AFTRA and Narrative Ink Deal for AI Voice Replication

SAG-AFTRA has reached a deal with Narrative, a company specializing in AI voice replication, to ensure that voice actors are fairly compensated when their voices are used by AI. The agreement includes the creation of a platform where actors can train AI models with their voices and set royalty rates for their use.

?? Universal Music Group and Meta Expand AI Partnership

Universal Music Group (UMG) and Meta have expanded their partnership to focus on AI-driven content, particularly in the realm of music. This new deal aims to ensure that artists and songwriters are fairly compensated when their work is used in AI-generated content, such as short-form videos on platforms like Instagram.

?? Runway Gen-3 Turbo: A Leap Forward

Runway has introduced the Gen-3 Turbo update, significantly increasing the speed of its image-to-video generation process. The new Turbo mode is seven times faster than previous versions, allowing users to create high-quality videos more efficiently.

New Noteworthy Papers??

LONGWRITER: UNLEASHING 10,000+ WORD GENERATION FROM LONG CONTEXT LLMS

Abstract: Recent advancements in long context large language models (LLMs) have significantly expanded their ability to process inputs up to 100,000 tokens. However, these models struggle to generate outputs longer than 2,000 words, a limitation rooted in the characteristics of the Supervised Fine-Tuning (SFT) datasets. To address this, the paper introduces AgentWrite, a pipeline that decomposes ultra-long generation tasks into subtasks, enabling LLMs to produce coherent outputs exceeding 20,000 words. By creating the LongWriter-6k dataset and incorporating it into model training, the authors successfully scale output lengths to over 10,000 words. The paper also introduces LongBench-Write, a benchmark for evaluating ultra-long generation capabilities, with their 9B parameter model achieving state-of-the-art performance.

Key Highlights:

• Output Limitation Insight: The model’s maximum generation length is capped by the longest outputs in its SFT dataset, despite exposure to longer sequences during pre-training.
• AgentWrite Pipeline: A novel approach that breaks down long tasks into manageable subtasks, allowing existing LLMs to generate outputs exceeding 20,000 words.
• LongWriter-6k Dataset: A new dataset specifically designed to train models for ultra-long output generation, ranging from 2,000 to 32,000 words.
• LongBench-Write: A benchmark developed to evaluate models’ capabilities in generating ultra-long text, where the authors' model outperformed larger proprietary models.
• Challenges with Instruction Backtranslation: The paper discusses the shortcomings of using instruction backtranslation for generating long-output SFT data, highlighting the need for higher quality long texts and better alignment with real user instructions.

Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models

Abstract

Language models (LMs) have shown promise in various decision-making tasks but are limited by simple acting processes. The paper introduces Language Agent Tree Search (LATS), a framework that integrates reasoning, acting, and planning within LMs. By combining Monte Carlo Tree Search with LMs' in-context learning and incorporating environment-based feedback, LATS enhances exploration and decision-making. The approach shows significant improvements, achieving state-of-the-art results in programming accuracy and competitive performance in web navigation, while maintaining or improving reasoning capabilities.

Key Highlights

• LATS Framework: Integrates reasoning, acting, and planning using Monte Carlo Tree Search and LM-powered value functions.
• External Feedback: Incorporates an environment for adaptive problem-solving, improving upon current methods.
• Experimental Results: Achieves a 92.7% pass@1 accuracy for programming tasks and an average score of 75.9 in web navigation, comparable to gradient-based methods.
• Limitations: Higher computational cost and reliance on environments that allow state reversion. Potential for improved efficiency with time and research.
• Future Directions: Aims to scale LATS to complex environments and multi-agent systems, with ongoing improvements to efficiency and applicability.

Impact Statement

LATS enhances LM performance through iterative decision-making and reflection, which may improve interpretability and alignment but also raises security risks. Further research is encouraged to address these concerns and optimize the framework's efficiency.

EfficientRAG: Efficient Retriever for Multi-Hop Question Answering

Abstract

EfficientRAG introduces a novel retriever designed to handle multi-hop question answering more efficiently. Unlike traditional Retrieval-Augmented Generation (RAG) methods that rely on multiple calls to large language models (LLMs), EfficientRAG generates queries iteratively without requiring LLM calls at each step. This approach filters out irrelevant information and has been shown to outperform existing RAG methods across three open-domain multi-hop question-answering datasets.

Key Highlights

• EfficientRAG Framework: Generates queries iteratively and filters irrelevant information, reducing dependency on LLMs.
• Performance: Surpasses existing RAG methods in multi-hop question answering, showing improved recall and effectiveness on three benchmark datasets.
• Limitations: The framework has not been tested with larger LLMs due to resource constraints and is primarily evaluated on open-domain datasets, which may not fully represent in-domain

RAGCHECKER: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

Abstract

RAGCHECKER introduces a comprehensive evaluation framework for Retrieval-Augmented Generation (RAG) systems. Addressing the challenges of evaluating complex, modular RAG systems, RAGCHECKER provides a suite of diagnostic metrics for both retrieval and generation modules. Unlike existing evaluation methods, which often lack granularity and reliability, RAGCHECKER employs claim-level entailment checking to offer fine-grained insights into system performance. The framework has been validated through meta-evaluation, showing superior correlation with human judgments compared to other metrics. Extensive experiments with eight RAG systems across ten domains demonstrate RAGCHECKER's ability to reveal meaningful patterns and trade-offs in RAG architectures, guiding improvements and enhancing system development.

Key Highlights

• Framework Introduction: RAGCHECKER is designed to evaluate both retrieval and generation processes in RAG systems, offering detailed diagnostic metrics.
• Evaluation Methodology: Utilizes claim-level entailment checking for a fine-grained analysis, improving on traditional response-level metrics.
• Performance Validation: Meta-evaluation confirms that RAGCHECKER correlates better with human judgments compared to existing evaluation methods.
• Experimental Findings: Extensive testing of eight RAG systems across ten domains uncovers insights into retrieval improvements, noise introduction, and model performance.
• Contributions: The paper highlights the introduction of new diagnostic metrics, validation of framework effectiveness, and the ability to provide actionable insights for enhancing RAG systems.

Limitations and Future Directions

• Evaluation Scope: While RAGCHECKER provides a detailed analysis, its application is currently limited to the evaluation of RAG systems and may require adaptation for broader use cases.
• Scalability: The framework's effectiveness in evaluating larger or more complex RAG systems and its adaptability to diverse domains are areas for further exploration.

MUTUAL REASONING MAKES SMALLER LLMS STRONGER PROBLEM-SOLVERS

Abstract

rStar introduces a novel self-play mutual reasoning approach designed to enhance the reasoning capabilities of small language models (SLMs) without the need for fine-tuning or superior models. This method employs a two-step process involving mutual generation and discrimination. Initially, a target SLM uses Monte Carlo Tree Search (MCTS) augmented with human-like reasoning actions to create high-quality reasoning trajectories. Another SLM, with comparable capabilities, acts as a discriminator to validate these trajectories. Trajectories that receive mutual agreement are deemed more reliable. Experiments across five SLMs demonstrate that rStar significantly improves performance on various reasoning tasks, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA, with notable accuracy gains.

Key Highlights

• Methodology: rStar uses a generator-discriminator self-play mechanism to enhance SLM reasoning capabilities during inference without fine-tuning.
• Performance Improvement: Demonstrates significant accuracy improvements on diverse reasoning benchmarks, including a boost from 12.51% to 63.91% for GSM8K with LLaMA2-7B and from 74.53% to 91.13% with LLaMA3-8BInstruct.
• Experimental Validation: Effective across five SLMs and a range of reasoning tasks, showing substantial performance gains over traditional multi-round prompting and self-improvement methods.
• Contributions: Highlights the potential of rStar to advance SLM reasoning abilities, revealing strong capabilities in SLMs even before specialized fine-tuning.

Limitations and Future Directions

• Scalability: The approach’s effectiveness with larger models or in more complex domains remains to be fully explored.
• Generalization: Further studies are needed to assess rStar's applicability to a broader range of tasks and domains beyond those tested.

A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?

Abstract

Translating natural language (NL) queries into SQL queries (NL2SQL) can greatly ease access to relational databases and support various commercial applications. The advent of Large Language Models (LLMs) has significantly improved NL2SQL performance. This survey offers a comprehensive review of LLM-powered NL2SQL techniques, exploring the lifecycle of NL2SQL from four key aspects: (1) Model: Techniques addressing NL ambiguity, under-specification, and mapping NL to database schemas and instances; (2) Data: Strategies for collecting training data, synthesizing data to address training data scarcity, and developing NL2SQL benchmarks; (3) Evaluation: Methods for assessing NL2SQL approaches from multiple perspectives using diverse metrics; and (4) Error Analysis: Identifying and analyzing errors to enhance NL2SQL models. The survey also provides guidelines for developing NL2SQL solutions and discusses ongoing research challenges and future directions in the era of LLMs.

Key Highlights

• Model Techniques: Examines methods for resolving NL ambiguities and aligning queries with database schemas.
• Data Collection: Discusses approaches to training data collection, synthesis, and the creation of benchmarks.
• Evaluation Metrics: Reviews various metrics and methodologies for assessing NL2SQL performance.
• Error Analysis: Analyzes common errors to improve model performance and provide actionable insights.
• Guidelines: Offers practical advice for developing effective NL2SQL solutions.
• Future Challenges: Highlights current research challenges and open problems in NL2SQL using LLMs.

FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework

Abstract

FruitNeRF is a novel framework for fruit counting that utilizes advanced view synthesis methods to perform accurate 3D fruit counting from a set of unordered images captured by a monocular camera. The framework employs a foundation model to generate binary segmentation masks for various fruit types, regardless of their type. By combining RGB and semantic information, FruitNeRF trains a semantic neural radiance field to produce fruit-only point clouds through uniform volume sampling. This approach, leveraging neural radiance fields, enhances fruit counting accuracy by addressing issues such as double counting and irrelevant fruit detection. Evaluations using both real-world and synthetic datasets demonstrate that FruitNeRF achieves high accuracy, with F1-scores of 0.95 on synthetic data and 0.79 on the Fuji benchmark dataset.

Key Highlights

• Unified Framework: Utilizes a semantic neural radiance field for fruit counting, independent of fruit type.
• Foundation Model Integration: Employs a foundation model for binary segmentation, avoiding the need for extensive annotations.
• 3D Reconstruction: Converts 2D images into accurate 3D fruit counts, addressing challenges like double counting.
• Performance: Achieves F1-scores of 0.95 on synthetic datasets and 0.79 on the Fuji benchmark, demonstrating robustness across various fruit types and real-world scenarios.
• Scalability: Effective with only 40 images per tree at a resolution of 512 px × 512 px.

Results

• Synthetic Dataset: F1-score of 0.95 with ground truth masks and 0.88 with masks generated by SAM.
• Real-World Dataset: Detection rate exceeding 89% on self-recorded apple dataset.
• Benchmark Dataset: F1-score of 0.79 on the Fuji benchmark.

Future Research Directions

• Clustering Improvement: Enhance the sensitivity of clustering to hyper-parameter tuning.
• Time-Series Images: Explore the use of time-series images to reduce the number of required images and achieve real-time performance.
• Soft Fruits: Extend research to soft fruits like strawberries and raspberries.
• Orchard Rows: Investigate the framework's application to entire orchard rows using online pose estimation methods such as SLAM.
• Simulation Environments: Use simulation environments like SLAM in Blender for refining detection rates.

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Abstract

The xGen-MM framework, also known as BLIP-3, is a comprehensive approach for developing Large Multimodal Models (LMMs). This framework builds on the Salesforce xGen initiative and incorporates meticulously curated datasets, a training recipe, model architectures, and a suite of resulting LMMs. Key advancements include enhanced training data richness and diversity, a scalable vision token sampler replacing Q-Former layers, and a unified training objective that simplifies the training process. xGen-MM models, including a pre-trained base model, an instruction-tuned model, and a safety-tuned model with DPO (Deterministic Prompt Optimization), demonstrate strong in-context learning capabilities and competitive performance among open-source LMMs. The safety-tuned model aims to reduce harmful behaviors such as hallucinations. All models, datasets, and fine-tuning code are open-sourced to advance LMM research. Resources will be available on the project page.

Key Highlights

• Enhanced Framework: xGen-MM (BLIP-3) improves on the previous BLIP-2 framework by increasing training data richness, employing a scalable vision token sampler, and unifying training objectives.
• Models and Performance: Includes pre-trained base, instruction-tuned, and safety-tuned models, with competitive performance in visual language tasks and benchmarks.
• Unified Training Objective: Simplifies training with a single loss function, improving efficiency and effectiveness.
• Safety Improvements: The safety-tuned model incorporates DPO to mitigate harmful behaviors and improve safety.
• Open Source: All models, datasets, and fine-tuning code are open-sourced to support further advancements in LMM research.

Comparison with BLIP-2

• Training Data: Increased scale and diversity in xGen-MM compared to BLIP-2.
• Architecture: Replaces Q-Former layers with a vision token sampler.
• Training Process: Unified objective with a single loss function versus multiple objectives in BLIP-2.

Thank you for your attention. Subscribe now to stay informed and join the conversation!

About us:

We also have an amazing team of AI engineers with:

• A blend of industrial experience and a strong academic track record ??
• 300+ research publications and 150+ commercial projects ??
• Millions of dollars saved through our ML/DL solutions ??
• An exceptional work culture, ensuring satisfaction with both the process and results ??

We are here to help you maximize efficiency with your available resources.

Reach out when:

• You want to identify what daily tasks can be automated ??
• You need to understand the benefits of AI and how to avoid excessive cloud costs while maintaining data privacy ??
• You’d like to optimize current pipelines and computational resource distribution ??
• You’re unsure how to choose the best DL model for your use case ??
• You know how but struggle with achieving specific performance and cost efficiency

Have doubts or many questions about AI in your business? Get in touch! ??