LLM Fine-Tuning on Graphs; How To Evaluate LLMs; Uncovering Knowledge Gaps Using RAG; Claud 3 on Bedrock; Overcoming Limits Of RAG; and More.

Editor's Paper Recommendations?

Efficient Large Language Models Fine-Tuning on Graphs: Learning from Text-Attributed Graphs (TAGs) has attracted significant attention due to its wide range of real-world applications. The rapid evolution of large language models (LLMs) has revolutionized how we process textual data, indicating a strong potential to replace shallow text embedding generally used in graph neural networks (GNNs). However, existing LLM approaches that exploit text information in graphs need to improve their computation and data efficiency. In this work, we introduce a novel and efficient approach for the end-to-end fine-tuning large language models (LLMs) on tags, named LEADING. The proposed approach maintains computation costs and memory overhead comparable to the graph-less fine-tuning of LLMs. Moreover, it effectively transfers the risk knowledge in LLMs to downstream graph learning tasks with limited labeled data in semi-supervised learning. Comprehensive experiments demonstrate its superior computation and data efficiency, offering a promising solution for many LLMs and graph learning tasks on TAGs.

LLMEval: A Preliminary Study on How to Evaluate Large Language Models: Recently, the evaluation of large-language models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions: what tasks should be given to the LLM during testing, and what kind of knowledge should be dealt with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there has yet to be much discussion. This paper analyzes evaluation methods by comparing manual and automatic evaluation criteria. We utilize onsite crowdsourcing, public annotators, and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval, and conduct evaluations of 20 LLMs. 2,186 individuals participated, generating 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and draw 10 conclusions that can provide insights for evaluating LLM in the future. The dataset and the results are publicly available at?this https URL.

Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator: Large Language Models (LLMs) excel at understanding human instructions, driving the development of multimodal LLMs (MLLMs) with instruction tuning. However, acquiring high-quality multimodal instruction tuning data poses a significant challenge. Previous approaches relying on GPT-4 for data generation proved expensive and exhibited unsatisfactory performance for specific tasks. To solve this, we present Genixer, an innovative data generation pipeline producing high-quality multimodal instruction tuning data for various tasks. Genixer collects datasets for ten prevalent multimodal tasks and designs instruction templates to transform these datasets into instruction-tuning data. It then trains pre-trained MLLMs to generate task-specific instruction data and proposes an effective data filtering strategy to ensure high quality. To evaluate Genixer, a base MLLM model, Kakapo, is built and achieves SoTA performance across multiple datasets in image captioning and visual question-answering (VQA) tasks. Experimental results show that filtered data from Genixer continually improves Kakapo for image captioning and VQA tasks. For the SoTA Shikra MLLM model on image-region-related tasks, e.g., region caption and detection, Genixer also successfully generates corresponding data and improves its performance. Genixer opens avenues for generating high-quality multimodal instruction data for diverse tasks, enabling innovative applications across domains. The code and models will be released soon.

Harnessing Retrieval-Augmented Generation (RAG) for Uncovering Knowledge Gaps: The paper presents a methodology for uncovering knowledge gaps on the internet using the Retrieval Augmented Generation (RAG) model. The RAG system identifies and addresses gaps in information retrieval systems by simulating user search behavior. The study demonstrates the effectiveness of the RAG system in generating relevant suggestions with a consistent accuracy of 93%. The methodology can be applied in various fields, such as scientific discovery, educational enhancement, research development, market analysis, search engine optimization, and content development. The results highlight the value of identifying and understanding knowledge gaps to guide future endeavors.

--

Are you looking to advertise a product, job opening, or event to an audience of over 40,000 AI researchers and engineers? Contact us on?LinkedIn?to explore your options.

Enjoy the newsletter? Help us make it bigger and better by sharing it with colleagues and friends.

--

Industry Insights

• UC Berkeley Research Presents a Machine Learning System that Can Forecast at Near Human Levels
• Claude 3 Opus vs GPT-4 vs Gemini 1.5 Pro AI Models Tested
• Amazon adds GPT-4-beating Claude 3 to Bedrock
• Overcoming the Limits of RAG with ColBERT

Growth Zone

?To Overcome Your Fear of Public Speaking, Stop Thinking About Yourself

Expert Advice