Advances in non-generative Machine Learning

GenerativeAI has been in the news for the past couple of years. However, in the background, significant work has happened in the non-generative AI space. This topic is interesting because most products we build are not direct genAI applications. For example, a RAG application needs a good search to work well. So search may become a limiting factor for a RAG application.

The excitement around GenAI has channeled money to many relatively underfunded areas in Machine Learning. In the rest of this article, I will elaborate on some topics where I see steep improvements.

Search

A lot of funding and effort has gone into Search. ChatGPT showed the potential of generative models, but everyone immediately knew that chatGPT had to be augmented with new knowledge if its responses were to be useful. Search was the best way to get there.

Dense retrieval saw innovations in entailment tuning and multi-variate dense retrieval. Research has shown that the choice of retrieval unit (document, passage, sentence, or proposition) significantly impacts the performance of dense retrieval systems. A novel proposal is to use propositions—compact expressions encapsulating distinct facts—as retrieval units. Sparse Retrieval similarly saw a spurt of interesting ideas. One such example is the Learned Sparse Retrieval model called SPLADE.

Search infrastructure is another area of innovation. Vector databases improved a lot in terms of efficiency and performance.

Applications

1. Enterprise Search: Organizations utilize enterprise search tools to aggregate and manage numerous internal documents, enabling employees to find relevant information quickly and fostering collaboration.
2. Product Search and Recommendations: E-commerce platforms implement advanced search algorithms to enhance product discovery based on user behavior, preferences, and purchase history, leading to personalized shopping experiences.
3. Multi-modal Search: Applications such as - text-to-image search, image-to-text search, and image-to-image search have become very common, especially in the E-commerce search space.

Voice and Visual Search Integration - Multi-modal representations

Multimodal embedding is a sophisticated approach in machine learning that integrates multiple types of data—such as text, images, and sometimes audio or video—into a unified vector space. This technique allows for the simultaneous processing and analysis of different data modalities, enabling models to capture complex relationships and semantics across varied forms of information.

Applications

Multimodal embeddings have a wide range of applications across various fields:

1. Image Captioning: Generating textual descriptions for images by understanding visual content and relevant text.
2. Cross-Modal Retrieval: Searching for images using text queries or finding text that describes an image.
3. Recommendation Systems: Enhancing user experiences by suggesting products based on visual similarity or textual descriptions.
4. Sentiment Analysis: Analyzing mixed media content (e.g., videos with both audio and visual elements) to gauge sentiment more accurately

Federated Learning for Privacy-Preserving Search

Federated learning (FL) is a decentralized approach to machine learning that enables multiple parties to collaboratively train models without sharing their raw data. This method addresses critical privacy concerns by allowing data to remain on local devices, thus enhancing security and compliance with regulations such as GDPR and HIPAA.

Here are a couple of papers talking about the advances and challenges in Federated Learning.

Applications

• Healthcare: FL is increasingly being applied in healthcare settings, allowing hospitals to collaboratively train models on sensitive patient data without sharing it. This approach enhances diagnostic accuracy while preserving patient privacy.
• Financial Services: In finance, federated learning is used for fraud detection algorithms that can analyze client data across institutions without compromising individual privacy.
• Internet of Things (IoT): FL integrated with IoT devices enables real-time learning and model updates at the edge, reducing latency and bandwidth usage. This is particularly useful in dynamic environments like smart cities, where data is continuously generated.

Conclusion

While generative AI captures the most attention, a wealth of exciting research and applications is emerging across various domains. Every sub-field of machine learning is witnessing significant advancements, and new areas of exploration are continuously emerging.