Introducing Audio & Video processing in Data Cloud, Now GA

Unlocking the Power of Audio & Video Data

Today, Salesforce is announcing the beta capabilities to process audio and video in Salesforce Data Cloud alongside structured data.

Why does it matter?

The growing consumption of multimedia content via social media and user-generated content and the popularity of online meetings—spanning customer service calls, webinars and interview recording—provide valuable insights into customer interactions, preferences, and sentiments. For example, audio analysis of calls can detect emotions, track conversational trends, and identify keywords reflecting customer needs, while video data reveals behaviors like in-store product interactions or key moments in webinars that capture audience interest. Companies that can tap into this unstructured audio and video can gain a deeper understanding of customer behavior and preferences.

Challenges in Processing Audio & Video Data

Unlike structured data, unstructured audio and video data don't follow predefined models, making them challenging to process and analyze using traditional methods. Audio analysis requires complex Natural Language Processing (NLP) models, while video relies on costly computer vision algorithms. Managing this data also requires substantial storage, efficient data management, and adherence to privacy regulations like GDPR. And lastly, companies must consider having a robust infrastructure to handle high processing loads, particularly for real-time insights.

The Salesforce Advantage in Processing Audio & Video Data

As part of today’s beta, Salesforce Data Cloud is introducing audio and video transcription capabilities. This allows organizations to index customer interactions, perform audio analysis, and conduct similarity searches. And since Data Cloud is integrated with the Einstein 1 Platform, this audio and video data becomes part of Salesforce metadata, empowering analysts to visualize insights in Tableau, developers to create Salesforce Flow automations, and business users to improve the accuracy of Einstein Copilot’s responses without the need to fine-tune Large Language Models (LLMs). This also marks the start of future capabilities, such as advanced customer interaction analysis, sentiment analysis, and actionable insights. analysis, and actionable insights.

The Data Cloud Unstructured Data Processing Hub will use Whisper, an OpenAI model, to transcribe audio conversations. For video files, audio content will be extracted and transcribed using the same model. The transcribed text will then create embeddings in the Vector Database introduced a few months ago. The beta supports multiple audio formats (FLAC, MP3, WAV) and video formats (AVI, MOV, MP4), with English as the initial supported language.

Benefits to Processing Audio & Video

Analyzing audio and video data offers numerous advantages:

• Personalization: By analyzing customer sentiment and preferences from multimedia interactions, businesses can tailor products, services, and communications to meet individual needs.
• Improved Customer Support: Audio and video analytics can identify pain points in customer interactions, enabling faster issue resolution and proactive support.
• Targeted Marketing: Insights into the tone and mood of customer interactions can refine marketing campaigns, leading to more targeted messaging and higher engagement.
• Efficient Content Management and Search: Indexing and analyzing multimedia libraries allow users to quickly find relevant content, enhancing experiences in entertainment, e-learning, and corporate training.

Innovation in Action Within Industries

Audio-video processing has applications across various industries.

In retail, video analytics can track customer movements and behaviors, optimizing store layouts and enhancing in-store experiences.

In telemedicine, speech recognition aids in diagnosing conditions, while video analysis supports medical imaging and remote patient monitoring. Analyzing viewer engagement helps companies improve content recommendations and enforce compliance through automated moderation.

In financial services, audio-video recordings of meetings and financial transactions can be analyzed for compliance, fraud detection, and operational efficiency.

Conclusion

Extracting actionable insights from unstructured audio and video data is increasingly crucial for businesses to excel in a competitive market. Salesforce Data Cloud’s latest innovation in audio and video processing helps businesses deliver personalized customer experiences, optimize operations, and ultimately transform multimedia content into a strategic asset for growth and success.

More Information

Where: This change applies to Lightning Experience in Professional, Performance, and Unlimited editions.

Note: Transcription and indexing of audio and video files is a pilot or beta service that is subject to the Beta Services Terms and Agreements - Salesforce.com or a written Unified Pilot Agreement if executed by Customer, and applicable terms in the Product Terms Directory. Use of this pilot or beta service is at the Customer's sole discretion.

When: This functionality is available starting in August 2024.

How: In Feature Manager and enable Audio and Video File Transcription in Search Index. In the search index advanced setup, create a search index configuration for a UDMO based on audio and video files. Transcription processing occurs when you create the search index configuration.

Use Case: Sales meeting Support: Explore Conversations

Beta Limitations

The beta release of this feature includes these limitations.

• Indexing audio and video transcriptions is not supported in search index configurations created from data kits.

See Also

• Transcribe and Index Audio and Video Files (Beta)
• Create a Search Index Configuration