The Intelligent Co-Pilot

By Peter Wang , Director of Automotive and Digital Business Consulting, Nagarro

This article was originally published in the summer edition of the German Chamber of Commerce I China Ticker. Read the full issue online?here .

?

The Limitations of the Existing In-car User Experiences

Since Karl Benz invented the first automobile, humans have never stopped dreaming about cars. Today, the trend for electric vehicles is gradually replacing combustion engine vehicles. From traditional OEMs (Original Equipment Manufacturers) to new car manufacturers, product and service innovation is becoming the theme of the industry. However, apart from autonomous driving, no disruptive products have yet to come close to what the iPhone did to the mobile phone market. The pain points of the current in-car experience can be summarized as follows:

·???????Most interactions are rule-based interaction, not intelligent.

·???????Virtual assistants have limited conversational capability, not natural.

·???????Infotainment apps lack valuable functions, not useful.

·???????Virtual assistants have weak learning capability, not personalized.

How Generative AI will enable brand-new Customer Experiences

The breakthrough of generative AI is poised to revolutionize the in-car user experiences in the coming years or even months. By generating realistic and engaging content, generative AI can help to create a more enjoyable driving experience. The following diagram shows an example of how a generative AI driven co-pilot can enhance the customer’s experience in everyday driving

The intelligent co-pilot can anticipate the preferred in-car climate settings, proactively suggest safe driving operations under extreme weather, tell users how to benefit from unfamiliar functions, and collect user feedback on the features used.

?

From the start, the intelligent co-pilot is deemed to be the user’s concierge for all in-car services. Integrating with in-car sensor data and using machine learning models, it will anticipate the driver’s needs according to context, seamlessly offering assistance to access any car settings, service, or information needed through a single conversation, thus without having to manually select individual applications. The co-pilot will process user feedback and interactions and continuously improve the quality and performance of its service.

?

The co-pilot will power a more natural and multimodal in-car Human-Machine Interface (HMI) for images, videos and sounds.

Natural Language Communication

Thanks to the natural language dialogue ability of the LLM (Large Language Model), the intelligent co-pilot can interact with users in a natural and engaging way. Car makers or OEMs can program the interactions in a way that is more representative of the brand’s culture.

Visual Content Generation

The intelligent co-pilot can generate visual content based on its understanding of the user’s needs and preferences. It can also generate metaverse (AR/VR) content for entertainment under safe conditions, when appropriate, as determined by the driving conditions or context.

Car Control Interaction

The intelligent co-pilot can anticipate the preferred in-car climate settings, proactively suggest safe driving operations under extreme weather, tell users how to benefit from unfamiliar functions, and collect user feedback on the features used. Through in-car cameras and sensors, the co-pilot can detect the user’s mood, mental and physical status, and provide music, chatting, or massage services.

Because the co-pilot will have situational awareness it can offer proactive assistance. For example, the co-pilot could greet the user with different content each day to add to the driving experience. If the co-pilot finds the user is anxious in traffic, it could proactively select to play a piece of the user’s favorite music to help them relax. Over time, a personalized profile is created to reflect the driver’s identity and values, and customize the in-car HMI to fit the driver’s preferences, such as changing the voice and tone of the HMI. In effect, personalization will make the in-car experience more enjoyable.?

Combining knowledge graphs and machine learning, generative AI can help achieve more powerful, comprehensive and contextual accuracy, enabling co-pilots to provide proactive assistance to users on a continuous basis.?

Technical Design Considerations

The foundation model is at the center of the architecture. The foundation model, essentially, is an AI model that supports multi-modal input and context (such as text, image, video, audio and code, and generates executable code based on APIs (Application Program Interfaces)) to complete specific tasks.

The core concept for the intelligent co-pilot is to design a new AI ecosystem by connecting the foundation model with the vehicle side models and application systems through APIs. The foundation model then breaks down the user instructions in specific tasks and comes up with a reasonable solution outline to help select the most relevant API-generating executable code. Auto-code verification mechanisms are also included to confirm the reliability and trustworthiness of the generated code through a feedback loop.

?

Multimodal Conversational Context

Contextual data collected by sensors is sent to on-board models to predict preferred service and generate instructions for LLM.

The following architecture shows a reference design of the intelligent co-pilot:

?

This architecture provides a flexible and scalable framework for the intelligent co-pilot. It can be adapted to different vehicles and applications, and it can be easily updated with new features and capabilities.

?

New Opportunities for OEMs

Innovation and Differentiation in the Market

Generative AI will become an important tool for OEMs in product and service innovation that can collect valuable product feedback to optimize product design. It also gives intelligent co-pilots a digital soul that can represent a brand’s culture.

Sales and Aftersales Services

The new form of sales and after-sales service brought by generative AI will accelerate the business model transformation of OEM services to users, thereby bringing more sales and profits.

Customer Satisfaction and Loyalty

It is anticipated that the use of intelligent co-pilots, as it becomes more personalized, will stimulate users’ desire to continue using the product, thereby ensuring further loyalty to the brand.

Collaboration with Partners

New developments in generative AI may bring more opportunities for cooperation between OEMs and ecosystem partners as more user requirements are discovered from collected data.?

Challenges that should be Addressed

Data Privacy and Security

Data privacy and security will of course be addressed. The access to user privacy data must be strictly controlled and protected. As much as possible, the collected data should be kept at the vehicle side. Desensitized and encrypted data will be uploaded to the cloud. The content and service provided to users must comply with relevant laws and regulations.

Data Quality and Availability

Since generative AI relies on large amounts of high-quality data to train its models and generates realistic and responsible content, OEMs will need to set up well-defined data collection and governance policies. The product design should balance the trade-off between performance and resource consumption, optimizing the generative model’s architecture, parameters, and inference methods to reduce the computational cost and latency of generating content.

User Experience

Only when the co-pilot brings users a truly qualitative experience will the product be sought after by users. Managing user expectations at the early stage is critical, albeit while testing is underway.

Ecosystem Integration

The design and development of the co-pilot require multiple technologies and interdisciplinary R&D empowerment. A dedicated product and technical team is essential to ensure uniformity in control at the top-level of design, as well as component development through cross-departmental collaborations. Traditional OEMs will have to make changes in their organizational and operational structure to be competitive in the market.

Summary

Currently, OEMs may face a variety of challenges, but none as disruptive as the introduction of generative AI; there lies the future of the automobile industry. While Tesla is leading the world in the new field of autonomous driving, there is still something missing. People want more enjoyment from their experience, even if the car is driving itself. Generative AI can create an intelligent co-pilot that will make the driving experience more engaging, personalized and enjoyable. Generative AI is a breakthrough technology that will no doubt revolutionize the automobile industry. OEMs that embrace generative AI will be well-positioned to face this new era in automotive transportation.

Peter Wang is the Director of Automotive and Digital Business Consulting at Nagarro. He is passionate about designing innovative products and solutions. Peter has extensive consulting experience in automotive and digital commerce working with multinational companies in China. He holds a master's degree in software engineering from Peking University.

About Nagarro: Nagarro (Frankfurt: NA9) is a leader in digital product engineering and drives technology-led business breakthroughs. They help to transform, adapt, and build new ways into the future through a forward-thinking, agile and caring mindset. Today, they count 19,000+ experts across 34 countries, forming a Nation of Nagarrians, ready to help their customers succeed.