Infinity possibility with Data, AI and Humans if... - AWS re:Invent 2023 Hightlights #2

A powerful relationship between Data, AI and human is unfolding right before us. Generative AI is augmenting our productivity and creativity in new ways, while also being fueled by massive amounts of enterprise data and human intelligence.

Swami Sivasubramanian, VP of Data and AI at AWS, has shown how we can use company data to build differentiated generative AI applications and accelerate productivity for employees across the organization.

He tracked back 200 years of technological innovation, the exploration of new technologies to reduce manual labor and complicate tasks.

He mentioned that the connection between data, generative AI, and human intelligence, like symbiotic relationships in nature with the example of whale sharks and remora fish. The remora fish cleans the whale shark and keeps it healthy, and the whale shark, in turn, keeps the fish safe from predators.

He laid out the four essentials of building generative AI applications:?

1. Access to a variety of foundation models (FMs)
2. A private environment to leverage your data
3. Easy-to-use tools to build and deploy applications
4. Purpose-built ML infrastructure

Access to a variety of foundation models (FMs)

Anthropic's Claude 2.1 and Meta's Llama 2 70B models: Both are available now on Amazon Bedrock and suitable for large-scale tasks, such as language modeling, text generation and dialogue systems.?Claude 2.1 offers a 200K token context window and improved accuracy in long documents.

Amazon Titan image generation in Amazon Bedrock: Now, AWS users can generate realistic, studio-quality images at large volume and low cost using natural language prompts in Amazon Bedrock. Also, AWS’s Generative AI Innovation Center will be providing AI certifications and helping customers with their Bedrock generative AI applications.

Amazon Titan Multimodal Embeddings:?Allows organizations to build more accurate and contextually relevant multimodal search and smart recommendation experiences. The model converts images and short text into embeddings — numerical representations that allow the model to easily understand semantic meanings and relationships among data. These are stored in a customer’s vector database.

However it can be a headache to train foundation models.

Amazon SageMaker HyperPod: Helps reduce time-to-train foundation models (FMs) by providing a purpose-built infrastructure for distributed training at scale, helping to reduce the time it takes to train models by up to 40%.

A private environment to leverage your data

AWS Clean Rooms ML (Preview) helps you and your partners apply privacy-enhancing ML to generate predictive insights without having to share raw data with each other. The capability's first model is specialized to help companies create lookalike segments. With AWS Clean Rooms ML lookalike modeling, you can train your own custom model using your data, and invite your partners to bring a small sample of their records to a collaboration to generate an expanded set of similar records while protecting you and your partner’s underlying data.

Easy-to-use tools to build and deploy applications

• Amazon Neptune analytics: An analytics database engine that can help surface insights by analyzing tens of billions of connections in seconds with built-in graph algorithms, enabling faster vector searches with both graphs and data.

• Amazon Q generative SQL for Amazon Redshift Serverless: Enables data engineering teams to accelerate data pipeline build. Q can write SQL queries faster using natural language and help simplify the process for custom ETL jobs

• Amazon OpenSearch Serverless vector engine: This will lead to more efficient searches and processes. Amazon MemoryDB for Redis: This will support vector search, leading to faster response times and allowing tens of thousands of queries per second. It’s a useful application, in particular, for fraud detection in financial services. Databases like MongoDB and key-value stores like Redis: These will be available as a knowledge base in Amazon Bedrock.

• Amazon Q data integration and many more...

Purpose-built ML infrastructure

ML infrastructure starts with a strong data foundation, that is comprehensive, integrated and properly governed.

AWS is committed to a zero-ETL future and fundamentally simplifying customer ETL jobs, accelerate innovation.

Amazon OpenSearch Service zero-ETL integration with Amazon S3 can offer seamless search, analyze, and visualize log data stored in place in Amazon S3 using a single tool.

Where is human in all this?

To summarize,

AI technology can, and will, inevitably change everything from customer support to back office capabilities. But due to the AI skills gap, leveraging AI technology effectively is a time-consuming and resource-intensive process for many organizations.?

One of the themes that emerged from Dr. Sivasubramanian’s keynote was the importance of making AI tools and technologies accessible to everyone—from AI and ML experts to novices new to the scene.?He also explained soft skills like creativity, ethics, and adaptability will become even more important with the emergence of AI.

there's infinity possibility with Data, AI and Humans if...

we choose collaboration over silo thinking

take actions to innovate on ideas over debating or just thinking about it

reinventing towards the future by learning from the past over holds on to the obsolete knowledge from past

For more information check out the blog by Dr. Swami Sivasubramanian here.