Web AI Monthly #23:??1st ever summit for client side AI - including a prediction for the future on how we will evolve to use the internet ??
Jason Mayes
Web AI Lead @Google 13+yrs. Agent / LLM whisperer. On-device Artificial Intelligence / Machine Learning using Chrome | TensorFlow.js | MediaPipe. ?? Web Engineering + innovation ??
Enjoy the content? Help me help you, by giving us a share with your colleagues in your weekly team updates to get more eyes on great work in the Web AI field.
Tag Jason Mayes if you make something noteworthy for future editions - many of our readers work for top global tech companies or startups. We have subscribers ranging from decision makers (think C-level, VPs, and Directors) to folk on the frontlines using this stuff day to day (SWEs, web engineers and researchers). You never know who may see your creations.
Web AI Summit 2024
We did it! On 18th October, 2024, 谷歌 hosted the very first?Web AI Summit?to bring together top minds from around the world working with machine learning models?client-side in the web browser. Fun fact: Did you know that 66% of all CPU cycles globally are spent running content in a web browser? Thanks to Moh Haghighat at 英特尔 for that gem from his talk.
I decided to make the event public for the first time this year following our huge growth in Web AI usage here at Google, growing from around 1M yearly downloads to approximately 1.2B in 2023 - now was the moment to bring us together to see if there was interest in doing this regularly, and the TLDR was that interest was higher than we anticipated!
Our lineup included presenters not only from Google’s teams such as Chrome and MediaPipe, but also active 3Ps in the space such as Hugging Face, Intel, Microsoft, LangChain, MLC and beyond. From consumer packaged goods SKU detection to advanced healthcare solutions - talks covered a wide range of industries and subject areas showing just how far Web AI can reach across industry verticals.
Who attended?
We had over 1,100 registrations from folk spanning 22 countries, 59 cities, and 179 different Google offices joining us for this historic event - with a full house the whole day - it was great to see how engaged everyone was for the talks.
A mixture of software engineers, business decision makers, and executive leadership were in the audience, creating a productive synergy between technical expertise and strategic planning for 2025 and beyond.
Where can I watch the talks?
Great question - they shall be going live on this brand new Web AI YouTube playlist over the coming weeks as we edit them! Bookmark the playlist, subscribe, or add notifications for when new videos are added if you want to know when. The first batch should be up by the time you are reading this.
In the meantime here is the full list of talks that will be added to the above playlist along with links to presenter's socials if you wish to connect with anyone to ask questions.
Welcome to Web AI Summit 2024
An overview for the state of Web AI in 2024 and why the Web AI Summit was created. See what's possible with?machine learning on-device, as well as where it is heading, to get the 101 before you watch the other talks in this series. This talk is suitable for everyone and covers subject areas such as generative AI, LLMs, diffusion models, WebGPU, WebAssembly, and emerging APIs like WebNN along with examples from industry that are already using Web AI today. Watch until the end to hear how the future of the internet itself may evolve with Web AI.
Transformers.js: State-of-the-art Machine Learning for the Web
Learn about Transformers.js, an exciting new JavaScript library that empowers developers to build never-before-seen web applications. It is designed to be functionally equivalent to Hugging Face's Python transformers library and supports over 120 architectures across a diverse set of tasks and modalities. Users can choose from over 1,000 pretrained models or convert their own to run locally in the browser, offering privacy-preserving, low-latency, and scalable machine learning. The latest addition of WebGPU support enables highly-performant execution of models by utilizing modern GPU capabilities directly in the browser.
The Web Neural Network (WebNN) API: Where we are and What's Next
Advanced web technologies like WebAssembly and WebGPU have recently brought real AI capabilities to the browser. The proposed Web Neural Network (WebNN) API aims to build on that momentum, enabling AI workloads to run faster and more efficiently on a variety of devices, including devices with AI accelerator hardware (NPUs), all based on web standards. This session starts with a brief overview of WebNN, then describes recent developments, including API shape, device support, framework support, and browser implementations. We also describe the plan forward, as we work to get feedback from the community.
Web AI on AI PC
Intel showcased WebNN, an emerging unified W3C web standard API for on-device web ML acceleration across client AI execution engines: CPU, GPU, and NPU. Currently in Developer Preview on Chrome or Edge browsers and integrated in popular ML frameworks (e.g., ONNX Runtime Web), WebNN delivers “near-native” performance and power characteristics. Learn by example via exciting WebNN demos and adoption previews that bring a new class of experiences to the web.
ml5.js - Friendly Machine Learning for the Web
This talk focused on ml5.js - an open source library built on top of TensorFlow.js with a goal of making machine learning approachable for a broad audience of artists, creative coders, and students. This project has been built as a collaborative effort at NYU’s ITP Program, drawing inspiration from Processing and the p5.js project’s focus on making coding accessible and inclusive. ml5.js aims to expand this mission to the domain of machine learning, bridging the gap between the technical complexity of machine learning and the creativity of beginners and artists.
WebLLM: A High-Performance In-Browser LLM Inference Engine
This talk covered WebLLM, a high-performance in-browser LLM inference engine. WebLLM allows building AI-enabled web apps that are fast (native GPU acceleration via WebGPU), private (100% client-side computation), and convenient (zero environment setup). For developers, WebLLM features an OpenAI-API style interface for standardized integration, supports chat applications and efficient structured JSON generation, and offers built-in support for Web/Service Workers to separate backend executions from the UI flow. In this talk, we explore WebLLM’s key features, overall architecture, and how developers can build AI-enabled web applications with it.
State Isn't All You Need, But It Helps: Building Better LLM Apps in the Browser
Exciting new advances from projects like WebLLM, Transformers.js, and Chrome AI have brought local LLMs closer than ever to anyone with a browser. This has immense potential to expand the frontiers of web development, but these small models are more limited than state-of-the-art hosted models and require more careful considerations around design and prompting. This talk focused on addressing these constraints by covering techniques for implementing practical apps that make the most of small models using the powerful toolkit provided by LangGraph.js, a new framework for orchestrating stateful LLM apps.
Visual Blocks: Visual Prototyping of AI Pipelines
Visual Blocks for ML?is a visual programming platform that empowers rapid AI and multimedia prototyping. In this talk, we will showcase how to build interactive AI pipelines, perform interactive data augmentation, and test pipelines with live data using simple drag-and-drop actions. We also highlight a range of community-contributed pipelines and custom nodes demonstrating diverse applications in interactive graphics, large language model chains, computer vision, and multi-modal solutions. Finally, we encourage all Web AI practitioners to contribute their own ML pipelines and custom nodes, further enriching the shared platform and inspiring innovative use cases.
Exploring alternative interactions in JavaScript
The latest advancements in AI have mainly focused on large language models and new ways of creating and consuming content. However, AI also offers the opportunity to rethink the way we interact with interfaces. Using JavaScript and models focused on body tracking or audio classification, web developers have a unique opportunity to experiment with alternative interactions to create more innovative web experiences.
领英推荐
Overview of Chrome built-in AI
Sharing what we've been up to in Chrome for built-in AI, what we've learned, and what's next. We'll talk about how we see the Prompt API, our status for high-level task APIs such as summarization, write / rewrite, learnings from the early preview program, and where we are going from here.
Web AI in Industry: How TensorFlow.js has driven what you see on the supermarket shelves
This talk showcased how one of the top 10 largest consumer packaged goods (CPG) companies in the world utilized Web AI to expand its in-store trade marketing strategy in Brazil and how it evolved into an open-source project that has been beneficial to other companies in the industry.
Lessons learned from being customer zero of Chrome's built-in APIs
In this talk, Thomas summarized some of the things Developer Relations has learnt in their role as customer zero of Chrome's built-in APIs. Using an example of an AI-powered synonym finder app, he will show how to work with the Prompt API focusing on aspects from tweaking the prompt, to reliably parsing the output, to optimizing the app for maximum performance.
The Future of AI is Now: Real-life Case Studies for on Client-side AI Adoption in Web Apps
This lightning talk reveals the practical power of client-side AI not just for the sake of using AI, but for enhancing user experiences. The talk took a deep dive into a real-world case study featured in Google I/O 2024, showcasing how CyberAgent, the powerhouse behind one of Japan's top blog services, plans to leverage the magic of client-side AI to empower users with effortless blog title generation. Join us to learn how CyberAgent maximized the potential of client-side AI through innovative use case design and a user-centric approach.
Why are Web Extensions fantastic for AI?
In this talk we will showcase the potential of AI and Chrome Extensions. Chrome Extensions allow you to control the browser, observe web content, and add your own UI. When combined, AI and Chrome Extensions can make the browsing experience truly helpful and more productive. This talk will give an overview on how extensions on the WebStore are using AI today and where we see the biggest potential.
Beyond the Banner: The Power of Web AI to Personalize Paid Media
Discover how Web AI is revolutionizing personalized paid media by introducing groundbreaking virtual try-on ads across every screen entirely locally in the browser. See the latest in augmented reality try on technology in this talk by GEENEE from handbags to sunglasses - they have it all.
MediaPipe Web: Bringing cross-platform AI tech to the browser
Learn about MediaPipe's cross-platform approach to building AI pipelines and bringing them to the browser. We'll highlight some of the benefits of our method and talk about a few of the major products we help power (like Google Meet). Then we'll cover our latest technological advancements and developer APIs. These offerings include solutions for traditional machine learning tasks like image segmentation, as well as generative AI tasks like LLM inference. Finally, we will give a sneak peak into the future with some exciting demos!
Transforming access to healthcare through Web AI
IncludeHealth, a virtual physical therapy provider, harnesses the power of Web AI to break down logistical and economic barriers, allowing patients to receive personalized, measured care anywhere, any time, and on any device.
ML Training on the Web: Building Simple ML for Google Sheets
Learn how we built Simple ML for Sheets, a free Google Sheets add-on for ML and AI. Simple ML for Sheets uses on-device Machine Learning powered by WebAssembly, JavaScript and Chrome’s new built-in AI to unlock advanced Machine Learning tasks for all users. This talk highlights the tools we used to successfully bring Simple ML for Sheets to market and how our team’s open source libraries help developers achieve their own ML successes on the web.
To the future: A prediction on how our usage of the internet itself will evolve with Web AI
So I promised you a prediction. Having worked in the web industry now for well over 15 years, and leading Web AI here at Google shortly after its birth, where do I think it is all going in 5, maybe 10 years from now?
Well, I believe that in the future every website that can call itself "modern" and "usable" will have an "AI.json" file in the domain root, just like we have a robots.txt file today for web crawlers. What does this file do? It will act as the interface to Large Action Models (LAMs) and Large Language Models (LLMs) such that they can perform common tasks on our behalves on popular websites we use frequently - without us having to figure out arbitrarily changing UX's every few months.
Instead, you will press a command on your web browser, and type or speak what action you are there to perform, and the LLM or LAM will do most of the work for you and then hand back to you to press the checkout button or such that may need human confirmation.
Of course, your regular UX will be there for humans who want to peruse a site at leisure, but my guess is the children of tomorrow wont grow up thinking like that. Instead they will grow up in a world where they speak to LLMs from a young age and are used to telling such models what they want done and will be happy to delegate trivial things to such AI agents in the future.
What do you think? Let me know in the comments below.
See you next time!
If you're new to this space and want to learn Web AI, you can get started fast with my free Google Developers course here (no background in AI needed, just a love for JavaScript and curiosity for AI - I will teach you from zero). Or get inspired through our growing collection of Web AI talks on YouTube or via my Show & Tell - I got you either way!
See you next time with even more great content and please do tag me (Jason Mayes ) if you make or find something for future editions - I need your help to find the latest and greatest news lovely #WebAI community as things are moving so fast!
Cheers!
Jason Mayes (that Web AI guy).