Web ML Monthly #13: Visual Blocks, Tech Talks, and Figma plugins

Hey Tensors!?We're about halfway through 2023 already - time sure does go fast, and that means a tonne of updates in the Web AI / ML world too :-)

If you enjoy the content, please do give us a share with friends, colleagues, family, dogs, and cats - everyone is welcome.?Let's go!

Visual Blocks ML - an open Web AI powered visual framework to get ML powered ideas to production faster.

I am particularly proud to share an update to a project I have been involved with for over 2 years now with Na Li , Jing Jin , Ruofei Du , Michelle Carney , et al and it's finally out in the wild!

Visual Blocks from Google is an open source graphical programming framework for rapid prototyping and experimentation of production ideas powered by machine learning. It enables developers and decision-makers alike to work together to solve problems through powerful reusable building blocks, iterate, and then deploy with ease for all to try. You can build an end to end ML pipeline to share with your team in minutes!?Check my video tutorial below to see how:

In the video above you can learn how to use Visual Blocks through real-world examples so you can adapt this knowledge to your own products and services.

From creating your own social media worthy augmented reality effects on a live webcam feed, to running LLMs and generative models in a Colab (yep we support Google Colabs too to bring your Python code to life with the power of front end web technologies). You can then experiment chaining inputs, other models, and outputs together in powerful ways - the options are almost limitless as to what you can achieve using Visual Blocks.

See it in action in the GIF below with Google's new PaLM2 API allowing you to combine it with Visual Blocks nodes to do powerful things like generate emails based on some inputs and then refeed those generated emails to another instance of the LLM to pick which email is best and explain why its the best ??

We would love to hear your feedback over on the Visual Blocks Website: https://goo.gle/VisualBlocksML for feature requests and more! What will you use it for? Tag me if you have questions or make a cool graph flow!

What's new with Web ML in 2023?

Missed Google IO? Well, I got your back. Check my short roundup of all the latest innovations in the Web AI space powered by browser technologies (JavaScript, WebGL, WebGPU, Web Assembly) below:

The performance gains are off the charts! Really, things have now progressed 100x since the start of this journey with the release of WebGPU now being in Chrome stable (thanks Corentin Wallez ) and was featured in Chrome's keynote:

This really is shaping up to be the year for Web ML in the browser.

Didn't click the video above yet? Well here is one more reason - it has cats in it, AI generated cats. Yep, we can even run diffusion models in the browser now in under 10 seconds! This was so big that it even got a dedicated shout out in the developer keynote at Google IO - well done to Matthew Soulanille for his epic work on porting this model. No more high server GPU costs here if you have a decent local GPU with plenty of VRAM.

Anyhow, go watch the video and get back to this newsletter. Got a question? Leave a comment on the video or reach out to me on LinkedIn!

Figma Plugin: Automatic naming of layers in a design project

So the Figma community have been hard at work embracing the power of Web ML as their plugin ecosystem is essentially powered by JavaScript.

As a designer, you often have many layers in your design file, and by default it may be called something like "Untitled layer 298" when you get to your 298th design element (you can get to high numbers like this pretty fast as every button, text on a button etc can be a new layer).

Well, with the power of AI, Hugo Duprez made an open source plugin that can automatically name your layers to something more sensible, saving designers a lot of time and thought. Check out his demo and tweet right here:

I look forward to see how the design community continue to embrace Web AI and TensorFlow.js to do some really great things in the future. This market is just getting started in the grand scheme of things and is ripe for innovation right now.

Community Demo: Road network extraction on satellite images

Shout out to Quadeer Shaikh for tagging me in his latest TensorFlow.js work that is able to take any satellite image and then extract the roads from the image to then automatically be plotted on a map. Really awesome work. Check his post below for more details - its great to see folk solving real problems they are facing with Web ML:

See you next time!

Finally, if you've made something cool or seen a demo out in the wild, be sure to tag it with?#WebML?on LinkedIn / Twitter / social so we can find it for a chance to be in our newsletter, future events, or even our?YouTube show and tell!

I will leave you with this fun Web ML demo of me dancing with copybaras thanks to GEENEE 's latest TensorFlow.js Augmented Reality demo (you will want to turn the sound on for this one - super fun):

See you next month with even more great?#WebML?content that's?#MadeWithTFJS?and beyond. Cheers!

Jason