Welcome to the TensorFlow.js monthly newsletter!

Welcome - what's this all about? Web ML and innovation essentially.

TensorFlow.js ?is an?open source machine learning library ?that can run anywhere JavaScript can - from client side in the browser to server side via Node.js, it's one of the only forms of TensorFlow that allows you to deploy Machine Learning (ML) anywhere, at scale, and in a frictionless "zero install" manner that anyone can use just by visiting a link.

It's based upon the original TensorFlow library written in Python and aims to re-create this developer experience and set of APIs for the JavaScript ecosystem which is used by over ~70% of professional developers globally . Since I moved to the team in 2020 we have seen ~10x growth in model usage by folk all around the world and no signs of stopping - so now is the time to get onboard the ship of Web ML before it sails as more and more developers are using in production use cases every day. It's very rare to be at the start of a new evolution of an industry but here we are taking JS into the world of AI and ML.

As AI and ML become more common place across all industries, web engineers, developers, and creatives will find clients have higher expectations for smarter web apps as standard. From automatically cropping images so the human in the image doesn't get chopped in half when on a mobile phone, to smart text summarization in blog search results, TensorFlow.js is currently one of the only production quality ML libraries that can help you deliver super powers in your next web app, opening up a whole bunch of opportunities for JS developers no matter what industry your client may be in. From sports and healthcare apps like Google's dermatology assist , or Include Health's remote physiotherapy , to human computer interaction , TensorFlow.js can touch and benefit them all and many more.

This newsletter aims to provide insights and updates around the ever growing TensorFlow.js ecosystem from both the TensorFlow.js team here at Google, but also the wider community highlighting some of the best work I have seen out in the wild that may just inspire your next innovation in the web industry. So sign up, stay tuned, and enjoy the ride!

I'm new, give me the 101

To kick this all off for February, I would like to share with you a 101 on TensorFlow.js for those of you who may be new or discover this newsletter at some point in the future. For our regular users, feel free to scroll on down to updates for this month further down below, for everyone else - grab a coffee and enjoy a whistle stop tour of the state of AI in JS with the video below. Turn up the volume and enjoy.

February Updates

New models this way come!

TensorFlow.js has partnered with MediaPipe to bring some of their popular models to the JavaScript ecosystem enabling even greater device support being able to run on the TensorFlow.js backend via WebGL for GPU acceleration across even more devices. For example on devices like iOS - all from the comfort of your favourite web browser.

First up is an update to the GHUM 3D Pose Estimation model that we released last year which now has the ability to also perform full body segmentation in addition to the pose estimation as shown in the animation below.

Many of our users have been using the popular BodyPix TensorFlow.js segmentation model since launch, however this new GHUM model offers both 2D and 3D pose estimation with much more accurate segmentation across the body. In the animation below you can see how it works well even when part of the body is out of view yet we can still mask the background easily from the moving person:

Furthermore in this same release we also have a selfie segmentation model specifically designed for people sitting in front of a webcam that is well suited for this sort of environment where only the top half of the body is in view typically and where the user may be wearing things like headphones etc as shown below:

Try the live demos for both these new segmentation models:

1. GHUM 3D full body segmentation
2. Selfie segmentation

Next up is an update to our highly used hand pose model with improved accuracy for 2D, novel support for 3D, and the new ability to predict key points on both hands simultaneously.?See the animation below to see it in action:

From gesture recognition to sign language recognition, this hand model could be used for many creative web apps. Just one example is shown in the video below:

What will you make? Try the demo for yourself here .

Community highlights!

From self driving cars to visualizing auto encoders to generate images of faces in real-time in the browser, we had a number of great highlights in the recent past from our Show & Tell Community. Check this TensorFlow YouTube playlist for just a few examples of the great things people are making from all around the world!

Be sure to be on the look out for next live show, that will hopefully come out end of March or early April with even more great demos from the community.

Lions, tigers, and urm robots?

So I also wanted to give a sneak peak of a little side project I am personally working on. JavaScript is not limited to working on the software stack only even though many non JS developers associate with JS purely with the browser. Many developer kits like the Raspberry Pi can run Node.js and thus it opens up the world of hardware too.

Below is the newest member of the TensorFlow.js team sporting some TensorFlow.js sticker swag. Stay tuned for updates to see how TensorFlow.js can be used with robots like this to do some pretty great things - assuming I do not destroy it in my learning process - first I must carefully remove the head and attach my own Raspberry Pi instead to control the 12 servos in arms/legs of this quadruped.

My hope is to give it a high level ML abilities via TensorFlow.js in the browser enabling live model retraining in real time to then do various tasks while keeping the low level things like balance on the robot itself so the battery can be used more for moving around and not expensive ML model execution that could drain the battery fast.

Stay tuned for the world of On Network Machine Learning (ONML) which is not quite on device and not quite cloud. A hybrid of the two powered by TensorFlow.js in a low latency manner. Well that's the plan. Got thoughts? Let me know.

Send me your finest finds

If you make or see something cool out in the wild, be sure to tag it with #MadeWithTFJS on LinkedIn / Twitter / social so our team can find it for a chance to be in our news letter, future events, or even our YouTube show and tell!

Until next time friends. See you in March with our next update!