What is TensorFlow.js?

TensorFlow.js is an open source machine learning library that can run anywhere JavaScript can. It's based upon the original TensorFlow library written in Python and aims to re-create this developer experience and set of APIs for the JavaScript ecosystem.

For those of you who are more visual and prefer videos, check out this short overview, else continue reading below:

Where can it be used?

Given the portability of JavaScript this means we can now write in 1 language and perform machine learning across all of the following platforms with ease:

• Client side in the web browser using vanilla JavaScript
• Server side and even IoT devices like Raspberry Pi using Node.js
• Desktop apps using Electron
• Native mobile apps using React Native

TensorFlow.js also supports multiple backends within each of these environments (the actual hardware environments it can execute within such as the CPU or WebGL on GPU). A "backend" in this context does not mean a server side environment - the backend for execution could be client side in WebGL for example. Supporting multiple backends like this ensures compatibility and also keep things running faster than vanilla JS. Currently TensorFlow.js supports:

• WebGL execution on the device's graphics card (GPU) - this is the fastest way to execute larger models (over 3MB in size) with GPU acceleration.
• Web Assembly (WASM) execution on CPU - to improve CPU performance across devices including older generation mobile phones for example. This is better suited to smaller models (less than 3MB in size) which can actually execute faster on CPU with WASM than with WebGL due to the overhead of uploading content to a graphics processor.
• CPU execution - the fallback should none of the other environments be available. This is the slowest of the three but is always there for you.

Note: You can choose to force one of these backends if you know what device you will be executing on, or you can simply let TensorFlow.js decide for you if you do not specify this.

Client side super powers

Running TensorFlow.js in the web browser on the client machine can lead to several benefits that are worth considering.

Privacy

You can both train and classify data on the client machine without ever sending data to a 3rd party web server. There may be times where this may be a requirement to comply with local laws such as GDPR for example or when processing any data that the user may want to keep on their machine and not sent to a 3rd party.

Speed

As we are not having to send data to a remote server, inference (the act of classifying the data) can be faster. Even better, you have direct access to the device's sensors such as the camera, microphone, GPS, accelerometer and more should the user grant you access.

Reach and scale

With one click anyone in the world can click a link you send them, open the web page in their browser, and utilise what you have made. No need for a complex server side Linux setup with CUDA drivers and much more just to use the machine learning system.

Cost

No servers means the only thing you need to pay for is a CDN to host your HTML, CSS, JS, and model files. The cost of a CDN is much cheaper than keeping a server (potentially with a graphics card attached) running 24/7.

Server side features

Leveraging the Node.js implementation of TensorFlow.js enables the following features.

Full CUDA support

On the server side, for graphics card acceleration, one must install the NVIDIA CUDA drivers to enable TensorFlow to work with the graphics card (unlike in the browser which uses WebGL - no install needed). However with full CUDA support you can fully leverage the graphics card's lower level abilities which can lead to faster training and inference times. Performance is on parity with (or faster than) the Python TensorFlow implementation as they both share the same C++ backend.

Model Size

For cutting edge models from research you may be working with very large models, maybe gigabytes in size. These models can not currently be run in the web browser due to the limitations of memory usage per browser tab. To run these larger models you can use Node.js on your own server with the hardware specifications you require to run such a model efficiently.

IOT

Node.js is supported on popular single board computers like the Raspberry Pi, which in turn means you can execute TensorFlow.js models on such devices too.

Speed

Node.js is written in JavaScript which means that it benefits from just in time compilation. This means that you may often see performance gains when using Node.js as it will be optimized at runtime, especially for any preprocessing you may be doing. A great example of this can be seen in this case study which shows how Hugging Face used Node.js to get a 2x performance boost for their natural language processing model.

Next steps

Now you understand the basics of TensorFlow.js, where it can run, and some of the benefits, check out some of the amazing projects people are making with it in this video playlist or see a few I picked below:

Get inspired!

For even more inspiration check out the #MadeWithTFJS hashtag on Twitter or LinkedIn to see new projects that come out every week! Happy hacking, and of course if you have any questions, feel free to reach out to me.