All About JavaScript (JS)

“Why??Oh God why?”?

This is all too often the refrain heard from developers when forced into a corner by JavaScript.

So why? Why has this, quite honestly, Frankenstein’s monster of a language managed to grow so huge in its popularity?

It's pretty much the only language out there that you can use to build an entire product from top to bottom. Surely not! I hear you say. Yep. Ever heard of Pouch DB? The JavaScript version of Couch DB. Quite frankly, something that shouldn't exist (I’m being a bit over the top here as it syncs with Couch DB in the end, but you get my drift).

My last two blog posts have been quite high-level, and I've talked about software architecture and design from a big-picture perspective.

For this blog though, I thought I'd indulge my inner nerd, and talk about it at a relatively low level. I'm sure that Mr Torvalds will laugh at this, as someone who considers the C programming language to be high-level, but we don't all think in assembly.

In order to understand JS’ rise to fame we need to look at its history and purpose. For its origin, the clue is in the name - this is a scripting language. You've probably heard of quite a few like Python and Bash.

So, what's the deal? What's the big difference? Predominantly, it's in the way that the code is turned into machine instructions. Traditionally, programming languages are compiled, this means that another program, called the compiler, takes the code that you've written and turns it into machine ready instructions. This means that you can distribute it to anyone?and, providing you have compiled the program for their machine, they can click and run it.

Scripting languages, on the other hand, are run through an interpreter when the program is actually executed. This is known as jitting, or just in time-ing.

Node is a perfect example of this. It takes a JavaScript file and line by line translates into host machine instructions. You could feed it absolute nonsense on one line and it wouldn't error until it actually hit that line.

If you give a JavaScript file to a non-technical person, they are likely not going to be able to run it.

What's the benefit? It's quick and easy. For illustration, let's compare “Hello, World!” in a couple of different languages.

C

#include <stdio.h>

int main(int argc, char **argv)
{
??? printf("Hello, World!\n");
??? return 0;
}        

GO

import 
??? "fmt"
)

func?main() {
??? fmt.Println("Hello, World!")
}        

Now, admittedly, hello world is a very simple program, but let's compare this to some scripting languages.

JS

console.log("Hello, World!");        

Python

print("Hello, World!")        

Bash

echo "Hello, World!"        

Yep, all of them on a single line. When you combine this with the fact that you can write mobile applications, web applications, APIs, databases, and pretty much everything else you can think of with JavaScript, it becomes quite obvious why it's so popular.

Challenge

I'm not JavaScript’s number one fan and was seriously sceptical that the language was suitable for more than just making a few divs jump around on a website. Now whilst I will still gravitate to more type safe, optimised and compiled languages, I have to admit that I have slightly changed my tune. Or more specifically, been forced to.

Does this mean I'm a JS convert? Nope -?but I've certainly accepted that its capabilities are much broader than I thought.

I recently had the fortune (or misfortune) to use it for a task that I would normally assign to a language like C or Go, and so was forced to flip it over and dig into its shady underside.

The following scenario is based on that real world situation.

Let's assume that we're making a game that uses the Xbox Kinect (remember those) to live stream the relative position of players’ joints in real time. Now to make it slightly trickier, we're going to have hubs of players that are all connecting to the same game host that will be responsible for streaming their data to our server. This means that alongside the X, Y, & Z data for each of the joints, we want a timestamp so that we can synchronise it on our server and for our viewers. We will collect this data in the cloud, stream the X, Y, & Z back to any viewers and make sure it is all in sync.

One last thing to consider is that we want to open this up to as many people as many different situations as possible. This means that we want to optimise, and reduce as much as possible, the amount of data that we are sending over the network.

Okay, let's get into it and see what the data is going to look like.

First a quick dive into binary, we need to know the difference between a byte and a bit. A bit is either a 1 or a 0, and a byte is a sequence of eight bits. It's just like any other number system, except you only have two digits to play with, the 0 and the 1. So 0 is 0, 1 is 1 and 2 is 10, 3 is 11 and so on and so forth.

This means that the maximum value for a byte is 255 or:

11111111        

Now, it just so happens in our imaginary game, the maximum value that either an X, Y, or Z value can have is 15. Isn't that convenient!

byte x = 7;? // 00000111
byte y = 3;? // 00000011
byte z = 15; // 00001111
// notice the most significant 4 bits (left 4 digits) are always 0        

Why would we send data which we know is always going to be zero?

It turns out that we don't need the full eight bits of a byte to carry this information, and seeing as our aim is to decrease the amount of data going over the network by as large amount as possible, we can pack two values’ worth of data into one byte.

byte xy = x << 4 | y;        

What have we actually just done? Let's break it down a bit.

byte x = 7; // 0000011
byte y = 3; // 00000011

// we bit shift left by 4 bits
byte preparedX = x << 4; // now 01110000

// we OR the bytes together
// this takes the two inputs and if there is a 1 in either column
// the output value will also have a 1 in that column
byte xy = preparedX | y; // now 011100111        

The left most 4 bits of xy are now the value for x and the right most 4 bits of xy are now the value of y.

Imagine that we are streaming, 30 joints worth of information, 60× per second per player. Each joint has an X, Y?and Z value. Without packing the information together, this is three bytes per joint. This is 90 bytes per player per frame, 5,400 bytes per player per second. Imagine now you have 10,000 consecutive players. This is 54,000,000 bytes per second or (51 MB). Now, this is a slightly unrealistic scenario, but if we follow the logic, using our data packing method?we can save 25.5 MB every second. This is a huge saving!?

The timestamp value is an interesting one.?We could just ping 64 bits of data over the network every second or so which would give us the ability to push the millisecond unix time stamp but?clearly that is not enough for our mega, world beating, record busting game which needs to be mega optimised!

If we take a look at the current unix millisecond timestamp (at time of writing) 1679326283575 (this is the number of milliseconds that have passed since 1st Jan 1970) we can see that the first 2 bytes are all 0s. This means that in order to send the timestamp we only need 48 bits to actually go over the network or 6 bytes.

Now to make things slightly more complex, we are going to send the higher 4 bytes of information from a master server and then all the other servers where our players are will just send the lower 2 bytes of information. This does mean that we can reduce the amount of data even further.

Now in JavaScript, we don't have access to byte level information, or more explicitly, numbers in JavaScript are always 64-bits. This means that each number we have in JavaScript is eight times larger than a C style byte that we've been dealing with above.

However, assuming that this packing was done in a language which could do it at the byte level, we are now responsible for collecting this information with our JavaScript app and streaming it to our JavaScript back-end.

So let's get going. First off, we're probably going to have some kind of callback JavaScript app that will be fired when there is data available from the underlying system.

const?dataReady = (data: number[]): void => {
??? // Do something with the data
??? // Stream it to our server
};        

Now, if each time this function is called we receive 90 bytes worth of information (three bytes for each of the 30 joints) this implementation already isn't that ideal. The JavaScript (Typescript here) number type is always 64 bits (eight bytes) as discussed above, which means for the 90 bytes that we are being sent by the underlying system, we're actually eating up 720 bytes completely unnecessarily. We can minimise the impact by doing the following:

const?dataReady = (data: number[]): void => {
??? // Clip each 64 bit number into the 8 bit byte
??? const?byteArray = Uint8Array.from(data);

??? // Do something with the data
??? // Stream it to our server
};        

Fine. We still aren’t as efficient as we absolutely could be, but at least we’re now only sending the 90 original bytes over the network.

Next job. Now we need to decode these bytes back again in our server and save the original values:

// we need some kind of structure to hold our unpacked data
// we will produce one of these for each byte sent
type Unpacked = {
??? item1: number;
??? item2: number;
};

const unpackData = (data: Uint8Array): Unpacked[] => {
??? const unpackedCollection: Unpacked[] = [];
??? for (const byte of data) {
??????? // item 1 is easy we just move the left 4 bits right 4
??????? // so 01110011 becomes 00000111 or 7
??????? const item1 = byte >> 4;
?????? ?
??????? // item 2 is slighly harder
??????? // we need to turn the left 4 bits all to 0s
??????? // we can do this with XOR (^)
??????? // start with item1 and move the bits left 4 again
??????? // so 00000111 becomes 01110000
??????? // now we XOR it with the packed value so
??????? // 01110011 ^
??????? // 01110000 =
??????? // 00000011
??????? // now we have the original y value 3
??????? const item2 = byte ^ (item1 << 4);
?????? ?
??????? const unpacked: Unpacked = {
??????????? item1,
??????????? item2
??????? };
??????? unpackedCollection.push(unpacked);
??? }
??? return unpackedCollection;
};

// do some more stuff now we have the original data        

Alright! You may be asking yourself at this point what's the big deal? Hasn't JavaScript performed exactly as any other language would? Well, apart from the fact that we've used up a bit more memory than we should have (and are likely a bit slower about doing it), you’d be exactly right.

But we're not quite done yet…

Next up, we need to deal with unpacking the timestamp values. This should be pretty simple at least for the full value. However, remember that for this value we are not dealing with one single byte but 6 and we will be sent them in different chunks. We have to convert it into a single number. Fortunately, a quick Google suggests DataView might be our friend:

// Timestamp 1679326283575
// This is 0000000000000000000000011000011011111111101001011000011100110111

const testBytes = Uint8Array.from([55, 135, 165, 255, 134, 1, 0, 0]);

const convertTime = (data: Uint8Array): number => {
??? const dataView = new DataView(data.buffer);
??? return dataView.getBigUint64();
};

console.log(convertTime(testBytes));        

Oh @!&$. When we run this, we seem to be getting back a value of 4,001,349,310,825,431,040. What is an earth is going on! A quick dig into the underlying architecture, and we soon discover that we haven't dealt yet with endianness.

I'm not going to go into endianness in too much detail but it specifies the order in which a computer stores the underlying bytes that make up a number. Big endian systems store the high part (most significant byte) of a number at the lowest computer address. You can think of this like storing the bytes left to right. In this scenario, our number would look like:

// Address 0? 1???????? 2??????? 3???????? 4?????? 5???????? 6?????? 7

// BE?? 0???? 0???????? 1?????? 134?????? 255????? 165?????? 135???? 55

// 00000000 00000000 00000001 10000110 11111111 10100101 10000111 00110111        

Little endian systems store of a high part of a number at the largest computer address, essentially in reverse. In this scenario, our number would look like:

// Address 0?? 1?????? 2?????????? 3?????? 4??????? 5?????? 6?????? 7

// LE? 55???? 135????? 165?????? 255????? 134???? 1???????? 0??????? 0

// 00110111 10000111 10100101 11111111 10000110 00000001 00000000 00000000        

We’ve been running this on a big endian system so we need to flip the bytes. Fortunately, DataView has support for this. The second argument of the getBigUint64 function is a true or false value when set to true, which will convert the number as if it was in little endian format. Let's update our function:

const testBytes = Uint8Array.from([55, 135, 165, 255, 134, 1, 0, 0]);

const convertTime = (data: Uint8Array): number => {
??? const dataView = new DataView(data.buffer);
??? return dataView.getBigUint64(0, true);
};

console.log(convertTime(testBytes));        

Cool! Now we’re getting our timestamp as expected. So now we need to hook up our handlers and combine and unpack the timestamp. We are also going to throw away the top 2 bytes away as discussed above as they are always going to be 0 so we will be getting 4 bytes from the master server and then 2 bytes from the game servers.

let highBits: Uint8Array = Uint8Array.from([]);

const recieveHighBits = (data: Uint8Array): void => {
??? // most significant 4 bytes
??? highBits = data;
};

const recieveLowBits = (data: Uint8Array, serverId: string): void => {
??? // least significant 2 bytes
??? if (highBits.length !== 4) {
??????? // have not received the latest time from master server
??????? return;
??? }

??? // most significant bits is 32 in length (8 * 4)
??? const highNum = new DataView(highBits.buffer).getUint32(0, true);
??? // least significant bits is 16 in length (8 * 2)
??? const lowNum = new DataView(data.buffer).getUint16(0, true);
??? // put them together
??? const time = highNum << 16 | lowNum;

??? // save the time for this server
};        

And the result is: -5,929,161. Not what we wanted or are expecting. There's only one thing for it, let's break out Go and see if we can puzzle out what on earth is happening here:

package main

import (
?"encoding/binary"
?"fmt"
)

func main() {
?var testHigh = [4]byte{165, 255, 134, 1}
?var testLow = [2]byte{55, 135}

?var highNum = binary.LittleEndian.Uint32(testHigh[:])
?var lowNum = binary.LittleEndian.Uint16(testLow[:])
?var finalResult = highNum<<16 | uint32(lowNum)
?fmt.Printf("%d\n", finalResult)
}        

Now running this produces a different result, 4,289,038,135. However, we are good Go programmers and there is something very obviously wrong with this code.

We have read the high 32 bits of our timestamp so we have:

10100101 11111111 10000110 00000001        

and we can see that the function `LittleEndian.Uint32` returns a `uint32` or in other words 32 bits (4 bytes) of memory space in which to save this data.

Next we perform the action `highNum << 16` which should produce the number:

10100101 11111111 10000110 00000001 00000000 00000000        

Ah! We have added an extra 2 bytes to the amount of memory required to store this number, we now need the full 48 bits.

In reality we have just created the number:

10000110 00000001 00000000 00000000        

and lost the most significant 2 bytes of the original data. We then perform the OR action so:

highBits << 16 | uint32(lowBits)        

which produces the result 4,289,038,135.

What's the fix? Simple, just ask Go to allocate us more memory. We cannot ask for 48 bits directly so we will have to go straight in with the 64.

package main

import (
?"encoding/binary"
?"fmt"
)

func main() {
?var testHigh = [4]byte{165, 255, 134, 1}
?var testLow = [2]byte{55, 135}

?var highNum = binary.LittleEndian.Uint32(testHigh[:])
?var lowNum = binary.LittleEndian.Uint16(testLow[:])
?var finalResult = uint64(highNum)<<16 | uint64(lowNum)
?fmt.Printf("%d\n", finalResult)
}        

Woop?woop! We have got there thanks to Go. So, what on earth is happening with JavaScript?

Well, here at least, some of my prejudices are vindicated.

There is no in-built way to ask JS for specific memory sizes. All numbers are 64 bit floating points in the background, and what is more they are signed! Essentially that means all numbers can be both positive or negative but this sacrifices the most significant (left most) bit. And as they are floating point numbers, they also have reduced capacity when it comes to integers (some space needs to be sacrificed to encode the value after the decimal place).

If we look at the getUint32 inline documentation it specifies that it returns a JS number, and so we would be expecting 64 bits.?But what if, in reality, it is only returning 32 bits of information just like Go? Not great if true as this fact is hidden beneath the very generic Number type. How can we prove this? Well, if this were the case what would the time variable actually be in binary?

// most significant 4 bytes
00000001 10000110 11111111 10100101

// least significant 2 bytes
10000111 00110111

// bitshift most significant << 16
11111111 10100101 00000000 00000000

// OR with least significant 2 bytes |
11111111 10100101 10000111 00110111        

Ok, well, what is the final binary when converted into decimal? We can just throw this into an online binary to decimal converter and we get: 4289038135 unsigned value and -5929161 signed value.

Wow! This proves it. JS is capping the memory to 32 bits and is interpreting the result as a signed number and seeing as the most significant bit is a 1 this means the result is negative.

How on earth do we fix this? Turns out JS has something called a BigInt that we can use when the space allocated to a Number is too small. Let's apply our fix:

let highBits: Uint8Array = Uint8Array.from([]);

const recieveHighBits = (data: Uint8Array): void => {
??? // most significant 4 bytes
??? highBits = data;
};

const recieveLowBits = (data: Uint8Array, serverId: string): void => {
??? // least significant 2 bytes
??? if (highBits.length !== 4) {
??????? // have not recieved the latest time from master server
??????? return;
??? }

??? // most significant bits is 32 in length (8 * 4)
??? const highNum = new DataView(highBits.buffer).getUint32(0, true);
??? // least significant bits is 16 in length (8 * 2)
??? const lowNum = new DataView(data.buffer).getUint16(0, true);
??? // put them together
??? const time = BigInt(highNum) << 16n | BigInt(lowNum);

??? // save the time for this server
};        

Phew! Finally, we are getting our time value correctly. There is one thing to point out from this however - BigInts are not the most optimised things to have around. A quick Google lets us know that a 17 bit binary number can take up to 24 bytes of space. 24 bytes!!!! That is an insane increase from 17 bits to 192.

Conclusion

We've gone relatively deep here so apologies if this one left you sleepy! However, what is the point of all this evaluation?

Well, at the risk of sounding like a broken record it is the same old, right tool for the right job. I have to admit that I have been impressed at JavaScript's abilities when it comes to what are essentially low-level operations. However it feels like attaching a huge bucket onto a tiny digger. In other words, perhaps not quite appropriate. My previous posts have talked about high level design and putting together the right architecture for an entire system, or ways in which systems can be improved, however, this clearly extends down to the small details too.

At the risk of being controversial, there are too many companies who proudly state that their entire stack is JavaScript or Python when in reality this is not a boast but a sign of poor design.

The reality is that each language and SDK has its own strengths and weaknesses and each should be considered for the job at hand. Combine this with a micro-service architecture (see previous blog post) and you have a really powerful tool for building absolutely anything.

So am I going to embrace JS 100%? Nope. Am I going to give it up entirely? Nope.

Ask me to build a nice web app and yes, I will more than likely use it. Ask me to write a server that unpacks game data like we have done above. Yup. I'm going for Go, C++ or an equivalent and you should too!