Node.js cluster forking and using the 5th channel!

Why?

Why the heck am I caring about cluster and fork? Because you care about performance!

So you have created your node application. Everything works, but performance is not where you want it to be. You've heard about the cluster module and decide to use it. It improves application performance, but you still have heavyweight tasks blocking the main thread. Well, the cluster module is made to do one thing: make forking that overloaded thread easier!

Assuming your application is like most of the web applications out there, it uses the HTTP(S) protocol. Like many of the applications out there you use express, hapi, koa, strapi, or some other web framework. You soon find that the cluster module may not be enough to get your performance up where you want it to be. Chances are you have some heavy processing in your application that's really killing your applications performance. That could be any number of things, but likely culprits are: rendering HTML from JSON data; calling multiple APIs and waiting to join the results; or something else.

For this article, we are assuming you have fixed all your usage of sync methods by replacing them with async methods or promises. We also assume that you are using streams and pipe when you can. At this point you have made all the optimizations you can make, and you still have some long running task or tasks.

What?

Cluster and fork will both do the same thing. Cluster basically makes forking a bit easier. The idea here is to move your heavy processing out of your main thread. Using cluster or fork you can spin off your long running tasks into child processes. Either one will do the same job, as cluster uses fork under the hood. When using cluster or fork, there are a few things to keep in mind. So for most of the rest of this article we will refer to the use of fork and cluster will be implied. We will also be talking mainly about fork, even though more people are probably using cluster instead.

First, there is overhead of using fork. Fork creates a process, so create it once. If you end up doing fork each time you need to do the heavyweight processing, you will take on the process creation and destruction as added overhead each time you want to do heavy weight processing. This will add to the time of your heavyweight processing.

In many cases people fork a process and pass JSON data or a string to the child process, have the child process render the JSON data, and then pass the rendered HTML back to the parent process. Then the parent process handles compressing the rendered HTML and sending it back to the client. So the normal way that data is passed between the parent process and the child process is using process.on() or process.once() and process.send(). The problem with using these built in standard channels is that they only handle JSON data or strings. This is fine if the only thing you want to do is render JSON to HTML, but if you want to do more this becomes more complicated.

In my case I wanted to be able to pass binary data between the child and parent process.

How?

In order to pass binary data we need to take advantage of a binary channel. When calling fork, the default channels are stdin, stdout, stderr, and an ipc message channel.

Don’t use the default channels if you want to pass binary data!

The default channels have limitations. The first 3 channels are used for stdin, stdout, and stderr, and the 4th channel (ipc) is JSON only. Stdout is where any console.log() or logging info is written. Using stderr is where console.error() or other logger.error() or any process errors are written. Stdin is used when you want to send input to the main process from another process. For this use case, we will avoid using the stdin, stdout, and stderr channels. Again, the 4th channel, the ipc channel, only takes JSON.

The process.on(‘message’, callback), process.off(), and process.send() methods ONLY pass JSON data or strings. Node Buffers cannot be passed over the ipc channel.

If you are reading data in from the the http request object, the data you are getting is a buffer. Most people call toString() on the buffer. Converting a buffer to a string is expensive. You are walking the Buffer over Buffer.length to convert each byte to a string. Maybe being able to pass that Buffer to the child process as a Buffer and having the child process convert the Buffer to a string would save some processing. Compressing the HTML in the child instead of the parent process would also save some overhead in the parent process.

There are other use cases, and sometimes you need or want a binary channel. A binary channel would allow you to pass a buffer or gzipped compressed data between the master process and the child process.

For this example, let’s assume your child process is taking a string, calling JSON.parse() and then rendering HTML. Let us also assume that we want to compress the data in the child process instead of the parent process. Lastly, let’s assume that the parent process is caching this rendered HTML.

Something I learned when using zlib.createGzip() is that this creates a stream and streams always call end or close which we don’t want. I ended up using zlib.gzip(buffer, callback). Obviously this could be wrapped in a promise, but that’s not something we will talk about here. Also, the compressed gzip data will take less space in RAM and having the child compress it means that parent does not have to do it for all the same subsequent requests. It ends up being faster!

Squirrel! Ok, so a side note here. I sometimes just code projects because I want to learn something or I see something missing and want to create a tool that I can use to solve a missing issue. One such instance was me creating an FTP client for my chromebook. I learned a lot.

For this example we are going to look at the communication between the parent and child processes as similar to how the message channel and data channels are managed in FTP.  The difference is we will be keeping the data channel open. If you are not familiar with FTP that’s ok. Just understand that there is a main channel where messages are passed between the FTP server and client. Common messages are list directory and change directory. In FTP the data channel is used for uploading and downloading files. 

In node.js fork() you typically get a message channel. Here we are going to get a data channel.

In using fork, you change the values of the “stdio” option when calling fork. It’s not well documented how the child_process.fork() 5th channel works. The documentation talks about passing “stdio” options but they don’t tell you that the child process also has to do work. Through a lot of trial and error and reading and re-reading, I found I had to try a few things. Ultimately, when calling form() you pass the following to stdio: [ 0, 1, 2, 'ipc', 'pipe' ]. So the options you would pass to fork would look something like:

const childProcess = require(‘child_process’);

const options = {
    cwd: process.cwd(),
    env: process.env,
    stdio: [ 0, 1, 2, 'ipc', 'pipe' ]
}; 

const cpuHeavyProcess = childProcess.fork( ‘heavyWeight.js’, 
    options );

This would be how the parent process is set up. However, the documentation does not talk about how the child process needs to be set up to work with this parent. I actually found only one example of how this worked, but even that example did not go into sufficient detail.

In child process, “heavyWeight.js”, you need to connect to the socket channel by doing the following:

const Socket = require(‘net’).Socket;
const channel = new Socket({
    fd: 4
});

Notice that for the child process, we use fd as 4 to indicate the 5th channel. Why 4 instead of 5? Think 0 based array. That means stdin is 0, stdout is 1, stderr is 2 and the ipc JSON message channel is 3, so this becomes fd 4. 

Yes folks, you now have channel 5!

You might think, but won’t these child processes all be using fd 4 overlap? No, the fd 4 is specific to each child. I don’t think that the fd here maps to a file system file description, but some mapping is happening under the hood. 

One gotcha when using the 5th channel is that streams call close. So if you pipe data to this socket channel from a stream, the stream will call close and close the socket. I have not found a way to re-open that socket without restarting the child process. In this case, I recommend avoiding this:

someStream.pipe(channel);  

That will call the close or end event on the channel which will send a null. The resulting null will close the channel. So if you are doing something like gzipping the data in the child process, you will want to use the callback flavor instead of stream flavor: 

new ReadableStream(yourData).pipe(gzipStream).pipe(request);

The problem with the code above is that each time you send the same data, you are compressing the same data. So if you are storing your data in memory, then you might want to consider compressing the data only once.

The resulting child process looks like this:

const zlib = require(‘zlib’),
    Socket = require(‘net’).Socket;

const channel = new Socket({
    fd: 4
});

// we just got a message from the parent
process.on(‘message’, (data) => {

    // compress the data without a stream
    zlib.gzip(data, (err, compressedData )=> {
        const timerID = setTimeout(() => {
            throw ('Took too long to write buffer!');
        }, 250);

        const resolveCallback = () => {
            clearTimeout(timerID);
            dataChannel.removeListener('drain', resolveCallback);

            // so now tell the parent process it has all the data
            process.send({ status: ‘completed’ });
        };

        // we do this as sometimes the buffer is not written immediately
        // so the set timeout above allows us 250ms to drain, which is a long time
        dataChannel.on('drain', resolveCallback);
              let bufferSent = false;

              // write the binary data over the binary socket channel             
              bufferSent = channel.write(compressedData, function(err) {
                   // all data has been written to kernel buffers
                   // so now tell the parent process it has all the data
                   if ( bufferSent ) {
                       resolveCallback();
                  }
              });
         });
    });
});

The resulting parent process looks like like this:

const childProcess = require(‘child_process’),
    zlib = require(‘zlib’);

const options = {
     cwd: process.cwd(),
     env: process.env,
     stdio: [ 0, 1, 2, 'ipc', 'pipe' ]
};

const heavyWrightProcess = childProcess.fork(‘heavyWright.js’, options);
 
 heavyWrightProcess.stdio[4].on(‘data’, bdata => {
     console.log(‘got data ‘);
     console.log(zlib.gunzipSync(bdata) );
     console.log(‘end of current buffer‘);
     process.exit(0);
 });

heavyWeightProcess.send(‘this is a short string of data’);

Obviously, this is not your full program, but it’s a start. So one thing to play with is using fs as the source of the data and and reading in a file. Once you read in the file, you then would do something like fs.readFile() and then in the callback call heavyWeightProcess.send( file-data.toString() ). The result will be the file being sent to the child process and compressed and then back to the parent process and uncompressed. The next step from that would be to have the child process read in the file and send a gzip version of it or to have it render some html from data passed from the server to the client.

In implementing an http server with children, you will need a queueing system to handle the workers. If you have 10 child workers, and you get an 11th request, you need to queue the 11th request and manage the queue. You don’t want more calls to process.send from the parent to the children than there are children, otherwise data will get jumbled. In this case heavyWeightProcess becomes an array of children and you build out a queue management system for the incoming requests and the child processing data.

One thing to note, is that sometimes, even though the drain event was fired, the parent process might not get all the data from the child when the parent process on('message') callback gets called. In those cases a fallback needs to be put in place to handle this. In my case I just send the uncompressed data as well. So for those few requests that have this issue there is a fail over. Best thing is to keep track of how much data is sent and track that that amount of data was received.

Queue system and http server is a bigger discussion. Good luck!