10 Essential Tips For The NPM Galaxy In The Node.js Universe

In my previous article here on LinkedIn about Node.js, 10 Commandments For The Beginning Node.js Programmer,

I attempted to define and describe a few of the requirements any programmer new to the Node.js world should be prepared for. In this article I would like to approach the NPM Universe, for those out there who are just becoming familiar with it.

As any Node.js programmer learns fairly swiftly, when one must accomplish various tasks be they complicated or mundane within a Node.js application, the place to gravitate to is the NPM site and begin to search for a module that will meet your specific demands. To wrap your head around the size of this community just take a peek at these charts. With way over a million module downloads a week this should tell you just how popular Node.js and NPM have become, and the critical place NPM has made for itself within the Node.js community.

It is imperative you understand the structure of how NPM works and how to make it work for you as one who is using NPM modules. No, you do not have to know every single NPM command but it would be wise to keep a few rules in your head before heading over and begin downloading NPM modules. These rules come from personal experience of making all the mistakes when I was a novice Node.js programmer. (I still continue to make them. Nobody’s perfect!)

In the following, I am assuming you have installed Node.js which also installs NPM and that you have a directory which is your project file directory. For the purposes of this article we will call our project: “NodeSmarts”.

Tip #1: Your first decision is based upon understanding how you will access your NPM modules which is either on a “global” basis or on a “local” basis - per your project. (Node itself is installed globally, and if you wish to check this after a correct install, just type node -v to see the version number from any directory.) In most cases you should install a node module per your project. This means that the module will only be available to Node.js from our root directory NodeSmarts and down. NPM modules will be installed in a directory off the root of your project automatically created and named, “node_modules”. So one of the subdirectories in your project will be NodeSmarts → node_modules. However, these will not be available from the parent directory of NodeSmarts. So you ask what if I have three projects all in different directories? Why not just install everything globally? The answer boils down to order and consistency. Each project may require different packages. Therefore, you should maintain a package.json file for each project. It is just clean programming style. Take a look here at the NPM documentation to understand the implications of global vs. local. 

For our purposes we will be installing on a local basis.

Tip #2: Before downloading your first NPM module let us cover an interesting command in the NPM lexicon, which for me at least, has turned out to be critical in the way I have been able to stay on top of the NPM world and my own installations. This command is: npm init

What npm init does is create a new package.json file, the importance of which we will get to in the next few tips. What you now need to do is open your terminal window, and get to the root of NodeSmarts directory. (cd /NodeSmarts)

Now once in your project root issue the command: sudo npm init 

(Use sudo to make sure you have rights to save this file!) You will be asked a few questions. If you do not know the answers, do not worry. A default answer will be given for you. At the end just type “yes”. Take a look at the screen below.

This creates a package.json file which will be the container for all your goodies downloaded from NPM. Remember, the beauty of this is that this is a JSON file so it can be edited and changed to meet your needs. The final JSON file looks like this:

By adding "private": true you will avoid the following warning every time you add a module.

npm WARN package.json package-name@version# No repository field.

Tip #3: By now you should be asking yourself, what the hell is so important about this package.json file? Why is this author spending so much time on it? So here I will give you a real scenario which happened to me not long ago, and you will see how much time and frustration having that package saved me.

I was working on a system in AWS which contained Node.js and a MongoDB. The code was moving along fairly well, until the point where I had to install Apache. On that day I was tired and really under pressure — (Hey! Stop laughing! We all have our excuses.) Installing Apache on AWS is a 2 minute deal. But then you have to make sure permissions for directories are set correctly. While installing, another member of the staff was also working on the system. Something went wrong, and to this day I cannot really say what happened but SSH would not come up and no matter what we did we were met with “connection refused”. For those of you not familiar with this little statement, it simply means “Guess what dope? You ain’t getting into the server!”. After an hour of panic and just getting the calm terminal message connection refusedI had to go to a backup plan. There are a few ways to correct this but for us at the time the simplest was to simply to rebuild the EC2 instance. The rebuild part was simple (if not tedious!). All the code base was also on our computers. This also included an up-to-date package.json file, which I made sure to constantly download to the local environment. So now instead of having to go back to the NPM site and find each package I needed and used, and start installing from scratch, I simply issued the command from our project root directory NPM update . Voila! In a minute or so all the NPM packages were reinstalled via the package.json file and we were back in business. 

Now you understand only one of the reasons why this file is so important to your project, and there are, trust me here, many other good reasons to keep it up to date.

Tip #4: Now that we are set with a basic NPM package.json file, you are ready to install your NPM modules. The actual installation of a module is fairly easy and straightforward. Indeed on any NPM module page on the upper right hand side you will find the installation parameter.

Though there are a few parameters you can use the ones to be aware of immediately are as follows:

npm install express is the same as typing npm i express

Additionally if you add a -g to the command the module will be installed globally (something we already discussed and not a great idea in most cases.) While on the install I will show you now how to automatically save your module into your package.json file:

npm install express --save will put the module into package.json.

What is great about this system is if you are just testing a module you may not want to immediately place it in your package.json, and wait until you are sure your project needs it. So install the first time without --save and once you have decided to put it into your project for good, then you can add it manually or much easier just run the npm install express --savecommand.

---

We are finally at the point of installing a module and using it. Some modules such as the above famous Express middleware will probably find themselves immediately in your arsenal. These are basically no-brainers in many projects. 

For our NodeSmarts project we are faced with having to create a cron job or schedule from within the code. This cron job needs to run through a MongoDB collection every 10 minutes.

If you go to NPM and type in Cron or Schedule, you will find yourself presented with a multitude of modules to chose from. The question now is which module do you chose? How do you go about making the correct choice based upon your specific needs?

Tip #5: Before downloading, define to yourself in basic terms what task you need to accomplish. You have no idea how much this simple rule will save you in time and frustration. 

Many modules are written by one or two programmers. Some programmers are capable of creating fantastic modules and by exposing methods to the one who is implementing the module makes the job of application creation that much easier. 

However, the same factor which allows for so many NPM modules to be offered to the public can also be your downfall. As I just said modules are written by programmers. Which means they are based upon the thought and programming process of the programmer. Additionally, the programmer of the module then usually writes the documentation for the module. Depending upon the skills of explaining in writing all that the module does and does not do, is something you, as the one who must implement the module must deal with. (And make no mistake. That documentation can make the difference between a module being downloaded 50 times to a module being downloaded 50000 times.) So in the documentation or description try to see if the module does what you want it to do, and it is written in an orderly and complete, I will repeat — complete — fashion.

Tip #6: In many cases there will be more than one module which will address the issue you are grappling with (simply do a search on the NPM main page on a keyword). In our case of NodeSmarts you are looking for a cron scheduler or a scheduler. So you search and one by one find the modules which catch your interest. Then read the documentation and see if the way the methods work make sense to your way of thinking and programming. If they do not, chances are the module is not for you, no matter how great it is.

While reading take the time to look at the right hand side of the page.

There are a few things you should pay attention to though by no means are they to be the only reason for your final decision. The first are the stats (boxed in red). See how popular the module is. The green and red boxes show you how well the module is kept up and updated. (You do not want to be dealing with a module written 5 years ago and never updated or touched since then.) Finally click on the github link and take a look at the following two lines in the pictures below.

Look at the stars, commits & releases. This should give you a fairly good picture combined with the NPM stats as to how much effort has gone into keeping this module up to date and how serious those who chose to download and implement the module methods in their code are about using it. Of course, do not expect modules that have been released just a week or two ago to have enormous numbers or gigantic following like the famous “Express” middleware does. 

All Github entries will not only contain instructions but you can also go into the code and peek around, if that sort of thing interests you. In other words, you can and should dig as deeply as possible into the overall status of the module.

Tip #7: By now you are staring at quite a few open tabs in your browser each populated with a different NPM module devoted to cron jobs or schedule processes. You have reviewed the ones that seem promising, read and studied the documentation, as mentioned in the above step. Now is the time to start whittling down that list to a manageable number like 2 or 3 options. Yet you still cannot decide which one is best for your needs. NP! Take them for a test ride.

Copy the install command for the cron module you want to test. Then in your terminal window issue the command by pasting it. Do not at this stage, repeat do not, use the --save option, unless you are sure that this is the module you need.

Go to your testing directory and create a small node program to test the module. Of course you should not forget to require it, otherwise your code will blow up on you. So at top of your file you will have:

"use strict";
let testcron = require('name-of-module');

The above is obviously just an example, feel free to name the let (the variable) anything you wish. Now write your code. See if you can expose the methods you require and if it does what you need it to do. Like it? Don’t like it? No big deal usually. Now do the same with the second and third module you have listed. Install without save, test with the methods, and in your mind decide how good it is.

Tip #8: Once you have decided on the module you will use, go back to the terminal window and now issue the command to install the module and put --save at the end of it. Once done that module is now in your in package.json file. 

But wait a minute! You just installed two other NPM cron modules during testing which are now sitting in your directory, and for whatever reason you do not want them there anymore. So go to your terminal window and issue this command: npm uninstall name-of-module Now that module is gone. What if you installed it in your package.json file and want to make sure it does not reinstall itself on the next update? Simple. Terminal window again and type: npm uninstall --save name-of-module Now it is gone from the package.json and from your node_modules directory. (Please note if you are at the point of dev vs. production environments with dev-dependencies then the process is a bit different. See this page for specific instructions.)

Tip #9: By now you are an expert in NPM module installation, testing modules and then manipulating your package.json file. So after a couple of weeks your package.json may look something like this, (or even worse!):

{
"name": "nodesmarts",
"version": "1.0.5",
"private": true
"description": "test system for Node Server",
"main": "nodesmarts.js",
"dependencies": {
    "async": "^2.1.4",
    "bluebird": "^3.4.7",
    "body-parser": "^1.16.0",
    "bufferutil": "^2.0.1",
    "bunyan": "^1.8.5",
    "bunyan-rotating-file-stream": "^1.6.2",
    "chalk": "^1.1.3",
    "cors": "^2.8.1",
    "cryptr": "^2.0.0",
    "dateformat": "^2.0.0",
    "dotenv": "^4.0.0",
    "express": "^4.14.1",
    "generate-password": "^1.3.0",
    "json-stable-stringify": "^1.0.1",
    "jsonfile": "^2.4.0",
    "jsonpointer": "^4.0.1",
    "locutus": "^2.0.7",
    "lodash": "^4.17.4",
    "merge-json": "0.1.0-b.3",
    "mongodb": "^2.2.22",
    "mongoose": "^4.8.1",
    "mongoose-encryption": "^1.4.0",
    "njwt": "^0.4.0",
    "node-cache": "^4.1.1",
    "nodemailer": "^3.0.2",
    "passport": "^0.3.2",
    "request": "^2.79.0",
    "secure-random": "^1.1.1",
    "socket.io": "^1.7.2",
    "split": "^1.0.0",
    "underscore": "^1.8.3",
    "underscore.string": "^3.3.4",
    "utf-8-validate": "^3.0.1",
    "uuid": "^3.0.1",
    "websocket": "^1.0.24",
    "ws": "^2.0.3"
},
"devDependencies": {},
"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [
"Medium",
"LinkedIn",
"Node.js",
"NPM",
"Programming"],
"author": "Ted Gross",
"license": "ISC"
}

But your code is beginning to make you crosseyed because at the beginning of the file you have 40 requires with variables and then methods. So here is just one of the possible answers you have to this. (It is certainly not the only possibility and may not be the best solution in many cases.) However it will make your code more manageable until you decide on your own methods.

Go to the NodeSmarts root directory and create a directory off of that one. In this case we will name it env (for environment). In that directory create a new .js file in your favorite IDE and in our example we will call it Helper_Node.js. We will put in this file on the “helper” modules. (I divide modules into categories, e.g. security modules, helper modules, database modules, etc. and etc. You are welcome to chose your own system of course, or just use one file.) Below is an example of the Helper_Node.js

module.exports = {
 mergeJSON: require(“merge-json”),
 express: require(“express”),
 bodyParser: require(“body-parser”),
 lod: require(‘lodash’),
 dateFormat: require(“dateformat”),
 ustring: require(“underscore.string”),
 Promise: require(“bluebird”),
 xrequest: require(‘request’),
 uscore: require(“underscore”),
 stringify: require(‘json-stable-stringify’),
 jsonpoint: require(‘jsonpointer’),
 jsonfile: require(‘jsonfile’),
 split: require(“split”),
 //sio: require(“socket.io”),
 //cors: require(“cors”),
 wsServer: require(‘websocket’).server,
 wsClient: require(‘websocket’).client,
 wsFrame: require(‘websocket’).frame,
 wsRouter: require(‘websocket’).router,
 //WebSocketPlus: require(‘ws’),
 //nMailer: require(‘nodemailer’),
 async: require(“async”),
 Sync: require(‘sync’),
 chalk: require(‘chalk’),
 bunyan: require(‘bunyan’),
 rotateBunyan: require(‘bunyan-rotating-file-stream’),
 fs: require(‘fs’),
 NodeNet: require(“net”),
 path: require(‘path’)
};

As you can see this is a JSON file and some modules are commented out so they do not get loaded. Each module has the syntax of:

variable name: require(module-name), 

Note the last three entries are for the Node.js environment itself and not NPM modules (so do not get mixed up).

Now comes the part where you will have to train yourself as you program. In your Node.js file back in the project root directory (or wherever), the file should begin as follows:

"use strict";
const nHelp = require("./env/Helper_Node");
const nSecure = require("./env/Secure_Node");

Now all those modules have been included when you run your Node.js program without hassle. The problem you will find is that when you wish to call a module say “chalk”, you no longer write:

console.log(chalk.red("This line will appear in red"));

instead you must add the calling const nHelp and type: 

console.log(nHelp.chalk.blue("This line will appear in blue")) 

But once you get used to it, at least in my case, it is a godsend in keeping your code from becoming spaghetti all over the place.

Tip #10: As you can see from all the above I love using NPM modules and they are usually incredible time-savers. The programming community which creates them gets 10 gold stars. Yet now comes my warning. 

Avoid NPM module madness at all costs!

One of the great things about Node.js is that it is really fast. Asynchronous or synchronous it simply does its job. Yet like anything else, the more code you dump into your project and must be loaded into memory the more the project has to process on initiation and in running. NPM module code sometimes may not clean up after itself. You may not remember to kill a process or close it when it should be closed. In other words by adding and adding to the modules used you must keep watch on the memory your Node.js application allocates to itself to run and of course the CPU usage. As you move along you will discover the joys of load-balancing, instances and other such goodies which will make things easier and your running environment better. Yet, in the end result, use as many NPM modules as you need to. Some are great for development and not required in production. When you go to production remove them from your production environment. Keep track of it all. Know what you have, what you need, what you like, what is required for production and what is only required for development.

Keeping all the above tips in mind, I am sure your Node.js and NPM experience will be great. Just don’t forget to bask in the sun every now and then. That too, is important!

Enjoy. Code away.

Other Articles in this series:

---

About the Author: Ted Gross served as a CTO for many years with an expertise in database technology, PHP and OOP. He has expertise in Virtual World Technologies & Augmented Reality. He has also published many articles on technological topics especially on Big Data & Chaos Theory (in professional journals and online @ Medium & LinkedIn). He is also an author of literary fiction, children’s books and various non-fiction articles. His short story collection, “Ancient Tales, Modern Legends” has received excellent reviews.

Ted can be reached via email: [email protected]; Twitter (@tedwgross); LinkedIn; Medium