HOW THE SHELL WORKS INTERNALLY

A shell is simply a program that takes commands entered by the user and calls the operating system to run those commands. There are several steps the shell follows, in the background, to achieve this. This article will help pull off a layer from the shell interface and peek inside it and explain how the shell works internally.

It is important to note that the shell is a command-line interpreter. It reads input from the user through an interface called a terminal emulator. Take a look at the picture below:

Notice the way the shell processes input from the user. When the user enters a command, the shell takes in and sends it to the kernel. According to Wikipedia, the kernel is a computer program at the core of a computer's operating system and has complete control over everything in the system.

A shell implementation is divided into three parts: The parser, the executor, and the shell subsystems.

The Parser: This is the component that takes the command line and splits it into tokens. We are only pulling off a layer, remember! Consider the command below:

ls -l *.c        

After the command above is typed into the shell and the ENTER key pressed, the shell takes this input and stores it into an array of characters including the newline character and \0 - which shows the end of a string.

The addresses provided in the image are arbitrary. The shell stores the command line in memory. When this step has been carried out, the shell takes this command line and passes it to the parser. The parser does numerous processing?such as removing?the newline character "\n". In this case the newline is overwritten with?null character - \0 - to show the end of the line. The shell separates the command line into an array of strings using the "space" as the delimiter. This process is called tokenization and each string is referred to as a token.

This is what it will look like:

{"ls", "-l", "*.c"} // An array of strings        

Once this has been achieved, the shell is happy with your input and has tokenized it accordingly, it sends this array of strings created above to the next part, the executor.

The Executor: This takes the Command Table (in the case, the array of strings) prepared by the parser and create a new process from it. Before a new process can be created, it will take the first element of the Command Table above, which is the command (all other elements are arguments passed to the command) and does some searches. It checks the built-in commands.

?? Now, you wonder, where are the builtin commands and how do they vary from other commands?

The shell comes with some built-in commands that make life easier for the user. These built-in commands are implementation-specific. They are usually stored in an array of structs. A struct is a user-defined data type. In this case, the struct usually consists of two elements, the name of the command (builtin) and a pointer to the appropriate function. This may be different depending on the programming language used to achieve this. So all the shell builtins are stored in an array to be retrieved when necessary.

This means that when a command is to be executed by the executor, it first checks the command in the builtins. If the command is present, it runs the appropriate function to execute that, else, it checks if the command provided is not a path, then it tries to look for the executable file with the command's name in the environmental variable PATH. This variable contains a list of paths where programs are stored. If you want to see the environmental variables present on your system, enter the command "env" and it will come rushing out to you. If the command is found in the PATH variable, then it creates a new process to run the command.

Here comes a question; What the heck is a process?

A process is simply a running program.

Taking a look at the example above, ls is not a built-in command. To confirm that, see below...

I used the type command to check the path to the command, which if I had done for the built-in command, the type command would have returned "builtin". I was able to check if the path was valid, and it is. Commands like exit, cd are builtins.

So you can see the difference. You can also try "type type" to see what type of command is it ??.

So ls is not a shell builtin, it is searched through the PATH variable. To view the content of the PATH variable, use "echo $PATH":

So from what you could see, you noticed that it contains a list of paths separated by a colon ":". Each of these will be searched accordingly until ls is found. From above, you can see that ls is found in the /bin/ directory. Once it is found, something interesting happens; the array of strings I referred to as the Command Table is modified:

{"ls", "-l", "*.c"}

becomes

{"/bin/ls", "-l", "*.c"}

Now the search is done and the Command Table modified, then a new process is created using the fork system call.

What is a system call? Now you will be convinced I'm a Psychologist right? Reading minds...??

Wikipedia says, "A system call is a programmatic way in which a program requests a service from the kernel of the operating system. It provides an essential interface between a process and the operating system" It is just a way for a process to use certain services provided by the operating system. And remember that the shell is a process!

So the shell requests the operating system for the service of fork. The system call?fork?(man 2?fork) creates a new child process, almost identical to the parent (the process that calls?fork). Once?fork?successfully returns, two processes continue to run the same program, but with different stacks, data and heaps.

Now two processes are running, the shell (which is the parent process) and a child process to looks very identical to the parent process. Immediately this child process is created, the shell makes another system call to the execve service which simply executes programs. It takes three arguments: the command name (or file name to be more specific), the Command Table already prepared, and the environmental variables (which is an array of strings also).

execve("/bin/ls", Command Table, environ)        

The execve system call takes place within the child process while the parent process waits by calling the wait system call. The wait system call waits for the child process to complete its life span and then stores the status code of the execution, whether it was successful or not.

Now we have understood what the executor works, let us look at one more thing:

You must be processing the whole steps in your head; The parser takes the command line inputted by the user, splits it up into tokens stored in an array of strings and sent to the executor. Next, the executor takes this array and checks if the first element of the array, which is the command name, is a builtin. If it isn't, it checks a list of paths maintained by the PATH environmental variable. Once the file is found and it is an executable file, it creates a child process to execute the program while the parent process waits for the status code. Once successful, similar output to that above is seen on the screen... Okay I understand it now, but how did the shell understood *.c?

This is where the third part of the shell implementation comes in - the shell subsystems. These are other subsystems that make up the shell such as the environmental variables, the wildcards, and the subshells.

We won't be going into details about what they do because I only promised to peel off a layer ??. But the wildcards play an important role in achieving that which you asked. I hope I have shed more light on this and answer some of your questions. If there are still questions you have, you can ask them in the comment section and I won't disappoint you.

Thanks for Listening ??

If you are interested in trying to implement a shell, you can check out the simple version my partner and I developed and contribute to it.