Pipelines

Pipes are used to redirect a stream from one program to another program. We can take the output of one command and redirect it to the input of another.

We use the pipe character ( | ) to separate two commands. The output of the first command will be passed to the standard input of the second command.

command1 | command2        

In the example given below, we are passing the output of the date command as an input to the rev command.

Piping ls to less

This example pipes the output of ls to less. The /usr/bin directory typically contains a bunch of stuff, so it can be nice to use less to read the results in a more manageable way.

ls -l /user/bin | less        

You can then scroll through using spacebar, or search for a specific command using /.

Redirect vs Pipe

Though both the > character and the | character are used to redirect output, they do it in very different ways.

The > connects a command to some file.

ls -l /usr/bin > list.txt        

The | connects a command to another command.

ls -l /usr/bin | less        

The tr command

The tr command takes text from standard input, and then we provide a pattern or a set of characters we want to match and then a set of characters that we want to translate them to.

In this example, we are translating lowercase s character(s) in a file to uppercase.

We can also use the tr command to delete a specific set of characters using the -d option.

We can delete both uppercase as well as lowercase letters in one go using character class alpha, the blank spaces using character class blank.

Piping tac to rev

In this example, we are calling tac with a file and then piping the output to rev. The final result is the content of the file printed "horizontally" and "vertically" reversed.

tac file.txt | rev        

The tac command reverses vertically, and then the rev command reverses horizontally.

Multiple Pipes

In this example, we are using the cat command to feed a file to head, which cuts it down to the first 7 lines of the file and pass it to tail, which then outputs the last 5 lines of that "chunk".

The result is lines 3-7 as output to the screen.

Suppose we want to display 3 largest files in the current directory, we can do that by piping ls, sort, and head.

ls -lh | sort -rhk 5 | head -3        

First, ls -lh lists out all the files in the current directory in human readable form. That output is passed to sort, which sorts based on the fifth field (the file size). The -h option is for human readable sort (comparing 100b, 40k, 1g, etc), and the -r reverses the order so that we end up with the largest files first.

Finally, that output is passed to head, which limits the results to the first 3.

Note: This is not the preferred way to find large files! Use the du command instead.

The tee command

The tee program reads standard input and copies it both to standard output AND to a file. This allows us to capture information part of the way through a pipeline, without interrupting the flow.

command1 | tee file.txt | command2        

Thanks for Reading. Hope you found it useful : )

P.S: I had to relocate to another city, so was entangled in shifting and other arrangements. That's why was unable to post for few weeks.