UNIX : Where there is a shell, there's a way
Abhishek Singh
Data Engineering Lead @ NatWest Group | Data Driven Apps, Data Products
Whether you talk about flat files, talk about system processes, talk about scheduling, talk about automation, at last you will end up praising UNIX knowingly unknowingly. Power of UNIX packed with different kinds of shell (bash, csh, ksh etc) is indispensable especially when you are trying to move the mountain inside your system by few lines of magic.
This ocean of features brings variety of uses under the same umbrella.
-
Tired of writing the same command over again and again
Join multiple commands into one liner which could be a combination of lengthy repetitive command sequences .
yes, I chose to echo "Hi Abhishek|UNIX is FUN|Yes|No"|awk -F"|" '{print $1,$3,$2}' OFS="," to get Hi Abhishek,Yes, UNIX is FUN -
Build a generic process
Generalize the sequence of operations done on one set of files having data, into a procedure which could be used for other similar types of data too. yes, may be passing as argument. -
Actual command + Your command + Some more command = What you wanted
Embrace the talent of pipes. Pipe multiple commands together into one to create various utilities. You would eventually end up inventing your own one liner.
-
For someone Lazy enough, try aliasing
Simple shell scripts or commands can be written as shell aliases, still this script can be made available to all users and all the processes. Bash users can add user-defined aliases to ~/.bashrc file
Two of my favorite aliases , alias ..='cd ..' alias c='clear'
-
Build your own envioronment
Enfold programs inside an environment that you can control. Best cases could be setting environment variables, switching to a special directory, creating a configuration file, redirecting output, creating logs, and then running a program. -
Impromptu data for Data Scientists
Create customized datasets on the go, and invoke applications like matlab, sas, idl, gnuplot to work upon them. Think of an ETL pipeline too.
-
Hurrying to send into production
You can build prototypes rapidly. Build a prototype, test on small input, again test output and scale it to production. Remember, with great power comes great responsibility.
Other Typical uses
-
Power on systems
System boot scripts (/etc/init.d)
-
Automate mundane tasks
System administrators uses for automating various aspects of computer maintenance, user account creation , permissions, access, authority, monitoring etc. -
Want to send birthday emails exactly at 12 o'clock
Create Application startup scripts, especially unattended applications which can scheduled. UNIX comes up with various inbuilt schedulers which on a priority basis gets your daily tasks done. Don't forget to check cron or at.
Bottomline, Automate Automate Automate and Automate!
Data Engineering
8 å¹´very well written.
Adobe Experience Platform | Adobe Experience Manager(AEM) | Adobe Analytics | Java | Java Script | Azure DevOps | UX/UI | Angular | React
8 å¹´Well Abhishek bhai