How can you optimize your scripts to avoid wasting system resources?

1. Use efficient data structures 2. Minimize disk I/O operations: 3. Avoid excessive network I/O: 4. Optimize database queries: 5. Use multiprocessing or multithreading: 6. Manage resources effectively: 7. Monitor resource usage:

Efficient algorithms and data structures Proper memory management, releasing resources when not needed Code profiling to identify bottlenecks Minimize I/O operations and batch when possible Implement caching for reuse of computed results Utilize parallelism and concurrency Choose optimized libraries and frameworks Lazy loading for resources Minimize loops and iterations Avoid global variables for better resource management Optimize database queries and use indexes Implement efficient error handling to prevent resource leaks Keep software and dependencies up-to-date Consider compiled languages for better runtime performance Regularly monitor and profile script performance

In my daily work I use PowerShell to manage Windows systems, it is really useful to install and update software, manage users and groups, configure systems. PowerShell is fully integrated with Windows. I believe there are general rules that can serve to optimize scripts of any language: -Avoid using nested loops -Avoid using global variables -Write readable and commented code -In PowerShell, avoid using arrays whenever possible. Arrays have a fixed size and each addition to the array creates a new array. Use a generic list instead, for example: $results = [System.Collections.Generic.List[object]]::new() Adding arrays is much slower than performing additions to List<T>

Scripts constitute a routine aspect of IT operations, forming a natural response to automate various processes such as: - Backup - Database indexing - Updates - Cache cleaning This practice is driven by two primary factors: - The absence of dedicated tools for a specific task - The intricacy of the process itself - Scripts provide a swift and dependable solution for these tasks - Certain IT professionals, including myself, take pride in developing proprietary, in-house scripts, particularly on Linux/Unix systems. In my experience working for a state company with a PostgreSQL database, we crafted numerous scripts for backup and indexing. These scripts proved so effective that they remained unchanged for over a decade.

Probably this is bash centric: Avoid piping too many commands. I suggest three at most Avoid using e.g ls * when you can simply * to expand to list of files in current directory Use finite loop, e.g is it really necessary to do while true do loop? Can it be replaced with 5 times for loop? Use command with most option possible e.g use find instead of ls -R | grep to find files recursively xargs is awesome! Use it when needed!

Optimizing scripts for resource efficiency. -writing clean code, -applying parallelization, -putting caching in place, -optimizing database queries, -containerizing apps with tools like Docker *Use auto-scaling and Infrastructure as Code (IaC) to dynamically modify resources and ensure consistent deployment. *To equally distribute traffic, use load balancing, and carry out routine optimization assessments. *Put security first by using best practices and conducting regular audits. *For team comprehension, thoroughly document everything. Utilize resource tracking tools such as AWS CloudWatch. *These procedures guarantee scalable, secure, and effective scripts that maximize system resources.

Using find and piping the output to xargs is a lot faster than the -exec option because it spawns only one process instead of a process for each file found. Unfortunately, the command you want to execute needs to support receiving many files in the command line, e.g. grep or rm.

Using built-in commands makes for optimized performance and resource usage in most cases. Whenever this is possible and the situation allows it, it's best to test two different versions of the script in order to make sure you are using the one that is optimized.

I think being able to leverage the built-in commands and functions and the coreutils (core utility commands such as tr, cut, find, grep) is very important as those have been battle tested and perfected over a long period to be highly efficient and performant. They are also evolved and maintained very diligently. Shell scripts shine here compared to other general-purpose programming languages as there is only so much you can do from a language perspective, so you are forced to use the coreutils and other commands by design.

I recently leveraged the /RegExp/ pattern instead of instantiating one using new RegExp(str) b/c allegedly my pattern is precompiled and more efficient. I gotta say, it appears to work.

Using variables and/or arrays for storing different types of data will probably always be the best way to avoid unnecessary operations in your scripts. Always double check your scripts for potential places, where you can input another variable or array.

Avoid unnecessary iterations. Use the advantage of functions. Stick to the minimum you need. Do not script what can be achieved with two commands, except when you have to automate. Keep it simple and stupid.

Another technique I use frequently in my scripts to make them more efficient is to make smart use of variables. I use variable to remember small bits of details for use further down. I also avoid storing entire file contents in variables to be processed in later steps. I found this to be a foot gun in cases of large files as it can lead to massive memory use, even OOM conditions in extreme cases. Streaming works very well for files of all sizes. I tend to use sed and awk in that order where possible and fall back to line based input using read in a while loop. It is time to move to Python (may be different for you) if it gets too cumbersome for advanced processing.

Think ahead for your data uses. I see too many beginner scripts making too many calls in the same script for the same data. It's not necessary to pipeline data directly from a call for it all the time. Call it once, get what you need, use that data repeatedly, release it.

More effective tools less wastage of any resources ? Decide which tools to use ? Making sure to setup a complaint strategy on how to use the tools Example : ITSM is the way to report incidents Git versioning is a way for continuous development and delivery Security protocols are must to follow before going to production ? Designing effective and standard way to use the tools ? minimise manual effort by introducing automation everywhere

Making my scripts idempotent is a technique that gives me many advantages. It can be done simply by adding checks for start and end conditions before and after each of the processing. First, it optimizes the runs by skipping already processed cases. Second, it becomes dead simple to add concurrency using the built-in parallelism in some commands or via background processes as the checks act as guards. Finally, idempotent logic is also very simple to debug for run-away cases like infinite-loops and control the amount of parallelism (which is just another check).

It's a good idea to periodically update your scripts with short informative error messages, which might save you some troubleshooting when using scripts you haven't touched in a while. Also tools like shellcheck are extremely helpful in writing scripts without headaches from a random syntax mistake on line 146.

Apart from the regular testing and debugging techniques using the stderr, I find using the PS4 with the xtrace option to create effective continuous logging that can help during development surely as well as create the audit records for production too. BATS is also a nice tool to add unit-testing for your Bash scripts, but I have not used it much.

Create a flowchart to optimise the structure of the script. And don’t be forced to make the script longer than it needs to be unlike the rules for this answer.

Try to cut your script into steps that are independent from each other's this way you may be able to scale part of your scripts if needed, and don't feed your app with bulk of data it will kill all the resources try to feed it data little by little and save where you left so you can resume at any time!

First reality check... Is a more efficient script going to improve margins? Give you a bonus that's a significant fraction of that improved margin? Get you out to karaoke on time? Don't bother unless you can answer yes to two or more. Second... learn to program. I don't mean "Hello World" in 20 languages. That includes SQL and relational calculus across multiple tables. Then note which languages allow you do do that expressively while reducing pain gluing together external programs. Then look for real bottlenecks. There's eff all you can do if you're waiting on AWS provided cloudformation to finish. Figure out alerts so you can go for a walk.

Plan the script calls and data sets. Be specific and focused in data handling process from source to destination. Avoid triggering blind select and general data queries unless necessary. Test with multiple scenarios. Check your script impact on system resources each time it runs. Know what it impacting and why. Review and enhance. Document all

Bash is growing with nice features being added every release. I was able to expand my use of arrays using some of the newer features there. It is always worthwhile for me to read the Bash manpage once in a while - I always end up learning something new. I think the same would apply for most of the other languages too.