Sysadmin is a Wonderful Job; Avoid These 14 Pitfalls and Be a Great One!

I've built dozens of Windows and Linux servers over the years, and bought them for my company in quantities of 50 or more in a single order. My teams at IBM's Cloud Division maintained the cybersecurity defense for hundreds of them over a decade, which required us to pass rigorous monthly internal audits that left us battered and bruised at times.

I'd like to encourage anyone working at a Help Desk or PC Support position to aspire to move up and become a server administrator, as I promoted most of my sysadmins from those areas. Being a sysadmin isn't an entry-level position, because there's so much to learn, but on the plus side, for a big corporation where you have a lot of responsibility it does pay extremely well.

If you're starting out as a sysadmin and many of the following terms or concepts are new to you, don’t feel bad; when I first got into cybersecurity, I was in the exact same position. At the beginning of my career, I had no idea just how much I didn’t know. Even now, some companies underestimate how much training, knowledge, and experience sysadmins need to prevent data breaches. Things just keep floating along fine…until they don’t.

A sysadmin must avoid a lot of pitfalls, but I’m positive the following info will increase your cybersecurity awareness. My goal is to spare you from a catastrophic data breach that’ll negatively affect your company’s bottom line, and most importantly to me, your cybersecurity career.?

1.??????Avoid Placing a Server into the Wrong Security Classification:

Smart companies have a corporate-wide policy that establishes different cybersecurity classifications for their servers, and focus on the highest priority servers. It makes no sense to guard every server with the same level of intensity, because diluting resources would weaken the security of the truly important servers. With that being said, when a sysadmin builds a server, they must place it in the correct security classification from Day One. Hackers will seek out servers that contain data they can sell, especially ones containing confidential records, or those exposed to the Internet. Over 30,000 websites are hacked per day, and that’s enough justification for a sysadmin to ensure they’ve placed their server into the correct security classification right from the start.

2.??????Never Miss Applying a Security Patch by the Due Date:

Hackers will take advantage of vulnerable servers, it’s their bread and butter, so never miss applying a security patch to your servers. I’ve popped up in bed out of a deep sleep because I dreamed I’d missed applying a patch to one of my servers. That’s how critical it is. The Equifax data breach of September 2017 resulted in the theft of 147 million customer records, and is the perfect example of a missed security patch. Someone at Equifax failed to follow the proper patch procedures, and 147 million customers like me paid the price. An unpatched server is like blood in the water for a hacker; breaching the network doesn’t guarantee a payoff, but finding a server with missing security patches means they’ve Hit the Hackpot.

3.??????Never Miss an IP Scan Due Date: ?

The job of an up-to-date IP scanner is to reveal security exposures, and if you fail to run them on a server during the allotted time period, there’s a good chance you’ll have a security vulnerability on your hands. Without an accurate IP scan you’re flying blind, and that’s when data breaches or audit failures will happen because you’ve missed applying a patch.

4.??????Failure to Harden a Server, or on the Flip Side, Don’t Overharden a Server:

To keep hackers out, a server must be hardened. This process entails ensuring system files have the correct settings, user accounts have only enough privileges to do the required work, vulnerability scans are run as required, patches are applied, unnecessary software is removed, default settings with security flaws for the operating system and applications are remediated, and anything that can allow a possible data breach must be neutralized. Unfortunately, overhardening a server can result in employees being unable to access the server. This is the result of a combination of internal errors; removing the security team from the Administrators group on Windows servers, commenting them out of the sudoers file on Linux servers, not allowing logins from remote, disabling direct access to the server via a serial cable, etc. External threats by hackers are made with bad intent, whereas accidental overhardening is done by inexperienced employees, but either way, the corporate bottom line takes a financial hit.

5.??????Adding Users Without Management Approval:

Adding user accounts to a server without management approval is indicative of a breakdown in the Change Control process. If accounts are added and there isn’t an approved change control request to validate them that’s a sure audit failure, or a crack by which a hacker can gain entry to breach the server. The change control process will serve you well, don’t shortcut it.

6.??????Unnecessarily Grant Administrator or Root Privileges to an Account:

Close scrutiny is called for whenever an administrator account is created, or a regular user is jumped up to an admin equivalent, because whoever owns the account has just been crowned royalty on that server. Not paying attention, nonchalance, work overload, or shortcutting the change control process will contribute to this problem. When a user account is given unnecessary admin rights and there’s a data breach, cybersecurity forensics auditors will find it as quickly as a dog finds a fire hydrant.

7.??????Accidental Deletion of a Volume:

This is the kind of error that makes a sysadmin wish they could roll back time, or crawl into a hole and pull it after them. I’ve literally stood there with my finger poised over the enter key, mind racing, just to make sure, for the fifth time, I’m making the correct changes before I actually move ahead and do it. I admit dealing with large amounts of important data made me nervous, but I never had to say “my bad” for making a mistake that cost my company money.

8.?Missed Backups:

A sysadmin must pay attention to the failure or success of their backups. It’s something that’s so basic, yet so easy to overlook. If a volume is erased by a hacker and can’t be restored to the latest amount of work, then that’s a corporate nightmare. If data is accidentally deleted and it can’t be restored from a backup the users will go nuts - even if they’re the ones who did it, lol!

9.?Not schedule a patch for the maintenance window:

Slightly different from completely missing a patch date, not scheduling a patch to be applied during the maintenance window can result in not meeting the metrics of a Service Level Agreement by the team managing the server. Missed SLA metrics will cause financial penalties to be triggered against the server management team, and money is taken out of their departmental budget to compensate.

10.?Sharing of Accounts and Passwords, Specifically the Admin Accounts:

It’s a major corporate no-no to share user accounts and passwords, specifically the admin accounts. After a data breach, a cybersecurity forensics team can’t easily pinpoint which employee was hacked if the account is shared. The same applies to an internal error that results in a loss of productivity, such as data loss. Say five team members share an admin account and know the password, and there’s a breach; suddenly they’re all under the microscope. The solution; company policy must be there are no shared accounts or passwords, period. For sysadmins, a single employee is to be assigned to the account, while others on the team will be members of the administrator group, or can elevate the privileges of their accounts as needed. ?Only the employee assigned to the administrator or root account can know the password, but their manager will have the password on file in case of emergency.??

11.?Failing to Open a Problem Management Record After Clearing a Hung Server with a Reboot:

Rebooting a hung server without logging a problem record is absolutely useless, because there’s no record of how many times it’s occurred or under what circumstances, which is invaluable when diagnosing intermittent problems. Without running diagnostics, it’s impossible to know if the issue is hardware or software related. A server that hangs up for unknown reasons is like a car you have trouble starting; the problem isn’t going to magically disappear, it’s only going to get worse over time.

12.?Closing a Change Control Request with Few Details:

Adding “Done” or “Completed” in the work description doesn’t help months down the road if problems come up, and a sysadmin should detail what they’ve done as well as possible. Imagine if a cop filled out a robbery report and all they added was “Money’s gone”? It won’t fly for them, and won’t fly for a sysadmin trying to remember what they did months later, when memories have faded, or a forensics team has questions because they need to understand why a data breach occurred.?

13.?Failure to Maintain Server Documentation:

A sysadmin is guaranteeing themselves nothing but trouble if they don’t maintain the required documentation for their server. On one hand, if a server is breached, they’re leaving the cybersecurity forensics team a tough task when they try to backtrack and figure out why it happened. On the other hand, if the server gets picked for an internal audit, they’re toast.

14.?Don’t Underestimate Hackers and Social Engineers:

It’s easy to underestimate the sneakiness of hackers and social engineers until a data breach has occurred. It’s far more difficult to build something than it is to burn it down, and hackers are arsonists; all they need is a match and gasoline to get the fire going. A cybersecurity expert has to master a vast number of security techniques, while a hacker will do just fine as a one-trick pony. They’ll target non-technical people in a company utilizing spear phishing attacks to gain the access they need, and then lock down corporate records for ransom demands, or steal intellectual property and other confidential information. They’re tricky. Sysadmins should never become complacent, and allow these bums to get the best of them. The optimal way to do this is to utilize your complete cybersecurity training skillset, follow the policies, processes, and procedures of your company, and to always stay vigilant.

To sum it up, friends, a successful sysadmin is someone who understands cybersecurity is a marathon, not a sprint. Today may be a victory for you, but tomorrow is another day, and just like with any other job, you’ve got to show up with a positive attitude. Otherwise, it’s a grind, and hackers will take advantage of any lapse in concentration. Stay focused, stay upbeat, and remember the no-no’s I’ve discussed, for I have no doubt you’ll have a long and totally wonderful cybersecurity career!