Why we need a backup…

This is a story about IT Security.

It is hard to believe that within three weeks we have had our Kia Rondo.  However it is easy enough to gauge… we brought it home (used) on New Years Eve, December, 2009… When I drove Theresa to the hospital to deliver Gilad it was still on its first tank of gas.

Now, the fact that it has taken us this long to learn our lesson is testimony to our diligence, but nonetheless the lesson would eventually be learned.  New cars, as you know, come with two sets of keys.  Used cars, unfortunately, do not.  More often than not they come with only one, as is the case with the Rondo.  Theresa and I switch off driving the two cars every so often (usually when one needs gasoline or other maintenance I get it).  As such, we are usually pretty good about leaving the keys on the secretary by the door.

This past week-end was a disaster for me.  I got home from two weeks in South America & Mexico on Thursday, jetlagged and exhausted from the travel.  So much so that Saturday and Sunday I essentially slept all day, although I did venture out in the evening… on Saturday I took Theresa to Niagara Falls for dinner, and on Sunday after they came home from Buffalo I took her to a movie.  When I came home Theresa had warned me that both cars needed gas, so we drove the Toyota on Saturday (and I filled the tank) and the Kia on Sunday (and I filled the tank).  As we arrived home after the movie, there was a confluence of many irregularities – a dog jumping at the door, a phone ringing, and a need for the restroom. 

The keys to the Kia ended up in my pocket…

…and the following morning they came to the airport with me…

…and then they came to Halifax with me.

I checked into the Maple Leaf Lounge at the airport in Halifax when I called my beautiful, loving, absolutely understanding wife whom I love dearly and who is always the first person I call when I land anywhere.  I heard Gilad crying in the background, which was strange for the time of the morning when he was usually at daycare.  ‘No, nothing is wrong with him… but he is rather upset that you took my car keys and stranded us here.’

Oh, crap.

To cut a long story short, after losing most of a day, a very understanding friend drove my very loving and wonderful and understanding wife to the airport parking lot and picked up my car from the long-term parking lot.  It was a huge hassle, but all was well.

At this point – if not several paragraphs ago – you have probably started wondering why I prefaced this tale of an absent-minded husband as a story of IT Security.  Keep reading and all will be made clear!

Many small and mid-sized businesses rely on one person to be the ‘Keeper of the Keys’ for their network – one user’s account is the Domain Administrator, or Root account.  Of course it is best practice to not share passwords, so that person is the only person who knows the credentials.  In some cases, that ‘person’ is not even an employee, but an IT Service Provider, who maintains their computer for them.  While the skies are clear this poses no problem.  Too often I have heard horror stories of things going very bad very fast.

Over the course of my career I have received no fewer than a dozen calls from companies who needed for me to reclaim their networks following a falling-out with their former IT Manager.  In most of these cases the company had decided to lay them off because they were going to outsource their IT services, although on a couple of occasions there was a fight between the owner and the IT guy who stormed off in a huff.  In one unfortunate case the IT guy died suddenly in a car accident.

On the other side of the same coin, I have on a number of occasions been told by IT service providers that their clients were late paying their bills, so they were going to deny them service and would not provide any credentials until all of the accounts were adequately settled.  I advised these IT pros that while I understood their frustrations, they were likely breaking the law and opening themselves up to legal action that would far outweigh any disputed monies.  I can only hope that they followed my advice and reversed their stances… As they did not name the client, there was no way for me to follow up on that.

While the IT guy who refuses to share the credentials is breaking the law (except for the guy who died, who was pretty action-proof) it is the company that suffers until the issue is resolved.  Resolving the issue – either technologically or legally – can be time consuming and costly.  It is also a situation that is very easy to avoid.

I do not think the solution is giving anyone in the company Admin/Root credentials… nobody should ever have higher credentials than they need to do their job.  What I would recommend, however, is that a second Admin/Root account be created with a long and super-complex password.  Those credentials should be stored separately and securely in sealed envelopes that hopefully will never need to be used.  However just like having a spare set of keys, it is a safety net against the sudden souring of the relationship between the SMB and the IT provider, whether that provider be an employee or contractor.

This plan is unfortunately not bullet proof.  It would be simple for the provider to either disable this account or change those credentials.  Legally speaking this would be an overt criminal act, but the jaded tech may not be concerned about that.  That is why it is crucial that companies manage their HR – specifically their layoffs – carefully.  If they are planning to lay off their administrator it is a good practice to use the following steps:

  1. Plan the timing carefully.
  2. Before you call your administrator into your office for that uncomfortable conversation, ensure that those credentials work, and access the Active Directory Users and Computers console using that account.
  3. When you know that he is waiting to come into your office, disable his account.

It is unfortunate, but a jaded former employee can cause a lot of damage.  I have heard horror stories of companies laying off their IT manager, but not disabling their account.  That laid off employee then goes back to their desk and starts wreaking havoc on the network.  The IT administrator is, unfortunately, not a position that you can lay off and give them two weeks notice, expecting they will faithfully continue to perform their duties.  If you are getting rid of the IT admin, you have to pay their settlement out but terminate their employment – along with their credentials – immediately.

If you think you may be protected by loyalty, remember that you are about to demonstrate a termination of that two0way loyalty street.  In cases I have been involved in neither long-time friendships nor family relations have protected the company. 

I am not saying that this will happen in every case, but you cannot gamble that it will not happen to you.  Don’t take the chance, and you will never have to write an article about how loving and understanding your wife is because you flew to Halifax with her keys Winking smile


4 responses to “Why we need a backup…”

  1. Great post, Mitch. Highly relevant to the SMB/SME. What’s also important, from an AD perspective, is to ensure that any other accounts the user has access to have their password changed… For example the domain administrator account. I’ve seen organisations who’ve had a SysAdmin leave, but not changed the domain administrator password for 6 months. Dangerous!

    Really, though, the Administrator account should be disabled…

    1. Hi Chris,
      That is a great point… the DEFAUALT Administrator account should be disabled. However we are still going to have an account with Admin provileges, and when SysAdmins leave ALL passwords should be changed. It is amazing that some orgs remember their AD accounts, but not the external firewall… the first thing he can get at from the outside!

  2. […] Why we need a backup… « The World According to Mitch […]

  3. Essays like this are so impotarnt to broadening people’s horizons.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: