The World According to Mitch

I spend a lot of time speaking with clients about their environments.  From time to time, my job is to ‘interview’ them, so that I can properly document their environments.

Recently I was speaking with a couple of admins at a private sector company who were very proud of their environments.  They had hired a sub-contractor to deploy much of their infrastructure, and they were pleased to answer my questions.  They had engaged my serviced to audit the work performed by the other contractor, and were pretty sure that the meeting would be pro-forma, and I would sign off on everything that had been done.

MDG: How often are backups performed?
Client: Daily for most systems, hourly for highly transactional servers.

MDG: How long are your backups retained?
Client: Hourly and Daily backups are retained for 30 days, weekly backups are retained for 6 months, monthly backups are retained for a year.

MDG: How often are your servers patched?
Client: Monthly… we think.

Those last two words send chills down my spine… and I hear it more often than you would think.

Is it our job to know everything about our environments?  Maybe, maybe not… but if you think and you do not know, then you should be following up and making sure.

Why does it scare me that this was their answer?  Because one of the people in the room was responsible for the security and stability of the environment, and an unpatched server will eventually be an unsecure server.

It is not surprising that in a large environment that the manager does not know every detail of the day-to-day operations of his network; he has people reporting to him who are responsible for these things.  In fact, the person responsible for testing, approving, and applying patches was not in the room for this meeting.  He was, we can assume (as this meeting was held on Patch Tuesday), somewhere testing patches.  The manager does not need to know everything… but he has to be able to get that information.

Seventeenth Century French philosopher René Descartes stated: “Cogito ergo sum” (French: Je pense, donc je suis; I think, therefore I am).  He was claiming that he knows that he exists, because he is able to think.  While I feel this philosophy can be disproven by a great many zealots who certainly are but seem unable to think, he was essentially saying that thinking is a good thing.  Socrates – the Athenian philosopher of the Fifth Century B.C.E., claimed that “The only true wisdom is in that you know nothing.”   He was not saying that stupidity is a good thing, rather that it is important to question everything.

So, is it better to think that you know how often your systems are patched, or to know that you do not know, and thus inquire?  While I have never spent a great deal of time studying philosophy (Athenian, French, or otherwise), I think when we are unsure, it is better to inquire.

In my follow-up meeting a few days later, the manager came equipped with a sheaf of printed reports that I had asked for… including the one that showed that patches were indeed applied on a monthly basis, and a list of pending patches, failed patches, and unprotected systems.  The client was doing exactly what they needed to do, and the consultant who had deployed their infrastructure had indeed implemented two separate and complementary patch-management systems, including System Center Configuration Manager (SCCM) with Windows Server Update Services (WSUS), and System Center Virtual Machine Manager (SCVMM) for their virtual servers and hosts.  My client, whose systems integrity were never an issue, was happy that he had gone to make sure, and in fact extracted reports that he had never actually checked before.  His systems were fine… and so was his peace of mind… now.

Going back to the philosophical questions for a minute, we have all heard the question: “If a tree falls in a forest and no one is around to hear it, does it make a sound?”  This is attributed to Eighteenth Century philosopher George Berkeley (in his work “A Treatise Concerning the Principles of Human Knowledge” published in 1710.  In systems administration, the unheard tree can lead to eventual disaster, depending on the scope.  If a system is not properly patched, it can be vulnerable to myriad vulnerabilities.  If systems are not reporting properly, it might mean that the systems are not available… or something more sinister.  That is why we have to check these reports, to make sure that what we believe to be our solid environment is indeed solid, and will remain so.

My client (the company’s IT Manager) had a mostly stable environment, but three systems listed on the reports he brought had not been patched in several months, thereby missing a critical patch that we knew had led to an exploited vulnerability.  The lack of noise – very few admins get active alerts that a system failed to patch – was deafening, and left unchecked could have had disastrous results.  Fortunately, that did not happen; the three unsecured systems were immediately flagged and quarantined, and after a few minutes with the Desktop Support Team were again right as rain.  All is well…

While we may wax poetic, IT is not about philosophy.  Knowing is important; Certainty is crucial; Silence can be Critical.

…And yet, as IT Professionals, just as with long-dead philosophers, it is important for us to keep asking questions, to keep actively seeking the truth, and questioning the silence.  If you don’t?  Well, that tree may fall on your head, and your thinking will mean you are… out of a job.