The World According to Mitch

Every so often I encounter a problem that needs to be escalated. This one had a lot of us scratching our heads for a couple of weeks.

It is a French server, but in simple terms: When trying to run the Hybrid Configuration Wizard to connect the Exchange Server to Exchange Online, we ran into myriad issues. This one reads ‘Hybrid agent installation: The installation terminated with the exit code 1603.’

There are numerous references to this online with several proposed solutions. None of them worked for me. (My favourite among these read: “If you reached this page, you most probably know, that Exchange is a piece of crap.” (Credit: see article) It was amusing… if not helpful.

I opened a support ticket with Microsoft, and they were as baffled as I was. We spent nearly two weeks trying to track the problem down. What should have been a reasonably simple issue was… not. Every possible solution led to three more problems, which when solved would all eventually lead us back to the original problem.

I have never claimed to be a Microsoft Exchange Server guru, but what I was trying to accomplish should not take a guru… and frankly, neither should solving these issues. Yet here we all were (I was joined by no fewer than three Microsoft premier support engineers on three continents, all of who had their escalation teams and mentors and so on). Nothing doing.

My client and I were in discussions on how to proceed, and I suggested that there might have been a security policy blocking our forward progress. I suggested we start with a clean server installation, and not a cloned image. It was at this point that he said to me the words that shook me. “That’s a good idea. I have been having troubles on one of my server projects too. I suspect that our team did not SysPrep the original image before cloning it.”

They did not clone the image.

Many years ago, when I started lecturing in IT, I remember telling my audience that when DNS is not properly configured, there would be any number of things that would not work… and it would usually not look (to the unskilled professional) like a DNS problem. The thing about this was, as true a statement as it is, there would always be signs that a qualified IT Professional would be able to see and start thinking ‘Aha! This could be a DNS issue.’ You would be one step closer to a resolution.

With conflicting Security Identifiers (SIDs) and Global Unique Identifiers (GUIDs) there are no signs pointing to it. It is so arcane a problem that it is unlikely that 5% of IT Professionals would have quickly said ‘AHA! Crashing GUIDs! Of course!’ It is the computer equivalent of a genetic mutation in an animal caused by a mad scientist trying to create a new species… but without telling anyone, and forgetting to carry the 1 in a critical computation. The animal would be sick, but nobody would know or understand why.

Over the next day I was able to do what I needed to do on the new server without any hassle or errors. The client was happy… and sad. “We are going to have to rebuild every server in our organization that was based on that image.” Yes, but you know what the problem is, and you know how to solve it. Isn’t that better than having 50-100 servers that were always sick for reasons you could never understand? He agreed of course.

My client had hired a guy who was mostly competent, but did not know everything, and did not know what he did not know. He also did not know that asking for help and guidance is not a sign of weakness, it is a sign that you are humble and willing to learn. It shows that you have what it takes to become a great IT Pro.