The World According to Mitch

**DISCLOSURE: While I am contracted to Microsoft Corporation, I am not an employee. The articles that I write are not meant to represent the company, nor are they meant to represent me as an employee or spokesman for the company. As has always been the case, all articles on this website represent me and nobody else.

Last week a friend tagged me on a post on LinkedIn with a cute picture of two people holding up cardboard signs. The first reads “Don’t deploy to production on Fridays.” The second reads “This Dude Gets It, Folks” Aside from the fact that this is obviously a photoshopped image, the message is clear and true. Production deployments, even the ones that have been tested in labs and planned out perfectly, can go awry and end up ruining your entire weekend.

I reposted the image because I thought it was cute… and inoffensive. Of late, this seems to be a difficult bar to meet.

Someone viewed this image and agreed, commenting: “Or after 6:00 pm any day.”

I thought about it for a few minutes to see if I agree with this or not.

Proper Planning Prevents Poor Performance

In a secure, well-managed IT infrastructure, there should be redundancies and failovers built in to allow daytime deployments… sometimes. With that said, I have seen too many systems that need to be taken down in order to be upgraded, for example. We cannot say “Thou shalt deploy only during production hours” because some systems do not allow for that functionality.

I have been involved in upgrades and migrations where the team decides in advance that it cannot be done during the regular workday. We plan to do it after hours. We plan our time so that maybe we come in at 2pm that afternoon rather than at 8:00am… there are all sorts of ways to make it work, knowing that yes, sometimes IT staff need to work evenings.

It is not only the IT staff that need to be planned appropriately. Nobody on my IT staff at any of the companies I have managed (other than very small businesses) were qualified to test the accounting software… or many other line of business applications that might fall into the category of ‘Gotta update after hours.’ So we work with those teams – usually coordinated through the Change Advisory Board (CAB) to make sure that someone is able to work with you – either on-site or remote – to make sure that once your upgrade seems to be complete, it is tested by users who can confirm that yes, all of the systems are a go.

This is not a step you should take lightly, and it is something that should not just be trusted to one person if possible. I worked on an upgrade of an accounting system several years ago for a customer, and the person tasked with testing it was a good guy, but maybe a little lazy; he was also asked to log in at 11:30pm from home to make sure everything worked. He had a couple of drinks with or after dinner, so maybe he was not at his best. He was able to log into the application, so he gave us the thumbs up. The next morning it was discovered that yes, the authentication worked… but the connection to the back-end database server was not reestablished, and so everything was essentially zeroed out. Fortunately, that was a simple service that needed to be restarted, so the next morning when it was discovered it took us five minutes to fix… a bad position to be sure, but still better than nobody being able to work for eight hours while we figured out and fixed the problem.

I started working at a company a few years ago and it was a great company with a great group of people. One of the in-house developers had a meme on his wall (with a picture of The Most Interesting Man In The World (from the Dos Equis beer ads) that read: I don’t always test my code… but when I do, I do it in production systems. It always made me cringe. So yes, there is a time to do things and sometimes it is during the day, but more often than not we are going to try to plan changes to our production after hours. Plan, prepare, test, and verify… and always have a roll-back plan. Yes, if you have tested everything then it should work but you have to account for the what ifs. I have been 35 hours into a major weekend systems migration when I decided that because of some unforeseen (and unforeseeable) circumstances (hardware failure due to power spike, and things like that) when I decided to roll back to the starting point so that on Monday morning the company could work as needed. Yes, it was a huge setback that cost my company money… but it did not cost my customer downtime or productivity, which was much more important.

The Universal Consultants Answer

I cannot tell you when to deploy anything. For years I have been calling out the Universal Consultants Answer (UCA): It depends. Some things can be deployed during business hours and some things have to be done after hours. Believe it or not, some things will need to be deployed over the weekend so yes, we will start on Friday afternoons. It is how it goes. You have to decide that on a case by case basis. Every situation is different, but a good project manager will speak to his or her engineers and determine what is required, and when it should be done. If you account for everything that you can then you will achieve success… whether during the day, evening, overnight, or weekend.

No two projects are the same but the rule is to account for everything you can anticipate, so that the things you cannot anticipate are usually minor and easy to overcome. As my weekend debacle from 2005 proves, that will not always be true… but it will always be manageable. Make sure there is no point of no return, that you can always fall back. It is just another aspect of proper planning that leads to success.

Conclusion

There is no one right time to do things, because all things are different. When to deploy a system depends on myriad factors that you should consider during the planning phase. Make sure you account for everything and then decide… and then make it so!