Start here

New Interior…

I nearly panicked when I boarded the Air Canada Boeing 777-300 that will take me to Vancouver, for the connecting flight to Seattle.  The First Class cabin did not have the pods that I was used to… just regular seats…

…or so it seemed.  In truth they have revamped the cabin and while the pods are gone, the seats are great.  There is a true lie-flat bed seat, and instead of the normal 3″x2″ pillow you used to find on your seat, there was a proper full-size pillow and a blanket that looks like a comforter.  Once we are in the air I look forward to testing it out.

The new in-flight entertainment system is a huge improvement… the screen looks to be a 15.6″ wide screen, and it does not have to fold out (which means it does not have to be stowed for taxi and take-off).

The pod looks and feels similar to the one on the Singapore Airlines 777 that I flew one leg on last year.  Make no mistake – it is nowhere near as nice as the Airbus A380 Business Class on that airline (see article) but it is certainly an improvement over the old Air Canada pods… which I rather liked.

The first time I ever flew in those pods was my flight to Hong Kong, shortly after the Microsoft Airlift for System Center Configuration Manager 2007.  My first time in this seat is heading to the WSSC vNext Airlift, seven years later.

The table is not very intuitive, but other than that (and more new ridiculous rules from the FAs) I approve :)

Welcome to What’s Next…

There is irony in the title of this post… What’s next.

I posted on Friday that it was my last day working full time at Yakidoo.  I really enjoyed my time there, and am glad that my next venture will allow me to stay on there on a limited basis.

This afternoon I am meeting a colleague at the airport in Seattle, and that will begin my first day at my new gig.  I will talk more about it in a few weeks, even though today will be my first billable day.  That is what’s Next.

However the reason he and I will be in Seattle – Bellevue/Redmond actually – is the Airlift for Windows Server, System Center (WSSC), and Windows Azure vNext… the next generation of datacenter and cloud technologies that Microsoft is ‘showing off’ to select Enterprise customers several months prior to launching them.  It will be a week of deep-dive learning, combined with the usual Microsoft Marketing machine.  How do I know?  It’s not my first kick at the can Winking smile

It is, of course, not my first such Airlift.  The first one I attended was for System Center Configuration Manager (SCCM) 2007, back in November of that year. It was a consulting firm that had sent me, in advance of my heading off to Asia to teach it.  I have since been to a couple of others, each either as a consultant, a Microsoft MVP, or as a Virtual Technology Evangelist for Microsoft.  I have not given this a lot of thought, but this will be my first Airlift / pre-Launch event that I am attending as a customer.  It will be interesting to see if and how they treat me differently.

I suspect that the versions of WSSC that I will learn about this week will be the first that I will not be involved in presenting or evangelizing in any way dating back to Windows Server 2003.  I will not be creating content, I will not be working the Launch Events, and I will not be touring across Canada presenting the dog and pony show for Microsoft.  I will not be invited by the MVP Program to tour the user groups presenting Hyper-V, System Center, or Small or Essential Business Servers.  I will not be fronting for Microsoft showing off what is new, or glossing over what is wrong, or explaining business reasons behind technology decisions.  It is, in its way, a liberating feeling.  It is also a bit sad.

Don’t get me wrong… I will still be blogging about it.  Just because Microsoft does not want me in their MVP program does not mean that I will be betraying my readers, or the communities that I have helped to support over the years.  I will be writing about the technologies I learn about over the next week (I do not yet know if there will be an NDA or publication embargo) but at some point you will read about it here.  I will also, if invited, be glad to present to user groups and other community organizations… even if it will not be on behalf of (or sponsored by) Microsoft.  I was awarded the MVP because I was passionate about those things and helping communities… it was not the other way around.

What else can I say?  I am at the airport in Toronto, and my next article will be from one of my favourite cities in North America… see you in Seattle!

Onboard SAN… Issues.

A client of mine is a small business with a couple of physical servers and a couple of virtualization hosts.  One of the physical servers, a Lenovo ThinkServer, has been acting as a file server, so it has really been very under-used.  It is a good server that has never been used to its potential (like myself) but has been nonetheless a very important file server.  It has eight hard drives in it, managed by the on-board RAID controller.

When the server rebooted for no discernible reason last week, we were concerned.  When it didn’t come up again, and did not present any hard drives… we realized we had a problem.

I was relieved to discover that it was still under warranty from Lenovo, with NBD on-site support.  I called them, and after the regular questions they determined that there might be a problem with the system board.  They dispatched one to me along with a technician for the next morning, Their on-site service is still done by IBM, and in my career I have never met an unprofessional IBM technician.  These guys were no exception.  They were very professional and very nice.  Unfortunately they weren’t able to resolve the problem.

Okay, in their defense, here is what everyone (including me) expected to happen:

1) Replace the system board.

2) Plug all of the devices (including the hard drives)

3) Boot it up, and during the POST get a message like ‘Foreign drive configuration detected.  Would you like to import the configuration?’

4) We answer YES, the configuration rebuilds, and Windows boots up.

Needless to say, this is NOT what happened.  Why?  Let’s start with the fact that low-end on-board RAID controllers apparently suck.  Is it possible that a procedure was not properly followed?  I am not sure, and I am not judging.  I know that I watched most of what they did, and did not see them do that I felt was overtly wrong.

The techs spent six hours on-site, a lot of that spent in consultation with the second level support engineer at Lenovo, who had the unenviable task of telling me, at the end of the effort, that all was lost, and I would have to restore everything from our backup.

I should mention at this point that we did have a backup… but because of maintenance we were doing to that system over the December holidays the most recent successful backup was twelve days old.

Crap.

Okay, we’ll go ahead and do it.  In the meantime, the client and I went to rebuild the RAID configuration.  We decided that although we were going to bolster the server – including a new RAID controller – we were going to try to rebuild the array configuration exactly as it had been, and see what happened.

Let me be clear… even the Lenovo Engineer agreed that this was a futile effort, that there was no way that this was going to work.  Of course it would work as a new array, we just weren’t going to recover anything.  I agreed… but we tried it anyways.

…and the server booted into Windows.

To say that we were relieved would be an understatement.  We got it back up and running exactly as it had been, with zero data loss.  We were not going to leave it this way of course… I spent the next day migrating data into new shares on redundant virtual servers.  But nothing was lost, and we all learned something.

I want to thank Jeff from Lenovo, as well as Luke and Brett from IBM who did their best to help.  Even though we ended up resolving it on our own (and that credit goes mostly to my client), they still did everything they could to make it right.

So my client has a new system board in their server, and hopefully with a new RASID controller, some more memory, and an extra CPU this server can enjoy a new and long, productive life as a vSphere host in the cluster.

…But I swear to you, I will never let a customer settle for on-board ‘LSI Software RAID Mega-RAID’ type devices again!

Happy week-end.

Private Posts

I want to thank all of you who reached out to me this morning when you could not get into my ‘private’ article.  I wanted to share something that I had written (that believe it or not is far too incendiary to post on this site) with an old army buddy, so I pass-word protected it.  I assure you that you have not missed any great wisdom regarding IT.  I was essentially venting about terrorism.

Thanks for your feedback and your continued loyalty!

Mitch

Protected: (Private Article)

This content is password protected. To view it please enter your password below:

Keep Up: How to configure SCOM to monitor the running state of services and restart them when they stop

Windows runs on services.  Don’t believe me?  Open your Services console and count just how many are running at any given time.  Of course, some of them are more important than others… especially when you are talking about servers that are critical to your organization.

A new customer recently called me for a DEAR Call (emergency visit) because their business critical application was not working, and they couldn’t figure it out.  I logged into the server, and at first glance there didn’t appear to be anything wrong on the application server.  However I knew that the application used SQL Server, and I did not see any SQL instances on the machine.  A quick investigation revealed that there was an external SQL Server running on another server, and it only took a few seconds to see why the application was failing.

image

Very simply put, the service was not started. I selected it, clicked Start the service, and in a few seconds the state changed:

image

A quick look showed that their business critical application (in this case SharePoint 2010) was working properly again.

My customer, who was thrilled to be back in business, was also angry with me.  ‘We spent tens of thousands of dollars on System Center Operations Manager so that we could monitor our environment, and what good does it do me?  I have to call you in when things stop working!’

Yell as much as you like I told him, but please remember the old truism… if you think it is expensive hiring professionals, try hiring amateurs.  After he had learned about the benefits of implementing a proper monitoring solution he told his IT guy to install it… and that is exactly what he did.

System Center Operations Manager (SCOM) is a monitoring framework, and really quite a good one.  In fact, if Microsoft included the tools within the product itself to monitor every component that it is capable of monitoring, it would have to come in a much bigger box.  Instead, what it gives you is the ability to import or create Management Packs (MPs) to monitor aspects of your IT environment.  It is up to you to then implement those MPs so that SCOM can monitor the many components of your infrastructure… and take the appropriate action when things go wrong.

Of course, there are much more in-depth MPs for monitoring Microsoft SQL Server, but for those IT generalists who do not need the in-depth knowledge of what their SQL is doing, simply knowing that the services are running is often good enough… and monitoring those services is the exact same step you would take to monitor the DNS Server service.

Although it is long, following these relatively simple steps will do exactly what you need.

1) Open the Operations Manager console.

2) In the Operations Manager console open the Authoring context.

3) In the navigation pane expand Management Pack Objects and click on Monitors.

image

4) Rick-click on Monitors and select Create a Monitor – Unit Monitor…

5) At the bottom of the Create a unit monitor window select the Management Pack you are going to save this to.  I never save to the default management packs – create your own, it is safer (and easier to recover when you hork something up).

6) In the Select the type of monitor to create section of the screen expand Windows Services and select Basic Service Monitor.  Click Next.

SNAGHTML33d5f150

7) In the General Properties window name your monitor.  Make sure you name it something that you will recognize and remember easily.

8) Under Monitor target click Select… From the list select the target that corresponds to the service you will be monitoring.  Click OK.

9) Back in the General Properties window uncheck the Monitor is enabled checkbox.  Leaving this enabled will try to monitor this service on every server, not just the one where it resides.  Click Next.

10) In the Service Details window click the ellipsis button () next to Service Name.

11) In the Select Windows Service window either type the name of the target server, or click the ellipsis button and select the computer from the list.  Then select the service you wish to monitor from the list under Select service.  Click OK.

SNAGHTML33e455ae

12) Back in the Service Details window the Service name window should be populated.  Click Next.

13) In the Map monitor conditions to health states window accept the defaults… unless of course you want to make sure that a service is NEVER started, at which point you can change that here.  Click Next.

SNAGHTML33e6ea20

14) In the Alert settings window select the Generate alerts for this monitor checkbox.  You can also put in a useful description of the alert in the appropriate box.  Click Create.

The saving process may take a minute or two, but when it is done search for it in the Monitors list.

14) Right-click on your custom monitor.  select Overrides – Override the Monitor – For a specific object of class: <Name of the product group>

image

15) In the Select Object window select the service you are monitoring and click OK

16) In the Override Properties window, under the Override-controlled parameters list, scroll for the parameter named Enabled and make the following changes:

a) Select the Override checkbox.

b) Change the Override Value to True.

c) Click Apply

d) Click Show Monitor Properties…

17) In the Monitor Properties window click the Diagnostic and Recovery tab.

18) Under Configure recovery tasks click Add… and when it appears click Recovery for critical health state.

image

19) Under the Create Recovery Task Wizard click Run Command and click Next.

20) In the Recovery Task Name and Description window

a) enter a Recovery name (Re-Start Service works for me!).

b) Select the checkbox Recalculate monitor state after recovery finishes.

c) Click Next.

21) In the Configure Command Line Execution Settings window enter the following information:

Full path to file: %windir%\System32\Net.exe

Parameters: start <service name>

Working directory: %windir%

Timeout (in seconds): 120

22) Click Create.

23) Close the Monitor Properties window.

24) In the Override Properties window click Apply then OK.

The doing is done, but before you pat yourself on the back, you have to test it.  I always recommend running these tests during off-hours for non-redundant servers.

1) Open the services.msc console.

2) Right-click on Services (Local) and click Connect to another computer…

3) Connect to the server where your monitored service is running.

4) Right-click on the service and click Stop Service.

It may take a couple of minutes, but if you get up and go for a walk, maybe make a cup of coffee or tea… by the time you get back, the service should be restarted.

There seems to be a reality in the world of IT that the more expensive something costs, the less it is likely to do out of the box.  It is great to have a monitoring infrastructure in place, but without configuring it to properly monitor the systems you have it can be a dangerous tool, because you will have a false sense that your systems are protected when they really aren’t.  Make sure that the solution you have is properly configured and tested, so that when something does go wrong you will know about it immediately… otherwise it will just end up costing you more.

2015: My New Year’s Resolutions

2015-Calendar-7-723x1024I can honestly say that I don’t think I have ever made New Year’s Resolutions… at least, I have never written them down.  At the end of 2014, my annus horribilus, I am looking forward to a spectacular 2015.

Yesterday afternoon – December 30th – I was looking at my dashboard on www.asana.com – a tool that my boss recently introduced into our organization that  helps us to keep track of our tasks – and realized how much I have been able to accomplish in the month or so since he introduced it.  Don’t get me wrong, I was probably almost equally productive in the previous months… you know, when I wasn’t distracted by illness, my separation, my parents’ separation, kids’ illnesses and whatnot.  It’s just that this tool not only helps me to track what I need to do, it helps me to stay focused on those tasks.  It is a tool that, going forward, I will be using more and more.

What my focus on Asana reminded me of on that particular afternoon reminded me of the old truism – if you fail to plan, you plan to fail.  I have thought to myself several things I want to do in 2015, and so maybe putting them down on paper (yes, I still refer to my blog as ‘on paper’), and try to remember them so that I do not lose focus.  I am planning out several goals for 2015, and I plan to succeed.

Professional Life

There are some changes coming in the next few weeks that I am not prepared to share yet.  However the abstract of it is that I plan to continue to grow professionally, grow my knowledge, my network, and my reputation.  I have a couple of certifications that I have been putting off completing and I was challenged by my friend Sharon last night to complete them.  Additionally I have to renew a couple of certifications, and I should do that. 

I plan to re-focus the main theme of The World According to Mitch back to IT.  Does that mean I am going to stop talking about topics such as Weight Management, Martial Arts, and my personal observations?  No… some of those will stay on that blog, but the Weight Management component is being moved to a new blog that I have created.  Going forward I am going to invite my readers to subscribe to my second blog if they want to follow that side, and this one if they are interested chiefly in IT.

I plan to deliver to my readers the high quality and relevance you have come to expect from my blog over the past decade.  Wow… I can’t believe it’s been nearly a decade since I started blogging.

I have been asked by some if I plan to try to get re-awarded by the Microsoft MVP program.  The answer is that I have no such plans.  I will continue to participate in technical communities, and contribute by way of blogging, articles, and speaking engagements.  However I never really stopped doing any of that.  If Microsoft does not feel that my contributions warrant an award, then I will do it for the same reason I have always done it – to help the communities.

baby-new-yearPersonal Life

I think I need to be a better father than I have been to my children.  That is not easy for me, and I am not sure how I will do it.  Spending time with my younger son is easy, but my older son has school, works, has homework, friends, is probably dating, and so on.  I am going to continue to try to figure that out.

I need to be a better provider.  The last year my finances have been a disaster, and as such when I moved out of the house into my condo I was not able to pay all of my bills and meet all of my financial obligations to Theresa and the kids.  Fortunately I think I know how I will be able to do that starting in 2015, and that is my goal… in addition to paying down all of my debts and credit cards and bills.

Most of you are by now aware that my wife and I separated this past year.  In 2015 I hope to continue the healing process.  It is not easy, but I will continue to grow.

I started to lose weight in 2014… and not because I stopped eating from stress and depression (were that only the way my mind and body worked I assure you that by now I would be a lithe 215lbs).  Since I started the journey I have lost about 40lbs, and I plan to continue on this path (although December has been a total clusterf*¢k in that regard).  I am hoping to lose another 70-90lbs in 2015.

I decided in June of this year that it was time for me to start training for my 3rd Dan Black Belt… but I was wrong.  Before I was able to do that, I realized I needed to get myself back up to the level befitting a 2nd Dan Black Belt.  I needed to refresh my memory of all of the patterns that I needed to earn my 1st and 2nd Dan Black Belts.  While I am not entirely there, I feel that I am now at a place where I can begin to spend the next five months in training and preparation, and to test for my next belt in June.  Whether I am ready in June or not, I plan to earn it in 2015.

I plan to continue to teach, and to improve as a Taekwondo instructor.  Under the tutelage of my Masters I will continue to try.

There are a lot of other things I would like to do, but I don’t think I am going to over-commit.  I hope that I am able to do what I have outlined here, and hopefully more.  As John Lennon said ‘Life is what happens to you while you’re busy making other plans.’  I think what I have outlined here is reasonable, achievable, and with a little luck a year from now I will be able to look back at this list and smile, and say ‘Mission Accomplished!’

Happy New Year to all of you!

DONATE

%d bloggers like this: