Rebuilding SWMI… Not the company, just the infrastructure

May 21, 2013 Leave a comment

I have been talking about rebuilding the domain for my company for several months, and finally sat down to do it this week-end.  Because I was essentially destroying the old domain there were a few steps I needed to perform before going ahead.

I performed a Backup of all of my data.  Nobody in their right mind would destroy an environment before they back up their data… especially if they are planning to actually delete the machines and start from scratch.

I performed a complete test-Restore of all of my data.  Now that my Mail Server is completely cloud-based this was easier than it might have been – If I had Exchange, SQL, and SharePoint it would have been more complicated, but also more crucial.  I always stress the importance of doing test-restores because the worst time to find out that your backup did not work is when you need to recover it.  Make sure that it works before you are faced with real data loss.

Planning was actually relatively simple for me, because the main environment was going to look very similar to the lab environment I had recently built for my Private Cloud camp.  I still had the planning documents for that, and I was able to follow them pretty closely for the first few machines. There was a time when I would have done the planning in my head, but now I make sure that I have all of my plans on paper before I go forward.  As the old adage goes, measure twice, cut once.  By having your thoughts on paper it is easier to stay on track… and if you do have to veer then you should document why you did.

Cleaning Up may not seem all that important, but destroying a cluster before destroying the domain is infinitely simpler than doing so afterwards.  It is doable of course, but there are PowerShell commands such as Remove-ClusterResource –force that one will get intimately familiar with if you do not think ahead.

Make sure you have all of the installation Media at hand… either on physical DVD or in an ISO repository.  This should not only include the obvious ones such as operating systems and applications, but also make sure you have the latest hardware drivers.  By looking at my Plan I know that I will need the following media:

Additionally I would need several bits that I would simply download as one-offs… the Report Viewer, Silverlight, and things like that.  However since my networking topology is already in place, I would be able to do that from within the virtual machines.

Now that I have everything ready to go, I am ready to move forward.  Building an environment from scratch (green-field) would be simpler, but there are some aspects that prevented that.  In your production environment (should you ever decide to start from near-scratch) you will have to run through the same sort of project plan.  Make sure you think it out – do not simply sit down one morning and expect to implement in the afternoon; rather make sure you observe your environment for a few cycles and build your plan over time so that you don’t run into any surprises.

In my next piece I will go through the actual build architecture of how I decided to build my server infrastructure; I will also introduce some actual build videos of the System Center components.  If there is something in particular that you would like to see please let me know by commenting! -M

Filed under Infrastructure, Server, System Center, Windows Server 2012

What not to Learn… Revisited for 2013!

March 12, 2013 2 Comments

In October, 2011 I posted an article called vPTA: What NOT to take away from my 1-day virtualization training!  It was only partly tongue-in-cheek on the environment that I have been using for several years to demonstrate server virtualization from a pair of laptops.  A few months later Damir Bersinic took that list and made some modifications, and published it on this blog as Things NOT To Take Away from the IT Virtualization Boot Camp.  Because we spend so much time in our IT Camps demonstrating similar environments, I decided it was a good time to rewrite that article.

Normally when I revisit an article I would simply republish it.  There are two reasons that I decided to rewrite this one from scratch:

  • The improvements in Windows Server 2012, and
  • My more official position at Microsoft Canada

Since writing that original article I have tried to revise my writing style so as to not offend some people… I am trying to be a resource to all IT Professionals in Canada, and to do that I want to eliminate a lot of the sarcasm that my older posts were replete with.  At the same time there are points that I want to reinforce because of the severity of the consequences.

Creating a lab environment equivalent to Microsoft Canada’s IT Camps, with simple modifications:

1. In our IT Camps we provide the attendees with hardware to use for their labs.  Depending on the camp attendees will work in teams on either one or two laptops.  While this is fine for the Windows 8 camps, please remember that in your environment – even in a lab where possible – you should be using actual server hardware.  With virtualization it is so simple to create a segregated lab environment on the same server as your production environment, using virtual switches and VLAN tagging.  In environments where System Center 2012 has already been deployed it is easy enough to provision private clouds for your test/dev environments, but even without that it is a good idea.  The laptops that we use for the IT Camps are great for the one- or two-day camps, but for longer than that you are going to risk running into a plethora of crashes that are easy enough to anticipate.

2. You should always have multiple domain controllers in any environment, production or otherwise.  Depending on who you speak to many professionals will tell you that at least one domain controller in your domain should be on a physical box (as opposed to a virtual machine).  I am still not convinced that this does not fall into the category of ‘Legacy Thinking’ but there is certainly an argument to be made for this.  Whether you are going to do this in physical or virtual, you should never rely on a single domain controller.  Likewise your domain controllers should be dedicated as such, and should not also be file or application servers.

3. I strongly recommend shared storage for your virtualization hosts be implemented on Storage Area Networks (SANs).  SAN devices are a great method of sharing data between clustered nodes in a failover cluster.  In Windows Server 2012 we have included the iSCSI Software Target that was previously an optional download (The Microsoft iSCSI Software Target is now free).  While this is still not a good replacement of physical SANs, it is a fully supported solution for Windows Failover Cluster Services, including for Hyper-V virtual machine environments.  It is even now recognized as an option for System Center 2012 private clouds.  As well the Storage Pools feature in the new Server is a compelling feature to consider.  However there are some caveats to consider:

A. Both iSCSI software targets and Storage Pools rely on virtual storage (VHDX files) for their LUNs and Pools.  While VHDX files are very stable, putting one VHDX file into another VHDX file is a bad idea… at least for long-term testing and especially for production environments.  If you are going to use a software target or Storage Pool (which are both fully supported by Microsoft for production environments) it is strongly recommended that you put them onto physical hardware.

B. While Storage Pools are supported on any available drive architecture (including USB, SATA, etc…) the only architecture that will be supported for clustered environments are iSCSI and SAS (Serial Attached SCSI).  Do not try to build a production (or long-term test environment) cluster on inexpensive USB or SATA drives.

C. In our labs we use a lot of thin-provisioned (dynamically expanding, storage-on-demand) disks.  While these are fully supported, it is not necessarily a best practice.  Especially on drives where you may be storing multiple VHDX files you are simply asking for fragmentation issues.

4. If you are building a lab environment on a single host, you may run into troubles when trying to join your host to the domain.  I am not saying that it will not work – as long as you have properly configured your virtual network it likely will – but there are a couple of things to remember.  Make sure that your virtual domain controller is configured to Always Start rather than Always start if it was running when the service stopped.  As well it is a good idea to configure a static IP address for the host, just in case your virtual DHCP server fails to start properly, or in a timely fashion.

5. Servers are meant to run.  Shutting down your servers on a daily basis has not been a recommended practice for many years, and the way we do things – at the end of the camp we re-image our machines, pack them into a giant case and ship them to the next site – is a really bad idea.  If you are able I strongly recommend leaving your lab servers running at all times.

6. While it is great to be able to demo server technologies, when at all possible you should leave your servers connected (and turned on) in one place.  If you are able to bring your clients to you for demos that is ideal, but it is so easy these days to access servers remotely on even the most basic of Internet connections.  If your company does not have a static IP address I would recommend using a dynamic DNS service (such as dyndns.com) with proper port-forwarding configured in your gateway router to access then remotely.

7. I am asked all the time how many network adapters you need for a proper server environment.  I always answer ‘It depends.’  There are many factors to consider when building your hosts, and in a demo environment there are concessions you can make.  However unless you have absolutely no choice it should be more than one.  For a proper cluster configuration (excluding multi-pathing and redundancy) you should have a production network, a storage network, and a heartbeat network… and that is three just for the bare minimum.  Some of these can share networks and NICs by configuring VLANs, but again, preferably only in lab environments.  Before building your systems consider what you are willing to compromise on, and what is absolutely required.  Then build your architectural plan and determine what hardware is required before making your purchase.

7a. While on the subject of networks, in our demo environment the two laptop-servers are connected to each other by a single RJ-45 cable.  BUY SWITCHES… and the ones that are good enough for you to use at home are usually not good enough for your production environment! Smile

8. When it is at all possible your storage network should be physically segregated from your production network.  When physical segregation is not possible then at least separating the streams by using vLANs is strongly recommended.  The first offers security as well as bandwidth management, the second only security.

9. Your laptop and desktop hardware are not good-enough substitutes for server-grade hardware.  I know we mentioned this before, but I still feel it is important enough to state again.

10. In Windows Server 2008 R2 we were very adamant that snapshots, while handy in labs and testing, were a bad idea for your production environment.  With the improvements to Hyper-V in Windows Server 2012 we can be a little less adamant, but remember that you cannot take a snapshot and forget about it.  When you delete or apply a snapshot it will now merge the VHDX and AVHDX files live… but snapshots can still outgrow your volume so make sure that when you are finished with a snapshot you clean up after yourself.

11. Breaking any of these rules in a production environment is not just a bad idea, it would likely result in an RGE (Resume Generating Event).  In other words, some of these can be serious enough for you to lose your job, lose customers, and possibly even get you sued.  Follow the best practices though and you should be fine!

Related articles

Filed under Infrastructure, IT Camps, IT Pro, Windows Server 2012 Tagged with , , , ,

To be, or not to be: If you are IT it is not your decision!

June 26, 2012 1 Comment

I made what I thought was a reasonably innocuous statement in front of an audience a few months ago, and couldn’t believe the pushback I got.

Our job as the IT providers – whether as in-house providers or as contractors – is not to make decisions.  In fact, people are often amazed by how few decisions we have to make.

There was a chorus of objections from this group of high-level systems administrators who protested that they made decisions all of the time, with regard to licenses, solutions, whose hardware to buy, what password policies to implement, and so much more.  They wanted to assure me that they made important decisions all of the time that would affect the user experience of everyone in their organizations.

Wrong.

As a service provider, and I hope that by now we can all agree that in most organizations IT is indeed a service provider, it is not our job to make decisions, it is our job to implement the decisions of others.  Our job is not to be decision makers, it is to be trusted business advisors.  That is an important distinction that we can never forget.

We don’t tell our clients what they need to do; they know what they need to do.  We simply advise them how they can use different technologies to do it, and then they make the decision.  It is our job to let them know what tools we can make available to them to facilitate their jobs.

Electronic communications is a great example of this.  A few short years ago it was our job to tell our organizations that they could better communicate with their customers, suppliers, and everyone if they would start using e-mail.  Then we often had to make a business case for using our own domain name – mitch@swmi.ca – rather than a public cloud (although we didn’t call them that) free address such as mitch@hotmail.com.  Of course it usually made business sense, but we so often had to make the case anyways.  From there it was servers – should our mail servers be in-house, or should we rely on our ISP (or another third party) for that service.  I even remember having to convince one boss that his e-mail address should be printed on his business card.

In the entire process above, I didn’t make a single decision.  I made recommendations, but it was the boss, the board, the committee that made the decisions.

So when this decision was made – our company will host our own mail servers – at least I could make the decision as to what mail servers to buy, right?

Wrong.

If I was an honest and trusted business advisor I would research what was available, cost out different solutions weighing in such factors as cost, reliability, features, and ease of use.  I would then present a number of options to the board (often at this point an IT Committee), and they would make the ultimate decision.  Again, I would make my recommendations, but the decisions were someone else’s.

Fast forward to 2012, the world is moving into the cloud.  Private Cloud or Public Cloud?  Whose solution?  I present my customers with recommendations.  I make my recommendations based on several factors, including operational expenses versus capital expenses, bandwidth requirements, service level agreements (SLAs), and so many other factors. Most of the time, because of my reputation as a trusted business advisor, my clients (and students) follow my advice.  However in the end they are free to make their own decisions.

I was in an interview with a potential client recently who came to me because they need to replace their current service provider, and we sat down for a great conversation.  Near the end of the chat he said to me:

Mitch, you obviously have the requisite skills and staff to do what we need, and I hope we can continue to work together going forward.  But you have a lot of very strong opinions.  What would you do if we disagree?  You tell me we should do <A>, I say that I want to do <B>.  What do you do then?

It was an almost obvious question that I had never been asked before.  I told him honestly ‘Mark, if we disagree on what to do then I am going to do my best to convince you that I am right.  I will make every proposal and reasonable argument, and will do everything I can to sway you to my side.  If I cannot do that, then the simple answer is that you are paying the bills, and that makes the decision yours.  In almost every case I will do what you ask me to do, because they are your servers and your infrastructure.’

Wait a minute… you said ‘almost’? Why the qualifier?

‘Very simple.  If you ask me to do something that will compromise the security of your organization’s systems then you will have to ask someone else to do it.  I compromise on everything else, but not on security.’

That, really, is the only major decision we can make… the decision to walk away when our customer (or boss) won’t take our advice.  Sure, others can delegate the details to us – what version of what server to use on what hardware – but the real decisions belong to others.

While this may be (to some) a bruise to our egos, the reality is we should be relieved; we have enough as IT administrators on us without having to shoulder the burden of those major decisions.  We are responsible for so much – and seldom get the credit we deserve for the jobs we do.  We are responsible not only for keeping our systems working, but also for giving the people who do make the decisions the best advice and suggestions.

Let someone else make the decisions Smile

Filed under Decisions, Infrastructure, IT Pro, Trusted Business Advisor Tagged with ,

%d bloggers like this: