Home » Clustering

Category Archives: Clustering

Broken Cluster? Clear it up.

Three years ago I wrote an article about cleaning up nodes of clusters that had been corrupted and destroyed (See Cluster Issues… how to clean out cluster nodes from destroyed clusters). 

Unfortunately the cluster command has been deprecated in Windows Server 2012 R2, so we need to go to PowerShell… which frankly is where we should be going anyways!

PS C:\> Clear-ClusterNode –Cluster Toronto –Force

In this example we had a cluster named Toronto that is no longer accessible.  Unfortunately one of the nodes was off-line when the cluster was destroyed, so it didn’t ‘get the message.’  As such, when we try later to join it to a new cluster we get an error that the server is already a node in another cluster.

The cmdlet only takes a minute to run, and when you do run it you are all set… you will immediately be able to join it to another cluster.

For the fun of it, I have not figured out yet how to (natively) run this cmdlet against a remote server, so you can either do it by connecting to each server or…

Invoke-Command –ComputerName Server1 –ScriptBlock {Clear-ClusterNode –Cluster Toronto –Force}

I covered this option in a previous article (Do IT Remotely) which shows how to run cmdlets (or any script) against a remote server.

No go forth and script!

Cluster-Aware Updates: Be Aware!

When I started evangelizing Windows Server 2012 for Microsoft, there was a long list of features that I was always happy to point to.  There are a few of them that I have never really gone into detail on, that I am currently working with.  Hopefully these articles will help you.

Cluster Aware Updates (CAU) is a feature that does exactly what it says – it helps us to update the nodes in a Failover Cluster without having to manually take them down, put them into maintenance mode, or whatever else.  It is a feature that works in conjunction with our patch management servers as well as our Failover Cluster.

I have written extensively about Failover Clusters before, but just to refresh, we need to install the Failover Clustering feature on each server that will be a cluster node:

PS C:\Install-WindowsFeature –Name Failover-Clustering –IncludeManagementTools –ComputerName <ServerName>

We could of course use the Server Manager GUI tool, but if you have several servers it is easier and quicker to use Windows PowerShell.

Once this is done we can create our cluster.  Let’s create a cluster called Toronto with three nodes:

PS C:\New-Cluster –Name Toronto –Node Server1, Server2, Server3

This will create our cluster for us and assign it a dynamic IP address.  If you are still skittish about dynamic IP you can add a static IP address by modifying your command like this:

PS C:\New-Cluster –Name Toronto –Node Server1, Server2, Server3 –StaticAddress 10.10.10.201

Great, you have a three-node cluster.  So now onto the subject at hand: Cluster Aware Updates.

You would think that CAU would be a default behaviour.  After all, why would anyone NOT want to use it? Nonetheless, you have to actually enable the role feature.

PS C:\Add-CauClusterRole –EnableFirewallRules

Notice that we are not using the –ComputerName switch.  That is because we do not install the role service to the servers but to the actual cluster.  You will be asked: Do you want to add the Cluster-Aware Updating clustered role on cluster “Toronto”? The default is YES.

By the way, in case you are curious the Firewall Rules that you need to enable is the ‘Remote Shutdown’ rule.  This enables Cluster-Aware Updating to restart each node during the update process.

Okay, you are ready to go… In the Failover Cluster Manager console right-click on your cluster, and under More Actions click Cluster-Aware Updating.  In the window Failover – Cluster-Aware Updating click Apply updates to this cluster.  Follow the instructions, and your patches will begin to apply to each node in turn.  Of course, if you want to avoid the management console, all you have to do (from PowerShell) is run:

PS C:\Invoke-CauRun

However be careful… you cannot run this cmdlet from a server that is a cluster node.  So from a remote system (I use my management client that has all of my RSAT tools installed) run:

PS C:\Invoke-CauRun –ClusterName Toronto

You can watch the PowerShell progress of the update… or you can go out for ice cream.  Just make sure it doesn’t crash in the first few seconds, and it should take some time to run.

Good luck, and my the cluster force be with you!

Cluster Issues… how to clean out cluster nodes from destroyed clusters

There are things that you just shouldn’t do in real life.  While many of them involve cold lamp posts and electric sockets, there are many in the IT field that inexperienced pros do that are avoidable, but once done seemingly impossible to recover from.

I came across one such issue some time ago when resetting my Virtual Partner Technology Advisor Toolkit (blog on this to follow).  I visited a partner with only two of my server-laptops, and they asked me to demonstrate creating a Failover Cluster.  I destroyed my existing Cluster and did just that.  Unfortunately the next day I discovered that my third server-laptop, which had been a node on the now destroyed Failover Cluster.  When I tried to join it to the new cluster I got a message that ‘This computer ‘Host1.alpineskihouse.com’ is joined to a cluster.’

image

Failover Cluster Service is so much better than its predecessor, and this is a very simple fix.  However if you don’t know it you can end up banging your head against the wall and assuming you have to reinstall your OS.  Not the case.  It is a simple command line:

cluster node <computername> /forcecleanup

so in the case of my alpineskihouse.com laptop-server, I would open a command prompt (Run As Administrator) and type:

cluster node host1.alpineskihouse.com /forcecleanup

It only takes a few seconds… it cleans out the registry and allows that server to be joined to a new cluster.

I thought of this because I encountered the situation in the Virtualization Boot Camp Challenge at Microsoft Canada on Saturday.  If I hadn’t found that link, one of the teams (the team that was until the last challenge in first place!) would not have been able to complete the challenge, and would not have finished in Second Place.

One of the teammates asked me how they could have achieved the same results using the GUI (Graphical User Interface) but you can’t… the GUI tools are great for day to day tasks, and even a lot of the more complicated stuff, but the truth is there are just some things that you have to do ‘under the hood’… in the Command Prompt.

I repeat over and over the importance of knowing the command line tools for the common tasks that we do every day.  While I always tell them that they have to know them for exams, the truth is that sometimes we need to use them in our jobs.  When they argue that they shouldn’t need to learn command line tools I tell them (and am not lying) that the command line tools often separate the ‘computer guys’ from the IT Professionals… if you are going to have the respect to learn your profession and be able to do things right, then you have to know at least some of the command line tools, and if you don’t know them then you have to at least know how to find them and use them.

Now go forth and Cluster… or I guess cluster.exe Winking smile

A Gotcha For iSCSI Software Target Users

Having built and rebuilt several demo environments with Failover Clustering using the Microsoft iSCSI Software Target 3.3, there is one gotcha that you have to be careful of: Make sure that you leave at least one domain controller un-clustered, and not stored on the Software SAN.

Here’s the deal: Microsoft Failover Clustering requires Active Directory for authentication.  If one of your clustered servers goes down it won’t matter because the domain controller will just fail over to another node.  However if all of your nodes go down then when the servers come back up again there will not be an available domain controller for them to authenticate to, hence the Failover Cluster will not be able to come back up, hence the DC will not come up.  In other words, your network is toast.

How to prevent this from happening:

They say that an ounce of prevention is worth a pound of cure, and in this case they are right: There is a very simple solution that will prevent this issue, which is to build a non-clustered DC on direct-attached storage as your second (or third?) domain controller.  My Software iSCSI Target has far more CPUs and RAM than the software SAN needs, and since the OS was already Windows Server 2008 R2 SP1, I simply installed the Hyper-V role on that server (I had already done so because my System Center Essentials VM is also not clustered) and used it as a host.  I created a VM with the Server Core installation of Windows Server 2008 R2 SP1, which performs great as a domain controller while taking up few resources (RAM, CPU, storage).  Two days ago when my electricity was off for an hour it took me an hour to recover.  This morning when the same thing happened the recovery happened automatically.

How to recover if your cluster does go down:

imageIf you find yourself in this situation, it is easy to want to jump off a bridge.  Don’t!  Firstly most bridges high enough to hurt yourself from have fences that are harder to scale than they look, and secondly someone will have to clean up your body.  So here’s what you do:

  1. On your iSCSI Software Target, disable the Target.
  2. From the same console mount the LUN VHD locally. (Right-click on the device; under Disk Access click Mount Read/Write. (See screen shot)
  3. Copy the VHD files onto another Hyper-V host.
  4. Create a new virtual machine on that host called DC-Temp and instead of creating a new VHD, point to the one you just copied.  Make sure that the new VM is connected to a network that is accessible to your cluster nodes.
  5. In the newly created VM you will have to assign a static IP address to your NIC.  You will likely have the best chance of success if you assign the same IP address that the original virtual DC had.  Remember, you have created a new virtual machine, even though it looks the same… the hardware is new, so there is no IP adress.

At this point your cluster nodes might have to be rebooted so they can authenticate to a domain controller, which means you will be able to manage your cluster.  At this point you will notice that all of the Cluster storage has failed.  You have to rerun the iSCSI Initiator and rediscover the Target on each node, and then from Failover Cluster Manager bring all of the shared storage back on-line. 

Your cluster should now be healthy again; you will have to bring each of your virtual machines back on-line (from the Services and Applications tab in FCM).  Remember that somewhere about the time your DC is about to go on-line, take the temporary one down so you don’t get crashing IP addresses and SIDs Smile

When you are back up, please see the paragraph entitled: How to prevent this from happening.  You’ll avoid having to do this next time.

Good luck, and happy virtualizing!

Creating a Highly Available Virtual Machine (Video)

In the past few months I have written a great deal about creating Failover Clusters for Hyper-V virtual environments.  In this video I will demonstrate how we take a regular virtual machine created in Hyper-V Manager and make it highly available.

The first step that is not recorded in the video is to place the virtual machine files in the proper locations.  The VM in question was originally created on a stand-alone host on a local hard drive… obviously not a good place to start.  I shut down the virtual machine and then exported it to a file share that was accessible to both the original host and one of the hosts in my failover cluster (otherwise known as a cluster node).  Once that was done I imported the virtual machine to the cluster node, using the settings shown here:

image

These settings ensure that the virtual machine files will be imported into the default locations for my cluster… because I have Cluster Shared Volumes enabled the virtual machine files will be placed into C:\ClusterStorage\Volume1.  The settings also ensure that if I want to create multiple destination VMs from the same source files I can do so because it will create a new unique ID (UID) for the virtual machine.

Once the virtual machine is in the proper location, I can now go through the steps required to make it highly available, as seen in the following video.

IT Pro Connection & the TechNet Flash

A couple of weeks ago Microsoft released Microsoft iSCSI Software Target for download, and I was thrilled.  I immediately decided to write about it, but before publishing what would end up the first of three articles, I decided to ping my buddy Rick Claus, IT Pro Evangelist for Microsoft Canada, and ask him if he wanted the articles for the CanITPro Blog (http://blogs.technet.com/b/canitpro). 

The first article, entitled All for SAN and SAN for All,  was published on April 7th.  It was simply an overview of SAN technology, and why having a software SAN that was supported by Microsoft was hugely important.

The second article, Creating a SAN using Microsoft iSCSI Software Target 3.3, goes through creating the target (LUN).  I included explanations and screen shots, but stopped short of creating a cluster.  We simply create the LUN and then connect to it with the iSCSI Initiator.

The title of my third article was changed without anyone asking or telling me.  I had originally called it At Last… Redundancy for All! but I have to settle for a more descriptive Creating HA VMs for Hyper-V with Failover Clustering using FREE Microsoft iSCSI Target 3.3.  Fortunately that is all that was changed, and it went live this morning (Monday April 18), eleven days after the first in the series.

There was a bonus to this series too… Microsoft Canada sends out the TechNet Flash every week, with the top piece normally being an editorial from one of the IT Evangelists.  Last week they invited me to write 150 words introducing the technology and linking it to the articles.  If you’ve ever spoken to me, sat through one of my presentations, or read my blog (duh!) then you will know that I am not a man of few words… but my original submission was exactly that!  It was then extended by 30 to expand on a thought, but it was cool nonetheless.  I am told it is the first time the ‘top page above the fold’ piece was given to a community member, and I am very excited about that! 

I hope the pieces help you, and I look forward to hearing your comments and feedback!

%d bloggers like this: