Home » Clustering

Category Archives: Clustering

Failover Cluster OUs

I recently created a proof of concept for a client that was built into their production environment.  The POC required me to create a couple of failover clusters, so I got the names from the customer, and created them… like I’d done a thousand times before.

Several weeks went by and the customer called me and asked why they weren’t able to move the cluster name object (CNO) into an Active Directory (AD) Organizational Unit (OU).  Before I went chasing unicorns I opened an AD console and tried to move it myself.  Sure enough, I got an Access Denied response.

Hmmm… I was using my Domain Admin account… Wasn’t that supposed to give me the keys to the kingdom?

Like every object in your computer, the CNO has security properties, and by default, these protect them from being moved.  Of course, you can change these permissions if you like, but I am not a huge fan of doing that if I don’t have to.  Instead, what I would rather do is place the CNO into the proper OU when it is created, and then leave it there.

The problem with that is the New Failover Cluster Wizard… there is no option to place the cluster into a non-default OU.  Well, that’s not entirely true… the option is there, and it is hiding in plain sight.  In the Cluster Name dialogue box, simply enter the canonical name… CN=ClusterName,OU=Clusters,DC=SWMI,DC=ca

Simple, huh?  It’s so simple that I’ll bet you will want to know how to script this, so you can do it over and over again, right?  So watch this:

PS C:\> New-Cluster -Name CN=ClusterName,OU=Clusters,DC=SWMI,DC=ca -Node Server1, Server2, Server 3

Simple as pie, right?  It is… but make sure you get your canonical name right, otherwise this will fail, and you won’t know why.

No get back to work!

Help! Where is my Client Access Point?

So you are building a Scale Out File Server (SoFS).  You are all happy because you read one of my articles (or POSSIBLY someone else’s… but really, why would you hurt me like that?) and you know you are good to go.  You have your cluster, you have your drives, and you have created your SoFS role.  Now all that’s left to do is to add a file share to the role.

2015-04-13_15-26-35

Huh?  What did you do wrong?

Relax… you didn’t do anything wrong.  When you create the SoFS role in Failover Cluster Manager, it will take some time to propagate the namespace throughout Active Directory.  How long? UCA.  However depending on how large your network topology is, it can take a little time.  Just go for lunch, a smoke, maybe get out and stretch your legs, go for a jog… when you come back you should be ready to go!

Broken Cluster? Clear it up.

Three years ago I wrote an article about cleaning up nodes of clusters that had been corrupted and destroyed (See Cluster Issues… how to clean out cluster nodes from destroyed clusters). 

Unfortunately the cluster command has been deprecated in Windows Server 2012 R2, so we need to go to PowerShell… which frankly is where we should be going anyways!

PS C:\> Clear-ClusterNode –Cluster Toronto –Force

In this example we had a cluster named Toronto that is no longer accessible.  Unfortunately one of the nodes was off-line when the cluster was destroyed, so it didn’t ‘get the message.’  As such, when we try later to join it to a new cluster we get an error that the server is already a node in another cluster.

The cmdlet only takes a minute to run, and when you do run it you are all set… you will immediately be able to join it to another cluster.

For the fun of it, I have not figured out yet how to (natively) run this cmdlet against a remote server, so you can either do it by connecting to each server or…

Invoke-Command –ComputerName Server1 –ScriptBlock {Clear-ClusterNode –Cluster Toronto –Force}

I covered this option in a previous article (Do IT Remotely) which shows how to run cmdlets (or any script) against a remote server.

No go forth and script!

Cluster-Aware Updates: Be Aware!

When I started evangelizing Windows Server 2012 for Microsoft, there was a long list of features that I was always happy to point to.  There are a few of them that I have never really gone into detail on, that I am currently working with.  Hopefully these articles will help you.

Cluster Aware Updates (CAU) is a feature that does exactly what it says – it helps us to update the nodes in a Failover Cluster without having to manually take them down, put them into maintenance mode, or whatever else.  It is a feature that works in conjunction with our patch management servers as well as our Failover Cluster.

I have written extensively about Failover Clusters before, but just to refresh, we need to install the Failover Clustering feature on each server that will be a cluster node:

PS C:\Install-WindowsFeature –Name Failover-Clustering –IncludeManagementTools –ComputerName <ServerName>

We could of course use the Server Manager GUI tool, but if you have several servers it is easier and quicker to use Windows PowerShell.

Once this is done we can create our cluster.  Let’s create a cluster called Toronto with three nodes:

PS C:\New-Cluster –Name Toronto –Node Server1, Server2, Server3

This will create our cluster for us and assign it a dynamic IP address.  If you are still skittish about dynamic IP you can add a static IP address by modifying your command like this:

PS C:\New-Cluster –Name Toronto –Node Server1, Server2, Server3 –StaticAddress 10.10.10.201

Great, you have a three-node cluster.  So now onto the subject at hand: Cluster Aware Updates.

You would think that CAU would be a default behaviour.  After all, why would anyone NOT want to use it? Nonetheless, you have to actually enable the role feature.

PS C:\Add-CauClusterRole –EnableFirewallRules

Notice that we are not using the –ComputerName switch.  That is because we do not install the role service to the servers but to the actual cluster.  You will be asked: Do you want to add the Cluster-Aware Updating clustered role on cluster “Toronto”? The default is YES.

By the way, in case you are curious the Firewall Rules that you need to enable is the ‘Remote Shutdown’ rule.  This enables Cluster-Aware Updating to restart each node during the update process.

Okay, you are ready to go… In the Failover Cluster Manager console right-click on your cluster, and under More Actions click Cluster-Aware Updating.  In the window Failover – Cluster-Aware Updating click Apply updates to this cluster.  Follow the instructions, and your patches will begin to apply to each node in turn.  Of course, if you want to avoid the management console, all you have to do (from PowerShell) is run:

PS C:\Invoke-CauRun

However be careful… you cannot run this cmdlet from a server that is a cluster node.  So from a remote system (I use my management client that has all of my RSAT tools installed) run:

PS C:\Invoke-CauRun –ClusterName Toronto

You can watch the PowerShell progress of the update… or you can go out for ice cream.  Just make sure it doesn’t crash in the first few seconds, and it should take some time to run.

Good luck, and my the cluster force be with you!

Cluster Issues… how to clean out cluster nodes from destroyed clusters

There are things that you just shouldn’t do in real life.  While many of them involve cold lamp posts and electric sockets, there are many in the IT field that inexperienced pros do that are avoidable, but once done seemingly impossible to recover from.

I came across one such issue some time ago when resetting my Virtual Partner Technology Advisor Toolkit (blog on this to follow).  I visited a partner with only two of my server-laptops, and they asked me to demonstrate creating a Failover Cluster.  I destroyed my existing Cluster and did just that.  Unfortunately the next day I discovered that my third server-laptop, which had been a node on the now destroyed Failover Cluster.  When I tried to join it to the new cluster I got a message that ‘This computer ‘Host1.alpineskihouse.com’ is joined to a cluster.’

image

Failover Cluster Service is so much better than its predecessor, and this is a very simple fix.  However if you don’t know it you can end up banging your head against the wall and assuming you have to reinstall your OS.  Not the case.  It is a simple command line:

cluster node <computername> /forcecleanup

so in the case of my alpineskihouse.com laptop-server, I would open a command prompt (Run As Administrator) and type:

cluster node host1.alpineskihouse.com /forcecleanup

It only takes a few seconds… it cleans out the registry and allows that server to be joined to a new cluster.

I thought of this because I encountered the situation in the Virtualization Boot Camp Challenge at Microsoft Canada on Saturday.  If I hadn’t found that link, one of the teams (the team that was until the last challenge in first place!) would not have been able to complete the challenge, and would not have finished in Second Place.

One of the teammates asked me how they could have achieved the same results using the GUI (Graphical User Interface) but you can’t… the GUI tools are great for day to day tasks, and even a lot of the more complicated stuff, but the truth is there are just some things that you have to do ‘under the hood’… in the Command Prompt.

I repeat over and over the importance of knowing the command line tools for the common tasks that we do every day.  While I always tell them that they have to know them for exams, the truth is that sometimes we need to use them in our jobs.  When they argue that they shouldn’t need to learn command line tools I tell them (and am not lying) that the command line tools often separate the ‘computer guys’ from the IT Professionals… if you are going to have the respect to learn your profession and be able to do things right, then you have to know at least some of the command line tools, and if you don’t know them then you have to at least know how to find them and use them.

Now go forth and Cluster… or I guess cluster.exe Winking smile

A Gotcha For iSCSI Software Target Users

Having built and rebuilt several demo environments with Failover Clustering using the Microsoft iSCSI Software Target 3.3, there is one gotcha that you have to be careful of: Make sure that you leave at least one domain controller un-clustered, and not stored on the Software SAN.

Here’s the deal: Microsoft Failover Clustering requires Active Directory for authentication.  If one of your clustered servers goes down it won’t matter because the domain controller will just fail over to another node.  However if all of your nodes go down then when the servers come back up again there will not be an available domain controller for them to authenticate to, hence the Failover Cluster will not be able to come back up, hence the DC will not come up.  In other words, your network is toast.

How to prevent this from happening:

They say that an ounce of prevention is worth a pound of cure, and in this case they are right: There is a very simple solution that will prevent this issue, which is to build a non-clustered DC on direct-attached storage as your second (or third?) domain controller.  My Software iSCSI Target has far more CPUs and RAM than the software SAN needs, and since the OS was already Windows Server 2008 R2 SP1, I simply installed the Hyper-V role on that server (I had already done so because my System Center Essentials VM is also not clustered) and used it as a host.  I created a VM with the Server Core installation of Windows Server 2008 R2 SP1, which performs great as a domain controller while taking up few resources (RAM, CPU, storage).  Two days ago when my electricity was off for an hour it took me an hour to recover.  This morning when the same thing happened the recovery happened automatically.

How to recover if your cluster does go down:

imageIf you find yourself in this situation, it is easy to want to jump off a bridge.  Don’t!  Firstly most bridges high enough to hurt yourself from have fences that are harder to scale than they look, and secondly someone will have to clean up your body.  So here’s what you do:

  1. On your iSCSI Software Target, disable the Target.
  2. From the same console mount the LUN VHD locally. (Right-click on the device; under Disk Access click Mount Read/Write. (See screen shot)
  3. Copy the VHD files onto another Hyper-V host.
  4. Create a new virtual machine on that host called DC-Temp and instead of creating a new VHD, point to the one you just copied.  Make sure that the new VM is connected to a network that is accessible to your cluster nodes.
  5. In the newly created VM you will have to assign a static IP address to your NIC.  You will likely have the best chance of success if you assign the same IP address that the original virtual DC had.  Remember, you have created a new virtual machine, even though it looks the same… the hardware is new, so there is no IP adress.

At this point your cluster nodes might have to be rebooted so they can authenticate to a domain controller, which means you will be able to manage your cluster.  At this point you will notice that all of the Cluster storage has failed.  You have to rerun the iSCSI Initiator and rediscover the Target on each node, and then from Failover Cluster Manager bring all of the shared storage back on-line. 

Your cluster should now be healthy again; you will have to bring each of your virtual machines back on-line (from the Services and Applications tab in FCM).  Remember that somewhere about the time your DC is about to go on-line, take the temporary one down so you don’t get crashing IP addresses and SIDs Smile

When you are back up, please see the paragraph entitled: How to prevent this from happening.  You’ll avoid having to do this next time.

Good luck, and happy virtualizing!

Creating a Highly Available Virtual Machine (Video)

In the past few months I have written a great deal about creating Failover Clusters for Hyper-V virtual environments.  In this video I will demonstrate how we take a regular virtual machine created in Hyper-V Manager and make it highly available.

The first step that is not recorded in the video is to place the virtual machine files in the proper locations.  The VM in question was originally created on a stand-alone host on a local hard drive… obviously not a good place to start.  I shut down the virtual machine and then exported it to a file share that was accessible to both the original host and one of the hosts in my failover cluster (otherwise known as a cluster node).  Once that was done I imported the virtual machine to the cluster node, using the settings shown here:

image

These settings ensure that the virtual machine files will be imported into the default locations for my cluster… because I have Cluster Shared Volumes enabled the virtual machine files will be placed into C:\ClusterStorage\Volume1.  The settings also ensure that if I want to create multiple destination VMs from the same source files I can do so because it will create a new unique ID (UID) for the virtual machine.

Once the virtual machine is in the proper location, I can now go through the steps required to make it highly available, as seen in the following video.

%d bloggers like this: