Drive Space Nightmares with Hyper-V

As you know, I have been using Hyper-V since before it was released, and am a huge proponent of the solution (although I am also a huge proponent of VMware).  The fact that Hyper-V is also included in Windows 10 makes my life easier – I use it on my Windows Client for several reasons.  In fact, at present I have four virtual machines on my Surface Pro 4, two of which I use on a very regular basis.

So when I notice from time to time that my C: drive is running out of space, I know immediately what the culprit is… my dynamically expanding drives have, in a word, expanded.

image

Not good… I need more than 3.18GB free space to be comfortable.  However when I look at the drives, I know that none of them are overly taxed… the VM I use most often (I use it to download files that I am not sure are safe so that I can ‘Sandbox’ them) is a dynamically expanding virtual disk that is as much as 80GB, but only 31GB is used.

image

That should be very comfortable… and yet there we see the usage.

image

A 53GB vhdx file for about 31GB of information.  It is easily explained of course… With a dynamically-expanding virtual hard disk the file gets bigger when you write to it, but when you then delete files and clean it up the file does not get smaller… or at least not automatically.  So what you have to do is this:

  • Shut down the virtual machine.  You cannot edit the disk while the VM is running.
  • In the Action Pane of Hyper-V Manager Select Edit Disk…
  • Click Next on the Before you Begin page.
  • In the Locate Virtual Hard Disk page navigate to the ‘offending’ vhdx file then click Next.
  • On the Choose Action page click the Compact radio and click Next.
  • On the Complete the Edit the Virtual Disk Wizard click Finish.
  • At this point the process will begin, and when it is done you should be good to go.

    PowerShell

    Yes, I know… you can do everything you want in the wizard… but let’s try a quick PowerShell cmdlet anyways Smile

    Optimize-VHD -Path C:\Hyper-V\Sandbox-PC\Sandbox-PC.vhdx -ComputerName MDG-SP4

    It only took a couple of minutes, and here are the results:

    image

    Almost 10GB freed up.  That makes life so much more comfortable.  Of course, since I use that virtual PC for these purposes a lot, I will want to keep an eye out for this creep and perform this script on a regular basis.  Hence why you might want to use PowerShell over the GUI.

    Converting Fixed Size VHDs to Dynamic Sized VHDs

    I was called in to help a company recently with a small Hyper-V environment.  They created a 2TB VHD file (they had good reasons for not using .vhdx files, but the steps in this article will work for both versions). 

    They realized that they had unnecessarily created the drive as a 2TB Fixed Size disk rather than a Dynamically Expanding disk.  While there are no real performance differences between the two in Server 2012 R2, when it comes to portability, the dynamically expanding drives have a huge advantage.  When they realized that they were planning for a SAN upgrade, the IT Administrator decided to take the preemptive step of converting the disk.

    It’s actually pretty easy… In the Hyper-V menu you click on Edit Disk…, and when the Edit Virtual Hard Disk Wizard comes up you go through the steps of selecting the appropriate file, and on the Choose Action screen select the radio button Convert.  On the next screen you specify the destination file, which eventually is where the problem is going to lie.  But for now you enter the filename, click Next, then click Finish to begin the process.

    For those of you who are PowerShell fans, you can accomplish the same task by running the following cmdlet:

    Convert-VHD D:\Hyper-V\HardDisk.vhdx -DestinationPath D:\Hyper-V\NewHardDisk.vhdx -VHDType Dynamic

    Great.  Now you go into the settings of your virtual machine, point it to the new file, and boot it up.

    Microsoft Emulated IDE Controller (Instance ID
    {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to Power on with Error ‘General access denied error’
    IDE/ATAPI Account does not have sufficient privilege to open attachment
    ‘D:\Hyper-V\NewHardDisk.vhd. Error: ‘General access denied error’
    Account does not have sufficient privilege to open attachment
    ‘D:\Hyper-V\NewHardDisk.vhd. Error: ‘General access denied error’

    Un-oh.  That doesn’t look good.  I click on the option to see the details, and it expands to the following:

    ‘VMName’ failed to start. (Virtual machine ID
    1DC704C1-6075-4F6C-B364-AFE4947304F3)
    ‘VMName’ Microsoft Emulated IDE Controller (Instance ID
    {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID
    1DC704C1-6075-4F6C-B364-AFE4947304F3)
    ‘VMName’: IDE/ATAPI Account does not have sufficient privilege to open attachment D:\Hyper-V\NewHardDisk.vhd. Error: ‘General access denied error’ (0x80070005). (Virtual Machine ID 1DC704C1-6075-4F6C-B364-AFE4947304F3)
    ‘VMName’: Account does not have sufficient privilege to open attachment
    D:\Hyper-V\NewHardDisk.vhd. Error: ‘General access denied error’ (0x80070005). (Virtual Machine ID 1DC704C1-6075-4F6C-B364-AFE4947304F3)

    Alright, it looks like I have a file permission error.  When I look at the Security tab on the Properties window of the source VHD file I see the following:

    image

    Great… all I have to do is add that user (group?) to the new VHD file, and I’ll be set.  The problem is… how the heck do I do that?  I certainly can’t do it from the GUI… No problem. 

    1. Open a Command Prompt with Administrator credentials.
    2. type the following command: icacls “D:\Hyper-V\NewHardDisk.vhd” /grant “NT Virtual Machine\1DC704C1-6075-4F6C-B364-AFE4947304F3”:F
    3. The response should read:

    processed file: D:\Hyper-V\NewHardDisk.vhd
    Successfully processed 1 files; Failed processing 0 files
    .

    If this is what you got, then you are ready.  You should now be able to start up your virtual machine without error.

    Good luck… now go forth and virtualize!

    Step-by-Step: Creating Differencing Disks

    Earlier this week I told you how you can use Differencing Disks in courseware, for lab development, for Virtual Desktop Infrastructure, for software development, and for any number of other uses.  They really are a very handy technology once you get to know them.  But how do you do that?

    Although usually I would show my Hyper-V demos on a Server, for today’s fun I want to remind you that you can still use Hyper-V on a Windows 8 desktop so I am using my Microsoft Surface Pro.  I have a virtual machine on this system called Windows 7 which obviously runs my Windows 7 Enterprise image.  I am calling this disk Windows 7.vhdx.  I want to try to install a piece of software that I am told might corrupt my Windows installation, and I don’t want to have to reinstall.  Instead I am going to create a Differencing Disk that will reference the original VHDX file, but will not make any changes to it.

    I open the settings for my virtual machine and navigate to the boot Hard Drive.

    clip_image0021. I select the hard disk (or disks) from the Hardware list. 

    2. I click New, having noted the name and location of the original disk.

    3. I create a new disk, going through the wizard selecting the file type (VHD or VHDX).

    4. In the Choose Disk Type screen I select Differencing instead of either Fixed size or Dynamically expanding.

    5. A screen that you might not have seen comes after you name your disk. In the Configure Disk screen browse for the parent disk – the disk that you are ‘replacing’.  Click Finish.

    **Note: Remember that if your VM has multiple virtual hard disks you may have to perform these steps on each one.

    At this point your Parent Disk will no longer be written to by your VM – all of the changes within your virtual machine will be written to the Differencing Disk that you just created.  It is useless without the Parent Disk, but the two together work great.

    There are a few caveats you should be aware of:

    1) Your Differencing Disk is not simply reliant on a disk called… When created it ‘connects’ itself to the Configure VHDX file; if that reference file is changed in any way then your Differencing Disk will no longer work.

    2) You can chain Differencing Disks simply by selecting another Differencing Disk as the Configure Disk.  When doing so make sure you document it well, because at any point if something gets corrupted along the way you had better know what you did in order to fix it!

    3) Notice that you were not asked to select a size of your Differencing Disk; that is because the size was set on the Configure Disk, and that is your limit.  Simple, right?  Wrong.  Remember that a single Configure Disk can be the parent of as many Differencing Disks as you want, and all of those Differencing Disks can add up quick, so make sure you keep an eye on your free space!

    There are no limitations on the Configure Disks – they can be either VHD or VHDX, Fixed size or Dynamically expanding.  As mentioned, your Configure Disk can even be another Differencing Disk.

    Differencing Disks are not new; the steps outlined in this article will work just as well in Windows Server 2008 R2 as they will in Windows Server 2012 or in Windows 8.

    Now here’s the fun part: PowerShell.  Yes, you can create your Differencing Disks in PowerShell just the same as in the GUI.  The cmdlet is actually pretty simple:

    New-VHD –Path “C:\VHDs\Windows 7 Diff.VHDX” –ParentPath “C:\VHDs\Windows 7.VHDX” –Differencing

    The outcome should look a little like this:

    image

    Notice though that you have a file that is 100GB (maximum file sixe).  That is because the base disk is 100GB.

    Now that you know how to do it, go play with Hyper-V… and know the Difference!

    By the way, even though I did this on Windows 8, you are much better off doing it on Windows Server 2012.  Download Windows Server 2012 and System Center 2012.

    Differencing Disks: Making courseware, labs, and boot camps easier!

    One of the features of Hyper-V that has made courseware development and delivery easier for me is the ability to create Parent (or Base) and Differencing disks.  It allowed me to reduce the ‘reset’ time at the end of a course (to prepare for the next delivery) from two hours per machine to one minute per machine.  If anyone wants to argue their value with me personally then that is the ratio they will have to argue.

    Here’s how they work: I create my virtual machine and configure it to where I will want the users to start – I install the operating system, join it to the domain (or create a domain controller), and install whatever extra software I would need.  I then shut down the virtual machine and create my ‘child VHD’ file as a reference (or Differencing) disk to the ‘Parent’ disk.  At that point I can literally reset the labs to that point in time by simply resetting the differencing disk.

    I have been working with the DPE team at Microsoft Canada delivering IT Boot Camps across the country for over a year.  At the end of each class we had an entire set of instructions to reset each set of machines, which essentially involved replacing the VHDX files from our backup.  Resetting the computers (22 of them) took upwards of three hours, and often enough the students discovered our Backup repository and made a mess of things, requiring us to restore that too from external media.  For our most recent camp, which we are calling Windows Server 2012 v2.0, we have altered things a bit, instead using VM snapshots – we can revert to the base snapshot very quickly – less than 5 minutes per computer, and we are usually done within 30 minutes… a substantial improvement to be sure. 

    The decision to use this technology instead of Base/Differencing Disks was not wrong; it is just a different way of doing things.  Having written and delivered several courses over the past decade (including several on virtualization) it has been my experience that while snapshots work well, there are a few ways that they can go wrong; it is usually human error that causes this, but they go wrong nonetheless.

    ***

    For my course From Virtualization to the Private Cloud I decided to go the other route – Base/Differencing Disks – because I felt that with so much focus on virtualization we might end up using snapshots occasionally, and although Hyper-V definitely supports long trees of snapshots, it is vital that they be properly documented, and it is just as easy to revert to the wrong one.

    Within my folder C:\VHDs I have a subdirectory called C:\VHDs\Diff where all of my Differencing Disks are stored.  At the end of the course I make sure the students delete all of the snapshots, and then turn off all of the VMs.  I specify Turn Off because all of the changes you might make are going to get discarded anyways, including the dirty shutdown.  I then go around and delete the AVHD directory, then replace that directory from the backup.  In less than a minute my entire environment (9-12 virtual machines) is reset and ready to perform as new the next day.

    However Base/Differencing Disks are not exclusively the purview of courseware.  Imagine having a test/dev environment where you needed to test different versions of the same set of code against the same master image.  Because a single Parent disk can support several Differencing Disks you can install the different versions and compare them to each other side by side.

    Imagine a VDI environment where your end users have Pooled VDI machines that needed to be reset at the end of every session.  A single Parent Disk could then support all of your VDI clients, and when the shift is over the Diff Disk simply gets reset and the next user to log on has his clean VM based on the master.

    As users begin to experiment and discuss Differencing Disks they are going to come up with several cases where they will be really helpful… I look forward to hearing from you what use cases you come up with, and how your experience with them is!

    In my next article you will read the Step-By-Step, how-to create, manage, and use Differencing Disks.  Stay Tuned!

    Converting VHDs to VHDX and other questions…

    Many of the articles I write for both The World According to Mitch and the Canadian IT Pro Connection come directly from people I meet through my travels.  They send me questions about technology by e-mail and rather than simply replying to them, if I feel the questions are relevant, I write them up as articles.  So if you meet me at one of my sessions and you ask me a question, do not be surprised if I ask you to e-mail it to me… oftentimes I will need to research the answer, but sometimes it is because I think that it would make for an interesting write-up.

    I have known Betty for as long as I have been going to her home town, and while she loves to give me grief I know that she is always attentive and learns from my presentations.  She recently sent me an e-mail with two very good questions on Hyper-V following my IT Camp on Windows Server 2012.

    QUESTION 1:

    I have several virtual machines that were created on Server 2008R2, and I would like to convert them to VHDX to take advantage of all the new features on Windows 2012. Is this possible?

    The process for exporting the virtual machine from Hyper-V on Windows Server 2008 R2 and then importing it as a virtual machine onto a host running Windows Server 2012 is fairly simple: Export, then Import.  However as I am sure you realize this does not convert the disk file format… ViVo in this case stands for VHD in, VHD out.  However the Edit Disk Wizard in the new Hyper-V is your friend here.

    1. Ensure that your virtual machine is powered down (or better yet disconnected).
    2. From the Actions Pane of the Hyper-V Manager click Edit Disk…
    3. On the Before You Begin page click Next.
    4. On the Locate Virtual Hard Disk page navigate to the location of our VHD file (use Browse if you like!).  Click Next.
    5. On the Choose Action page select the radio marked Convert and click Next.
    6. On the Convert Virtual Hard Disk page select the radio marked VHDX and click Next.
    7. On the second Convert Virtual Hard Disk page select the disk format you prefer (Fixed or Dynamically Expanding) and click Next.
    8. On the third Convert Virtual Hard Disk page enter the name and location of your new VHDX file and click Finish.
      Depending on the size of your source disk it may take a few minutes to create the new file; for larger disks you might want to run the Edit Disk Wizard to compact it before proceeding.  However once you are done you will have both the Source and the Destination disks, and all you have to do is edit the settings of your VM and attach the new drive, and you are ready to rock!image
  • Notice that your new file is about 145 MB larger than the original.  That is perfectly normal and nothing to be concerned about.

      PowerShell: I’ve Got The Power!!
      Thanks to folks like Ed Wilson and our very own Sean Kearney it is once again cool to use the command line… or rather, the cmdlet.   Nearly anything that you can do in the GUI can also be done in PowerShell, hence allowing us to create scripts to use at various clients or sites.  If you want to convert your VHD to VHDX in PowerShell here’s how:

    Convert-VHD -Path C:\ClusterStorage\Volume1\VHDsVM-1.vhd -DestinationPath C:\ClusterStorage\Volume1\VHDsVM-1.vhdx

    SNAGHTML6c7c42

    Again, it is important to remember that a) Your hard drive be off-line (or disconnected), and b) that once you have created the new VHDX file you must attach it to the VM before spinning it back up.  As well you will notice the difference in file size.  Nothing to be concerned by.

    (This cmdlet can also be used to convert VHDX files back to VHD files)

    QUESTION 2:

    Do the virtual machines have to be Server 2012 for me to take advantage of the new features of Hyper-V in Windows Server 2012, and especially the new .VHDX file format?

    Of course not.  Remember that the host and the guest have no real conception that the other is there; as long as you can install it on x86 hardware, you can install it in a Hyper-V virtual machine.  With that being said, there is a difference between can and is supported.  Remember that your Windows NT, 2000, DOS 3.3 and OS/2 Warp VMs are not supported by Microsoft… even though they will work just fine Winking smile 

    For Bonus Points:

    What is possible technologically is not always allowed legally.  It is important to make sure that all of the operating systems in your VMs are licensed on that host.  I have seen too many companies perform P2V migrations of physical servers that had OEM licenses attached to them, only to discover during an audit that they were out of compliance.  Make sure you have verified all of your licensing so that nobody will get their nose out of joint Smile

  • When I’m Sixty-Four… TERABYTES!

    Hard Disk Spindle
    Hard Disk Spindle (Photo credit: Fr3d.org)

    Okay, I am asking for a show of hands: How many of you remember 100MB hard drives? 80? 40?  While I remember smaller, my first hard drive was a 20 Megabyte Seagate drive.  Note that I didn’t say Gigabytes…

    Way back then the term Terabyte might have been coined already as a very theoretical term, but in the mid-80s most of us did not even have hard drives – we were happy enough if we had dual floppy drives to run our programs AND store our data.  We never thought that we could ever fill a gigabyte of storage, but were happier with hard drives than with floppies because they were less fragile (especially with so many magnets about).

    Now of course we are in a much more enlightened age, where most of us need hundreds of gigabytes, if not more.  With storage requirements growing exponentially, the 2TB drives that we used to think were beyond the needs of all but the largest companies are now available for consumers, and corporations are needing to put several of those massive drives into SAN arrays to support the ever-growing database servers.

    As our enterprise requirements grow, so must the technologies that we rely on.  That is why we were so proud to announce the new VHDX file format, Microsoft’s next generation virtual hard drive files that has by far the largest capacity of any virtualization technology on the market – a whopping 64 Terabytes.

    Since Microsoft made this announcement a few months ago several IT Pros have asked me ‘Why on earth would I ever need a single drive to be that big?’  A fair question, that reminds me of the old quote from Bill Gates who said that none of us would ever need more than 640KB of RAM in our computers.  The truth is big data is becoming the rule and not the exception.

    Now let’s be clear… it may be a long time before you need 64TB on a single volume.  However rather than questioning the limit, let’s look at the previous limit – 2TB.  Most of us likely won’t need 64TB any time soon; however over the last couple of years I have come across several companies who did not think they could virtualize their database servers because of 2.2TB databases.

    Earlier this week I got an e-mail from a customer asking for help with a virtual to physical migration.  Knowing who he reached out to, this was an obvious cry for help.

    ‘Mitch we have our database running on a virtual machine, and it is running great, but we are about to outgrow our 2TB limitation on the drive, and we have to migrate onto physical storage.  We simply don’t have any other choice.’

    As a Technical Evangelist my job is to win hearts and minds, as well as educate people about new technologies (as well as new ways to use the existing technologies that they have already invested in).  So when I read this request I had several alternate solutions for them that would allow them to maintain their virtual machine while they burst through that 2TB ‘limit’.

    1. The new VHDX file format shatters the limit, as we said.  In an upcoming article I will explain how to convert your existing VHD files to VHDX.  The one caveat: if you are using Boot from VHD from a Windows 7 (or Server 2008 R2) base then the VHDX files are not supported.
    2. Storage Pools in Windows Server 2012 allow you to pool disks (physical or virtual) to create large drives.  They are easy to create and to add storage to on the fly.  I expect these will be among the most popular new features in Windows Server 2012.
    3. Software iSCSI Target is now a feature of Windows Server 2012, which means that not only can you create larger disks on the VM, you can also create large Storage Area Networks (SANs) on the host, adding VHDs as needed and giving access as BitLocker-encrypted Cluster Shared Volumes (CSVs), another new functionality of the new platform.
    4. New in Windows Server 2012, you can now create a virtual connection to a real Fibre Channel SAN LUN.  As large a volume as you can create on the SAN is your limit – in other words if you have the budget your limit would be petabytes!

    With all of these options available to us, the sky truly is the limit for our virtualization environments… Whether you opt for a VHDX file, Storage Pool, Software- or Hardware-SAN, Hyper-V on Windows Server 2012 has you covered.  And if none of these are quite right for you, then migrating your servers into an Azure VM in the cloud offers yet more options for the dynamic environment, without the capital expenses required for on-premises solutions.

    Knowing all of this, there really is no longer any reason to do a V2P migration, although of course there are tools that can do that.  There is also no longer a good reason to invest in third-party virtualization platforms that limit your virtual hard disks to 2TB.

    Adaptable storage the way you want it… just one more reason to pick Windows Server 2012!