(NOTE: This article was written December 7, 2016. Not one word has been changed since that date. To understand why it can only now be published, read the article on this site called 107 Days: A Microsoft Security Nightmare. -MDG)
For reasons that will become obvious, I am going to delay posting this article until the issue has been resolved.
A few days ago a colleague of mine discovered the password to my Microsoft Account. I won’t go into the how and why… I knew that my password had been compromised and I took the immediate steps to change it.
Ok, I understand that things break… I tried a few times, and then I decided to follow the advice and try later. I trust my colleague not to actually use my password, so even though I felt uncomfortable with it being compromised, I knew I could wait a couple of hours.
Throughout the evening I tried (unsuccessfully) to change my password. As I was sitting with my father having dinner, as I had drinks and cigars with my friends… no joy, I still got the same message. ‘There’s a temporary problem with the service. Please try again. If you continue to get this message, try again later.’
I want to be clear… if my network had an error that was preventing users from changing their passwords I would consider it reasonably important, and I would take immediate steps to fix it. But having trusted Microsoft for so many years, I assumed this would be fixed eventually.
Four Days Passed.
Yes, it was literally four days before I decided that my passivity would not eventually lead to a solution. I sat down and figured out how to request support. I was hoping to be able to speak with a human being. Before I could, however, the Virtual Support Assistant got me to try this link and that link. It then made me go through seventeen steps to finally confirm that the account in question was mine… and once it confirmed that I really am me, it tried to reset my password… and I ended up with the same error message that ‘There’s a temporary problem with the service. Please try again. If you continue to get this message, try again later.’
Okay, it’s been nearly an hour… and I am chatting with someone who is quite obviously not their first round draft pick. After all, I asked for help with Outlook.com, not with something that people actually pay for. I spent twenty minutes explaining to him the situation, and the added (and I assume rare) complication that I have two accounts with the same address… my Office 365 account and my Microsoft Account are both the same address that are completely different. ‘Please don’t touch my Office 365 Account, I only want to change my Microsoft Account.’ This led to another five minute discussion on the meaning of the word change.
He had me fill out another form on-line. I did. At the end of that form I got a message that said that the product team would contact me within 24-48 hours to help me. I told the Support Agent that I had filled out the form. He told me that now I had to wait until they contacted me.
All in all, my Microsoft Account (which is the account I use for my MCT & MCP Benefits, Skype, and myriad other features) will have been compromised for the better part of a week… and there was nothing I could do about it. Yes, I could have contacted Answer Desk a few days earlier, so it would have been compromised for only three days. I want to know in what world is that considered an acceptable delay to be able to change a compromised password?
Some time ago I started using Multi-Factor Authentication (MFA) for many of my most important systems, which is why I am never concerned that my blog or my password vault could be compromised. For various systems I have a hard key (Yubikey) and soft keys (Google Authenticator and Microsoft Authenticator) which keep most of what I do safe. But most of the Microsoft systems do not support MFA and I am stuck with only a password. I use reasonably complex passwords so I usually am not concerned, but in a case where my password is compromised and I am not able to change it, I wonder how it is that a company as advanced as Microsoft (in this case) does not allow me to use MFA. I would love to be able to require my Yubikey in order to log in to Windows and many of the on-line systems I use, but it is simply not an option.
I am disappointed by Microsoft this week… and I hope that they take the lessons learned from this experience to improve. However I sit here today, thinking of the myriad occasions I stood on stage in over a dozen countries on five continents and defended Microsoft’s security systems as among the best in the world; I was always sure in my knowledge that I spoke the truth. Today I would not feel comfortable making that claim… and my faith in their systems, like shattered glass, will not be easily fixed.
I have held off talking about something for quite some time. I do not mess around when it comes to security, especially for my critical accounts. When the actual security of an account has been compromised, as was the case with my Microsoft Account, I do not advertise it.
On December 7th I sat in the Second Cup cafe on Bank Street in Ottawa and wrote an article called A Big, HUGE Microsoft Security FAIL. I wrote about how I had been unable to change my password and that their engine to do so was broken, but that it turned out it was not everyone, it was just me.
‘There’s a temporary problem with the service. Please try again. If you continue to get this message, try again later.’
It took several days for anyone at Microsoft to take me seriously, but my issue was finally escalated to a 2nd Level Support Tech named Gary (who, I want to be clear, was a nice guy, and as helpful as he could be under the circumstances). Gary and I got to know each other sometime in mid-December. Remember, the issue started happening the last days of November, I reported it on December 5th, and the case was escalated (grudgingly) around December 9th.
Gary spent a couple of hours trying to help, and then in discussions with the Product Engineering Team trying to get it fixed. By the end of the day he said something to the effect of: “Yeah, neither I not our engineers have ever seen a problem like this. It might take some time, but if you are willing we will work with you to get it fixed.”
Some time… It might take some time… that was on or about December 9th.
I am happy to say that the problem has now been resolved… As I sit and write this, with the resolution less than an hour old, it is 1:15pm, Wednesday March 22.
I spent nearly a decade touting the virtues of Microsoft’s security… and then from the day I informed them that my password had been compromised, and that a glitch in their system was preventing me from changing it, it took 107 days to resolve the issue.
So let’s take a quick rundown of some of the sites and services that are accessed with my Microsoft Account:
- Skype (One of the ways I communicate with hundreds of people)
- OneDrive (All of my files!)
- Microsoft Certified Trainer (MCT) account (including my MCT renewal, courseware downloads)
- Microsoft Certified Professional (including my MCP Transcripts)
- MSDN Subscription (including all my software licenses and keys)
- Windows Store (including credit card information)
- Microsoft Volume License Center (VLC)
- Microsoft Store (including credit card information)
- Microsoft Partner Portal
- MY WINDOWS COMPUTERS
And so, you can see, this is not like having my Words With Friends account compromised… This is extremely serious and far-reaching. This was… everything.
Once a week I would get an e-mail from Gary telling me that they had not yet resolved the issue… but they were still working on it, and he would continue to keep me informed.
On March 6th a hacker compromised my Skype account, and sent a link to dozens of my contacts with malicious content. Naturally those contacts let me know, and I reached out to Gary and told him that now that hackers had indeed compromised my account, they needed to resolve the issue and pronto. Gary replied with: “I have taken a look into your account, to look for any evidence of unauthorized access, and I did not see any. Was any account info changed? Can you still login?”
a few days later that he had not been able to open the embedded picture, and asked that I resend it as an attachment. Thank Heavens for that, because had he taken the next step immediately I would not have been able to renew my Microsoft Certified Trainer (MCT) credential in time.
So when Gary did finally get the picture (as seen above) he wrote (on March 18th):
If you received that message, then it could be that someone attempted to access the account.
To prevent that, I have placed a suspension on the account that will prevent any login activity. While my engineering team investigates this issue, no one will be able to break into the account. I have also left a note on the account so that the attacker will not be able to attempt to remove it.
Wonderful. You are suspending my account now, probably after the damage has been done, but all this is doing is punishing me. FIX THE DAMNED PROBLEM!
On the same day as I received this e-mail I wrote the following one line response:
Gary this is no longer acceptable. I am calling a lawyer.
On Tuesday (March 21) I received Gary’s reply:
In light of this recent reply, I have escalated this issue to a second team within Microsoft, and am awaiting to hear their response.
I understand the frustration, but please know that I cannot do anything to speed up the engineers and Ops teams working on this issue.
Wouldn’t you know it… The following day (that’s today, Wednesday March 22, 2017 – 107 days after I first reported the issue) I received a call from Gary that started with:
Well Mitch, it seems that when you threaten to call a lawyer things get done faster. I think we have solved your problem.
Indeed, before the phone call ended I had successfully changed my password.
One hundred and seven days after I first reported the problem.
One hundred and seven days since I told Microsoft there was a problem with their security.
One hundred and seven days since I told Microsoft that my account had been compromised, that someone had my password, and that I needed their help to secure my data and reputation.
One hundred and seven days. Actually it was only 105 days since I wrote the original article (which will be published shortly after this one, untouched since the original writing).
So why didn’t I publish sooner?
There are a handful… maybe four or five people who know the story and who understand some of my frustrations with this case. These are also people who know I have a great bully pulpit in the form of this blog. They have all asked me ‘Why didn’t you publish sooner?’ Two of them asked why I did not go to the mainstream technology media to let them know about this.
Simple… I have an account that is easy enough to guess, to which I could not change the password. If the wrong people knew about that they would have focused on getting that password and, once they had it, they knew I couldn’t change it. They would have literally owned me.
And so I sat quietly, seemingly patiently, waiting for Microsoft to fix the problem. I waited those 107 days knowing that when it was finally resolved I would a) breathe a big, huge sigh of relief, and b) sit down and write this piece, venting my facts and frustrations.
MICROSOFT! HOW DARE YOU? How can you let ANY problem, let alone one as serious as this, fester for so long unresolved? Do you think you owe me nothing? At this point I am still considering a lawsuit, and if you don’t think damaging my reputation and peace of mind is worth damages in a court of law then you are seriously misreading the system. You should be ashamed of yourselves, and you should be tracking down who is responsible for this travesty, this shame, and firing them.
I got that off my chest. I have, over the past two weeks, asked friends and colleagues for recommendations on lawyers. I might just reach out to one this afternoon. We’ll see.
When I posted my last couple of posts Microsoft was reporting that ‘All is Well.’ Dashboards were green, nothing to see here. That has now changed:
So we know now that the issues are pretty serious… at least, serious enough that they are now acknowledging them.
As for me, I have been having an issue with my Microsoft account that has been ongoing since December 5th. I have held off talking about it for security reasons, but with all that has been going on today, and the fact that yesterday my account might have been hacked, I am planning on writing about it tonight. And boy are you guys going to be shocked. All I will say for now is this: DO NOT TRUST MICROSOFT SECURITY.
Yes, I said it. For a decade I have been espousing the virtues and benefits of Microsoft’s security. Unfortunately I have had to change my position on this, and in a very big way. DO NOT TRUST THEM. It has cost me terribly, and I will tell you about it tomorrow.
So the worst fear of hundreds of millions of computer users has been realized today.
As I reported a little while ago, Outlook.com and Hotmail.com are down. But it goes much deeper than that. If login.live.com is down (see screen capture) that also means that OneDrive, Skype, and even XBox Live are out as well. If you are a Microsoft Certified Trainer and you were planning to download courseware today, that’s not available either. In fact, any service that requires authentication with a Microsoft Account is down right now. We are still awaiting word from Microsoft as to when these services might be restored. But for now, you (and I) will simply have to wait… in the figurative dark.
After receiving notification on my iPhone this morning that there was a problem with one of my Outlook.com accounts I went onto my computer to try it out. It did not work there either, so I did some investigating. It looks like this system is suffering a major outage today. I am not sure if this is strictly the e-mail, or if other services relying on the extremely popular Microsoft Account (formerly Microsoft Passport, Live ID) are out as well. One this is for sure… there are a lot of unhappy users on-line this morning who are not receiving their e-mail. More to come!
I have used Hyper-V for as long as there has been Hyper-V. Today I use it much less than I once did, but it is still handy for running VMs on my laptop. I run a particular VM called ‘Sandbox’ in which I do all sorts of things that I would not want to do on my live system… things that I can simply try and then wipe.
When I built the Sandbox VM I was spending a lot of time at home, and portability was not a huge issue. I ran it on one of my external drives, and it worked fine. I allocated 100GB and was good to go.
When I realized I was going to be on the road again I could just as easily take my external hard drive with me, but the shortage of USB ports on my Surface Pro meant making a decision… I was going to shrink the VHDX file and put it on my internal hard drive.
76GB free space. That’s going to be a problem.
Step 1: Shrink your partitions
My 100GB virtual hard drive (.vhdx file) meant that somewhere within the VM I had a 100GB partition (or at least a few partitions that added up to that). I had to shrink as much as I could.
- If you have a Pagefile.sys, Swapfile.sys, and Hiberfil.sys you should eliminate them now. Remember that even if you turn off Memory Paging the files don’t disappear until you reboot.
- Defragment the disk. We may not talk much about it anymore, but the old faithful defrag.exe C: still works.
- Use the Disk Manager console to shrink your C: as much as you can… but not too much. When I tried it I had the option to shrink it down to 11.5 GB… I’m pretty sure that would render my VM pretty useless. Pick a number that works for you. I chose 60GB.
- Using the diskpart tool delete any partitions at the end of your drive. I had a 450MB Recovery partition on mine.
Because it was a Recovery (read: SYSTEM) partition I needed to do the following:
Select Partition 5
Delete Partition OVERRIDE
Good… Now we can shrink the VHDX file.
In the older versions of Hyper-V this would have meant shutting down the VM. You don’t have to do that anymore… but you do have to run PowerShell as an Admin. Once you do:
Step 2: Shrink your VHDX file
The cmdlet is easy…
- Navigate to the directory where you keep your VHD file;
- Resize-VHD -Path .\Sandbox-PC.vhdx -SizeBytes 60GB
It will only take a minute and you will be done. Simple as pie!