Rackspace hosted Exchange suffered a devastating failure beginning December 2, 2022 and is still ongoing since 12:37 AM December fourth. Initially described as connection and login problems, the guidance was ultimately upgraded to reveal that they were dealing with a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be dealt with.
Consumers on Buy Twitter Verified reported that Rackspace was not reacting to support emails.
This has actually been quite the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours approximately. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer privately messaged me over social networks on Friday to relate their experience:
“All hosted Exchange clients down over the previous 16 hours.
Not exactly sure the number of companies that is, however it’s considerable.
They’re serving a 554 long hold-up bounce so people emailing in aren’t familiar with the bounce for several hours.”
The official Rackspace status page used a running upgrade of the blackout however the preliminary posts had no details aside from there was an outage and it was being examined.
The first official upgrade was on December 2nd at 2:49 AM:
“We are examining a concern that is impacting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace began calling it a “connectivity problem.”
“We are examining reports of connectivity concerns to our Exchange environments.
Users may experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”
By 6:36 AM the Rackspace updates explained the continuous problem as “connection and login problems” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “examination phase” of the interruption, still trying to determine what failed.
And they were still calling it “connection and login concerns” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
4 hours later on Rackspace referred to the scenario as a “considerable failure”and began providing their clients complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the issue and might bring the system back online.
The main guidance specified:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any more problems while we continue work to bring back service. As we continue to work through the root cause of the concern, we have an alternate service that will re-activate your ability to send out and get e-mails.
At no cost to you, we will be offering you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 up until more notice.”
Rackspace Hosted Exchange Security Event
It was not up until almost 24 hr later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was suffering from a security incident.
The statement even more revealed that the Rackspace professionals had actually powered down and detached the Exchange environment.
“After additional analysis, we have actually determined that this is a security occurrence.
The recognized effect is separated to a portion of our Hosted Exchange platform. We are taking needed actions to evaluate and secure our environments.”
Twelve hours later that afternoon they updated the status page with more information that their security group and outdoors professionals were still working on fixing the failure.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not released details of the security occasion.
A security occasion usually involves a vulnerability and there are 2 extreme vulnerabilities currently in the wile that were patched in November 2022.
These are the 2 most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack enables a hacker to check out and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assaulter has the ability to run harmful code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“An authenticated remote opponent can perform SSRF attacks to escalate advantages and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the assailant can potentially gain access to other resources through lateral motion into Exchange and Active Directory environments.”
The Rackspace failure updates have actually not shown what the particular issue was, just that it was a security event.
The most current status update since December fourth stated that the service is still down and clients are encouraged to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in resolving the occurrence. The availability of your service and security of your information is of high importance.
We have actually dedicated extensive internal resources and engaged world-class external proficiency in our efforts to reduce unfavorable effects to customers.”
It’s possible that the above noted vulnerabilities are related to the security incident affecting the Rackspace Hosted Exchange service.
There has been no announcement of whether client info has been compromised. This event is still ongoing.
Included image by Best SMM Panel/Orn Rin