04/04/2021 14:48 PST: We are currently aware of a partial outage impacting some servers hosted in our Utah datacenter location. Our datacenter team is currently looking into this. We will update this incident with more information as it becomes available.
Update: We are currently still looking into the situation and will update this incident with more details shortly.
Update #2: There was a utility power outage this morning, at which point load was successfully transferred over to generators. The on-site datacenter engineers are currently looking into why this was not sustained. We will have services restored as soon as possible and thank you for your patience.
Update #3: While attempting to receive communication from our upstream (the facility/building) we unfortunately have been provided limited information. It appears all hands are on deck thus limited information from our upstream. The folks providing updates are operating with limited information as the facility staff works hard to restore power. The information we have received so far is that switching over to generators caused the extended downtime, with difficulties to restore to ground power as a result. We will continue to dig for more answers / timeframes and update this incident accordingly with more information. We are just as eager and realize the limited information causes frustration.
Update #4: We received an update upstream. The issue was identified as one of the building’s generators which has historically worked for years, was recently load tested and had a mechanical failure. This resulted to power being cut to the core routers and fire suppression system being activated as a precaution. Unfortunately, the city’s fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. At this time we are awaiting for an emergency inspector to arrive onsite to give the approval so that power can be restored to cabinets and so that servers can be powered on again. We appreciate your patience and we will continue to update this incident with additional information as we are able.
Update #5: The facility is still awaiting for the inspector to come out and provide approval. Once we have additional information, we will be sharing it on this page.
Update #6: The facility hands are making some great progress, and the inspector is on-site. Some servers in Utah are powering up, and we will continue to update this status page with more updates as available.
Update #7: Here is a brief overview of the events as they transpired. Sunday afternoon the city power was disrupted and, as designed, the facility's backup generators automatically switched on. However, during that transition, one of its backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol. No fire damage was inflicted on customer's servers. However, there are a handful of machines that may need to be inspected for water damage, before they can be powered on (and/or rebuilt).
The current course of action that the facilities team is working on, is to restore power throughout the datacenter in accordance to the inspector's guidelines. We will continue to update this status page with additional updates once available.
Update #8: A majority of our servers in the Utah datacenter that did not experience water damage have been powered on. A couple more cabinets remain and the electricians are working on it.
Additionally, we are working on preparing a plan of action to rebuild servers that did experience water damage. As of right now, the preliminary plan is to build new machines of identical/similar specs, and then move the drive(s) over, followed by routing the same IPv4 space over to the new machine so that your server is back up seamlessly as it was before. As the drives were not submerged throughout this incident, the chance of data loss is very unlikely. Please trust that we are doing everything we can in cooperation with the on-site facility team to restore service for everyone in Utah.
We will continue to share additional updates here as they are made available to us.
Update #9: The facility staff in Utah are working to rebuild servers that experienced water damage. This is being done in a triage process system. As mentioned earlier, data loss is unlikely as drives were not submerged.
Update #10: We also wanted to mention that we've successfully restored some customers, upon request, to other datacenter sites of ours, such as Los Angeles and New York. We understand uptime is important, and for customers who are time-sensitive we do have alternatives available. If you are an affected customer in Utah and have offsite backups of your own, or have obtained an offsite backup service with us and would like to discuss getting deployments set up in other datacenter locations of ours, please email [email protected] and we will review our available inventory and work with you to get you set up as soon as possible in another datacenter location. Of course, at no additional charge to you.
Update #11: The servers that did not experience water damage are back online and accessible again. With regards to the servers that did experience damage, our upstream has indicated that while they are still assessing the situation, it could ultimately be additional weeks of downtime in efforts to rebuild the servers and restore service. While the risk is extremely minimal in terms of data loss as the drives were not submerged, it is important for you as well as us to understand that your uptime is critical, and service is ultimately what you seek for at the end of the day. We are open to help restore affected customers with servers available in other datacenter locations of ours. We have been reaching out and communicating to our customers that were affected by this Utah incident to offer alternative solutions. If you would like to explore the possibility of deploying service in another location, please feel free to email [email protected] and we will work with you to get you set up as fast as possible in another location. There will be no additional charge/cost to you for this.
Update #12: The on-site team in Utah is making great progress with getting more of our affected servers rebuilt & back online. The on-site team is still working to rebuild some of our affected servers that are still offline, but great progress was definitely made today and a good amount of our customers have noticed their servers start responding again today.
On our side we are also making great strides in communicating with our affected customers and working closely with them on solutions. For time sensitive situations, we have relocated a handful of customers who were open to it - to alternative datacenter locations such as Los Angeles and New York. As mentioned in the last update if this is something you were open to exploring please definitely email [email protected] so we can review our available inventory and get you set up in another datacenter location ASAP. For customers that are needing to stay in Utah (i.e. due to data, or due to specific location requirements or some other reason), please know that everything in our power is being done to help expedite and we hope to have everybody back online soon.
Update #13: A lot of servers in Utah have been re-racked and brought online each and every day in the past week, and by now, all infrastructure in this location should now be back online and accessible. Please feel free to contact us if you are experiencing any issues or if you have any questions.
