Our Los Angeles DC-02 datacenter (Multacom) is facing a facility outage, we are currently looking into this further with Multacom. We will update once we have additional information to share regarding this matter.
UPDATE: There is an active incident on the 61st floor of the Aon Center building. The Los Angeles Fire Department is on-site. This event has not affected our upstream provider, which is located on a different floor within the facility. However, as a precautionary measure, the building's power has been turned off pending further direction from the Los Angeles Fire Department. We will share further updates as they become available. Rest assured, we are in close communication with Multacom, and are taking all necessary precautions to bring services back online as soon as possible.
UPDATE: Based on ongoing conversations, LAFD is working with building engineers and is doing their best to restore power where possible. Building generators are also being activated, so RackNerd services should begin to be restored shortly. There is no particular update on which servers are coming online first, but this is just commencing now. Further updates to follow.
UPDATE: We are working very closely with our upstream, Multacom, and hundreds of physical RackNerd servers have already been powered back on. We will share additional updates once available.
UPDATE: For those curious, if you are experiencing slowness when attempting to access our website/client area -- this is due to the fact that the SolusVM API is facing elevated usage levels due to an increase in customers attempting to reboot their services. We are working on releasing a fix for this.
UPDATE: All shared/reseller hosting servers in Los Angeles DC-02 are back online, with the exception of the "fiber" server which is actively being worked on. VPS nodes are still in progress (hundreds of physical hypervisors are already back online), and are quickly being turned back on in phases.
UPDATE: Our website & client area is now stable and accessible, our support engineers are actively working to address the support ticket queue. RackNerd Management staff is actively engaged with our upstream as well.
UPDATE: Power has been restored to hundreds of additional physical servers just now, we will update once we have more information.
UPDATE: We are continuing to make significant progress in turning back online VPS node infrastructure. Hundreds and hundreds of physical hypervisors are already back online, and available for use.
UPDATE: We are still working closely with the facility staff on restoring power to the remaining devices, this is a process as it is being done in phases and we are also addressing some cabinets that have PDU issues. We appreciate your extended patience during this time.
UPDATE: Progress is still being made. We will share additional updates once available.
UPDATE: The team is still making headway, and we’ll update you with additional information on this status incident as soon as possible. Thank You again for your patience.
At this time, the following physical hypervisors are still powered off. We are in close communication with the facility staff, and these will be powered on as soon as possible:
LAXSSD6010nerd7DC02, LAXSSD6013nerd7DC02, LAXSSD7001nerd2DC02, LAXSSD1050nerd2DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3019nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4032nerd6DC02, LAXSSD5003nerd3DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD5007nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, LAXSSD1080nerd2DC02, LAXSSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD4034nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4001nerd6DC02, LAXSSD4036nerd6DC02, LAXSSD4037nerd6DC02, LAXSSD4038nerd6DC02, LAXSSD5801nerd2DC02, LAXSSD6020nerd7DC02, LAXSSD6024nerd7DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD103nerd1DC02, LAXSSD4042nerd6DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE: All shared/reseller hosting servers are stable and accessible. At this time there are <50 VPS nodes left to be powered on. We are working on the remaining VPS nodes and will share an update on here once available.
UPDATE: Network switches are the main culprit at this time - facility hands are working with the networking team for the fastest restoration. We understand the importance of uptime, and the impact of extended outages. Rest assured we are equally working towards the fastest restoration for the remaining devices.
UPDATE: We have brought online a batch of 30 physical servers, and we are still working on the remaining ones that are still offline.
UPDATE: We are currently going thru one by one on the devices that are still offline with the facility staff to address any individual server issues/power supply issues accordingly. Additional updates to follow. And the following servers are still powered off.
LAXSSD6013nerd7DC02, LAXSSD7001nerd2DC02, LAXSSD1050nerd2DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3019nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4032nerd6DC02, LAXSSD5003nerd3DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD5007nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, LAXSSD1080nerd2DC02, LAXSSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD4034nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4001nerd6DC02, LAXSSD4036nerd6DC02, LAXSSD4037nerd6DC02, LAXSSD4038nerd6DC02, LAXSSD5801nerd2DC02, LAXSSD6020nerd7DC02, LAXSSD6024nerd7DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE :We have brought online a few more physical servers, and we are still working on the remaining ones that are still offline. And the following servers are still powered off.
LAXSSD6013nerd7DC02, LAXSSD7001nerd2DC02, LAXSSD1050nerd2DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4032nerd6DC02, LAXSSD5003nerd3DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD5007nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, LAXSSD1080nerd2DC02, LASSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD4034nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4036nerd6DC02, LAXSSD4038nerd6DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02 LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE: We are currently going thru one by one on the devices that are still offline and we are still working on the remaining ones that are still offline. And the following servers are still powered off.
LAXSSD6013nerd7DC02, LAXSSD1050nerd2DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3019nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4032nerd6DC02, LAXSSD5003nerd3DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD5007nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, , LAXSSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD4034nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4036nerd6DC02, LAXSSD4038nerd6DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE: We are still working thru some cabinets that need PDU replacements and we appreciate your patience on the remaining devices that are still offline.
UPDATE: And the following servers are still powered off. Rest assured we are equally working towards the fastest restoration for the remaining devices.
LAXSSD6013nerd7DC02, LAXSSD1050nerd2DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3019nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4032nerd6DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD5007nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, , LAXSSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD4034nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4036nerd6DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE: We have more updates on the pending powered-off host nodes.
LAXSSD6013nerd7DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, LAXSSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD4034nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE: We have an update, as we resolved a few more host nodes as of now. Significant progress is being made, and we anticipate having a full conclusion/resolution on this matter soon. Please take a look at the pending host-nodes to be online.
LAXSSD6013nerd7DC02, LAXSSD930nerd2DC02, LAXSSD102nerd1DC02, LAXSSD4023nerd6DC02, LAXSSD3021nerd3DC02, LAXSSD3001nerd3DC02, LAX011nerd4dc02WIN, LAX014nerd4dc02WIN, LAX012nerd4dc02WIN, LAXSSD4020nerd6DC02, LAXSSD4002nerd6DC02, LAXSSD4006nerd6DC02, LAXSSD4009nerd6DC02, LAXSSD4024nerd6DC02, LAXSSD4039nerd6DC02, LAXSSD5006nerd3DC02, LAXSSD820nerd2DC02, LAXSSD940nerd2DC02, LAXSSD4005nerd6DC02, LAXSSD4008nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD5002nerd3DC02, LAXSSD5807nerd2DC02, LAXSSD6017nerd7DC02, LAXSSD810nerd2DC02, LAXSSD840nerd2DC02, LAXSSD880nerd2DC02, LAXSSD920nerd2DC02, LAXSSD970nerd2DC02, LAXSSD4010nerd6DC02, LAXSSD610nerd1DC02, LAXSSD6007nerd7DC02, LAXSSD6009nerd7DC02, LAXSSD900nerd2DC02, LAXSSD4012nerd6DC02
UPDATE: We are still working on it closely to bring back all the affected rest of the host nodes. We will share additional updates once they are available.
UPDATE: LAXSSD4002nerd6DC02, LAXSSD5001nerd3DC02, LAXSSD6017nerd7DC02, LAXSSD4020nerd6DC02, and LAX014nerd4dc02WIN are now back online.
UPDATE: We are currently assessing and evaluating the remaining devices individually from a hardware perspective. Following an event like this, some devices may require manual intervention or data recovery efforts. We are working through the list of the devices that are still offline, and will provide further updates as soon as they are available. We are doing our very best to restore services as fast as possible, as well as doing everything in our power to prevent any data loss. We're facing challenges, but we're actively working through them.
UPDATE: LAX012nerd4dc02WIN, LAX011nerd4dc02WIN and LAXSSD3021nerd3DC02 are now back online.
UPDATE: LAXSSD6013nerd7DC02 is now back online.
UPDATE: LAXSSD4006nerd6DC02 is now back online.
UPDATE: We are continuing to work on the remaining machines and have successfully brought several more online. However, some machines will require RAID array rebuilds. For the nodes that are experiencing data loss (e.g., fewer than 4 of 8 drives visible in a RAID-10 array), we are actively exploring solutions. We will provide customers with available options as soon as we have more information, so that customers who have yet to activate their disaster recovery plans, can do so with our proposed solutions.
UPDATE: The remaining devices are still being evaluated and worked on, we were able to bring up online a few more machines at this time. We will update once additional information is available.
UPDATE: A number of hypervisors are still undergoing RAID array rebuilds. Our team is working diligently to address the RAID-related issues that have arose on several nodes as a result of the unexpected power shutdown, particularly for machines with degraded or failed arrays. We are collaborating closely with our facility personnel to accelerate the process. We understand the urgency and appreciate your continued patience.
UPDATE: We're still working on these, additional updates to come soon.
UPDATE: Our efforts to bring the remaining hypervisors online are ongoing. We are taking extra precautions with the goal of data integrity. Further updates will be provided as we progress.
UPDATE: Progress continues on the restoration efforts, with more physical hypervisors now back to normal operation. Some nodes are still facing RAID rebuilds and potential data loss challenges, but our team is working diligently to try to avoid that situation. We will provide an update once more information is available.
UPDATE: We were able to salvage and bring back online an additional 2 hypervisors today thus far (LAXSSD104nerd1DC02 and LAXSSD5002nerd3DC02), and are still working through the final remaining list of nodes that require individual attention. We will update once more information is available, and we appreciate your extended patience during this time.
UPDATE: An additional node (LAXSSD102nerd1DC02) has been successfully salvaged and is back online. We are continuing our efforts and going through the list of the remaining nodes that require attention. We will update once additional information is available.
UPDATE: Good news, another node (LAXSSD3001nerd3DC02) has been salvaged and is now back online. We are continuing our efforts on working through the list of the final remaining nodes that require individual attention.
UPDATE: Another node (LAXSSD900nerd2DC02) has been successfully salvaged and is now back online. Additional updates to follow as we progress with the remaining nodes.
UPDATE: We are pleased to report that another node, LAXSSD4023nerd6DC02 has been successfully salvaged and is now back online. We will update as we continue to make progress/headway here.
UPDATE: Another node, LAXSSD4024nerd6DC02 has been salvaged and is now back online. We'll update once we have additional information.
UPDATE: We are still working through the final remaining list of nodes and will share additional updates once we have more information to share.
UPDATE: As of now (October 25, 10:30 PM PDT), we’re down to the last 3 nodes remaining on the list to work on, and these remaining ones are proving to be the most challenging to restore/salvage. We're continuing to work on these nodes, and will explore every possible option/avenue.
We understand the frustration of those who are still offline and have been waiting, and if you prefer to activate your disaster recovery plans and receive a new VM instead, please open a support ticket, and we’ll work to expedite that process.
We’ll keep pushing forward and will update everyone on this status incident as soon as we have more information. Thank you for your patience and understanding.
Final Update (Oct 26, 11:00 AM PDT): We are pleased to report that recovery efforts have successfully restored service to all customers whose data and VPS instances were recoverable. Any customers whose VPS instances could not be recovered have been contacted directly via email with details on their replacement VPS, which is fully operational.
At this point, everyone should be online, either with their original service or an issued replacement. If you haven’t yet received an email and believe your service was affected, please check your inbox for updates. You can also view your email log here: https://my.racknerd.com/clientarea.php?action=emails
If you encounter any issues, please check your instance in SolusVM and try rebooting if necessary.
While our team still very much remains all hands on deck as we work through an unprecedented workload, all customers are now back online, either on their original VPS or, in certain cases, on a fully operational replacement VPS. For any additional support, our team is here to help (just open a support ticket). We appreciate your extended patience throughout the next following days.
Thank you for your cooperation and understanding throughout this process.
