We are aware that LAXSSD5005nerd3DC02 has dropped into read-only mode. We will be taking down LAXSSD5005nerd3DC02 for an emergency maintenance to correct some I/O related symptoms this hypervisor is facing.
We will provide additional updates on this matter once we have more information.
UPDATE: On-site technicians are taking the server offline now to run some diagnostics.
UPDATE: Although generally a hotswap drive replacement would suffice, given the unique circumstances that have occurred specific to this host node, we are currently performing an offline RAID array rebuild after replacing the failed drives in the RAID-10 array.
The RAID rebuild is currently at 29%.
UPDATE: The RAID rebuild process is currently at 50%.
UPDATE: The rebuild process was not successful, at this time - we are attempting to replace the RAID controller with an identical model and will re-start the RAID array rebuild process.
UPDATE: The rebuild process is currently at 19%.
UPDATE: The rebuild process is currently at 42%, we are continuing to monitor the status of this rebuild.
UPDATE: The rebuild process is currently at 47%, we are continuing to monitor the status of this rebuild.
UPDATE: The rebuild process is currently at 52%, we are continuing to monitor the status of this rebuild.
UPDATE: The rebuild process is currently at 57%, we are continuing to monitor the status of this rebuild
UPDATE: The rebuild process is currently at 62%, we are continuing to monitor the status of this rebuild
UPDATE: The rebuild process is currently at 67%, we are continuing to monitor the status of this rebuild
UPDATE: The rebuild process is currently at 73%, we are continuing to monitor the status of this rebuild
UPDATE: The RAID rebuild was ultimately not successful and did not move past 73% even after an extended period of time. This is due to incomplete or unavailable RAID metadata. The good news is, data is still intact as of right now because of the fact we run a RAID-10 array for redundancy. As such, VM's are now back online and we are going to go with plan B, which is to migrate the VM's to another hypervisor/host node. We will begin working on VM migrations between today and tomorrow.
You will know the migration is done by logging into the SolusVM VPS Control Panel, which will tell you what node your VPS is on. Once it says a node name that is not LAXSSD5005nerd3DC02, it will mean that your VPS has been successfully migrated and booted up to the new host node already.
If any action is required from your end, you will receive an e-mail notification from us.
UPDATE: We will be closing this incident out, if anyone has any questions regarding the migration please feel free to reach out to us.
