Our monitoring has detected that there is a degraded/failed SSD on host node: "CHI119KVM". This has been resulting to unexpected reboots as well as performance degradation for VM's hosted within this specific host node in Chicago.
We are currently working on having an on-site technician replace the failed SSD with a new SSD. We will update this status incident once the degraded drive has been replaced, at which point, the RAID-10 array can rebuild accordingly.
UPDATE: We had replaced the failed SSD however are still showing symptoms of a degraded and unstable RAID array. Please standby while we continue to investigate this issue with the on-site technicians.
UPDATE: We are currently having the RAID array on CHI119KVM rebuild offline, in the interest of data integrity. We will continue to closely monitor the RAID rebuild status.
UPDATE: The RAID array rebuild has failed. We are currently having on-site technicians troubleshoot further and have provided them with clear instructions on how to move forward. Thank you for your patience and please be assured we are doing everything in our power to get this node back to normal operation as soon as possible.
UPDATE: On-site datacenter technicians are working on a RAID card replacement at the moment. We will update once we know more.
UPDATE: The RAID rebuild process is currently ongoing with the new RAID card. It is currently at 21%. We will continue to monitor the status of the RAID rebuild and as soon as it finishes, we will boot the server back online to production.
UPDATE: The RAID array rebuild is currently at 24%. We will continue to monitor the status of this rebuild and provide additional updates once available.
UPDATE: The RAID array rebuild is currently at 27%. We will continue to monitor the status of this rebuild and provide additional updates once available.
UPDATE: The RAID array rebuild is currently at 32%. We will continue to monitor the status of this rebuild and provide additional updates once available.
UPDATE: The RAID array rebuild is currently at 37%. We will continue to monitor the status of this rebuild and provide additional updates once available.
UPDATE: The RAID array rebuild is currently at 40%. We will continue to monitor the status of this rebuild and provide additional updates once available.
UPDATE: The RAID array rebuild is currently at 45%. We will continue to monitor the status of this rebuild and provide additional updates once available.
UPDATE: The RAID rebuild was ultimately not successful and did not move past 45% even after an extended period of time. This could be due to incomplete or unavailable RAID metadata. We are going to go with plan B, which is to migrate the VM's to another hypervisor/host node. We will begin working on VM migrations today.
You will know the migration is done by logging into the SolusVM VPS Control Panel, which will tell you what node your VPS is on. Once it says a node name that is not CHI119KVM, it will mean that your VPS is deployed on the new host node already.
If any action is required from your end, you will receive an e-mail notification from us. Please keep an eye out for your e-mail for any communication from us, if any.
UPDATE: We will be closing this incident out because we are working on Plan B, as described in the above update. If anyone has any questions - please feel free to reach out to us. We will reach out to you via e-mail if any action is required from your end.