We are aware that CHI170KVM, CHI178KVM, CHI179KVM, CHI176KVM, CHI172KVM, CHI171KVM, CHI169KVM, CHI174KVM, CHI175KVM, CHI177KVM, CHI200KVM are currently experiencing connectivity issues.
We are currently looking into this node further and will provide additional updates on this matter once we have more information.
UPDATE: We sincerely apologize for the recurring issues affecting a portion of our Chicago location. The issue appears to be confined to a switch within one of our racks.
Our team is actively working closely with onsite at this time to evaluate the best path forward, with the goal of ensuring this is permanently resolved - including the possibility of a full switch replacement if needed.
To be clear, this issue is isolated to one specific rack within Chicago and is not impacting the entire location. We appreciate your patience and understanding while our team works through this.
UPDATE 3:45 PM PDT:
Services are now restored and operating normally.
To minimize further downtime, we did not proceed with a full switch replacement at this time, as that would have resulted in a longer service interruption. Instead, we corrected the issue in place and have brought affected services back online.
That said, there is still a possibility that the switch may be failing, so a replacement may be required in the near future. In the meantime, our team is proactively preparing and configuring a replacement switch in the background so we are ready to act if needed.
If you are still experiencing any issues, please open a support ticket and we would be happy to assist.
UPDATE: We are seeing a recurrence of the previous issues with the Chicago based host-nodes "CHI170KVM, CHI178KVM, CHI179KVM, CHI176KVM, CHI172KVM, CHI171KVM, CHI169KVM, CHI174KVM, CHI175KVM, CHI177KVM, CHI200KVM". Our team is investigating this further.
UPDATE 3:10 AM PDT:
We are continuing to work with our upstream provider as their Engineering team resolves this issue.
We are continuing to monitor closely and will update this incident once there is meaningful progress to share.
UPDATE 6:30 AM PDT:
As the switch in the rack appears to now have completely failed, we are actively working with remote hands on getting a replacement switch in the rack and connections moved over.
We will provide an update as soon as we have more information on this.
UPDATE 9:02 AM PDT:
The failed switch has now been fully removed from the rack and replaced with a brand-new switch. Services within this rack are now restored and operating normally.
At this time, we expect connectivity to remain stable moving forward for the affected Chicago host nodes. To reiterate, this issue was isolated to one specific rack within our Chicago location and did not impact the entire facility.
We sincerely apologize for the inconvenience caused by this incident and appreciate your patience while our team worked through the replacement.
