We are currently investigating recent reports from customers experiencing higher than usual latency, packet loss, and/or intermittent connectivity when traversing certain network routes in our New York datacenter region.
Our team has performed a thorough review of our infrastructure side, and at this time we have not found any evidence of any hardware or software issues within RackNerd-operated infrastructure.
Based on our findings so far, this appears to be route-specific and may involve upstream network paths outside of our direct infrastructure, or potentially a core distribution router issue upstream at the datacenter. As such, we have escalated this matter to our upstream network providers for further investigation and review.
We will continue to closely monitor the situation and work with our upstream providers toward resolution. Further updates will be posted here as more information becomes available.
Thank you for your patience and understanding.
UPDATE: Our upstream has acknowledged our report and confirmed the issue, and this is being looked into by their Engineering team. We will provide additional updates as soon as we find out more details.
UPDATE 6:39 PM PDT:
We are seeing improvement and from what we can see, progress is being made, though we are awaiting final confirmation from upstream before calling this resolved, as we believe they may still be actively working on addressing this issue.
We will provide additional updates once available -- until this incident is marked as resolved you may experience intermittent connectivity or degraded network performance.
UPDATE 8:00 PM PDT:
Our upstream provider’s Engineering team is still actively investigating and working on the issue. While we have observed periods of improvement, we are not yet ready to mark this incident as resolved, as some customers may still experience intermittent connectivity, packet loss, or higher than usual latency depending on the network route being taken.
We are continuing to monitor this closely and remain in communication with our upstream provider. We will post another update once we have additional information or confirmation that the issue has been fully resolved.
Thank you again for your continued patience and understanding.
UPDATE 10:00 PM PDT:
Our upstream provider’s Engineering team remains actively engaged on this issue. At this time, we are continuing to see symptoms in some cases, meaning the impact may vary depending on where customers are connecting from, the network path being taken, and the specific destination/source involved.
While there have been periods of improvement throughout the evening, we are not yet comfortable marking this incident as resolved until we see sustained stability and/or receive final confirmation from upstream.
We understand how frustrating intermittent network issues can be, especially when they are inconsistent or route-specific in nature. Please know that our team is continuing to monitor the situation closely from our side, and we are continuing to follow up with upstream for further updates and resolution progress.
Until this incident is marked as resolved, customers in the New York region may continue to experience intermittent packet loss, elevated latency, or connectivity issues depending on routing.
We will provide another update as soon as we have meaningful new information to share, or once stability has been fully confirmed.
Thank you again for your continued patience.
UPDATE 12:05 AM PDT:
Our upstream provider is still actively working on this issue. The impact remains intermittent and route-specific, so some customers may see normal connectivity while others may continue to experience packet loss or higher latency depending on the route being taken.
We are continuing to monitor from our side and will post another update once we have more information or confirmation of sustained stability.
UPDATE 2:28 AM PDT:
We are continuing to work with our upstream provider as their Engineering team investigates this issue. At this time, symptoms may still vary depending on source location and network path.
We do not want to prematurely mark this incident as resolved until stability has been confirmed. Further updates will be posted as more information becomes available.
UPDATE 4:56 AM PDT:
This issue remains under active investigation with our upstream provider. We have observed additional periods of improvement, but intermittent packet loss or elevated latency may still occur for some customers depending on routing.
We are continuing to monitor closely and will update this incident once there is meaningful progress to share.
UPDATE 7:42 AM PDT:
Our upstream provider's Engineering team remains engaged, and we are continuing to monitor the situation closely from our side. The issue continues to appear intermittent in nature.
Until we have confirmation that the issue is fully resolved, this incident will remain open. We appreciate your continued patience.
UPDATE 1:30 PM PDT:
Our upstream provider’s Engineering team is continuing to investigate and monitor this issue. At this stage, we are seeing that the majority of customers with services hosted in our New York datacenter should no longer be impacted, and the remaining symptoms appear to be limited to certain IP ranges.
We are continuing to work with upstream to better understand the remaining scope and ensure the issue is fully stabilized before marking this incident as resolved. We will share additional information as soon as it becomes available.
Thank you again for your continued patience and understanding.
UPDATE 10:00 PM PDT:
At this time, this issue has stabilized, and we are closing this incident out.
Our team will continue to keep a close eye on the New York network, and if there are any additional updates to share regarding this matter, we will create a new status incident as needed.
We sincerely appreciate everyone’s patience and understanding while our team worked with upstream toward resolution.
