29th May 2024

San Jose Datacenter Packet Loss in West Coast Datacenters (due to Cogent carrier issues)

We are aware that servers located within our west coast datacenter locations (i.e. San Jose, Los Angeles, Seattle) are experiencing packet loss and/or higher latency than usual for routes traversing through Cogent carrier.

It appears that Cogent, one of our bandwidth providers within the mix is currently experiencing issues in the west coast. This means routes/paths that typically travel through Cogent may be affected.

Just to clarify, this is a wider/widespread issue throughout the US-west coast, and this is not a RackNerd-specific issue. As a prominent carrier, Cogent is a significant part of the Internet infrastructure, and its transit services are used directly or indirectly by many.

Network engineers at our upstream providers are also aware of the incident and are closely evaluating the situation.

UPDATE: The latest updates can be found directly on Cogent’s network status page here: https://ecogent.cogentco.com/network-status

As of now, the latest update from Cogent is:

“Customers transiting our network on the west coast of North America may be experiencing Latency and Packet Loss. This is due to multiple fiber cuts in our vendors network around the Los Angeles area. Our vendors have been contacted and are currently dispatching to further investigate the outage. There is NO ETR at this time. The master case is HD303782528.”

UPDATE: Although Cogent has not marked the issue as resolved on their official status page, we've observed signs of stabilization. To mitigate the situation, our upstream providers have temporarily rerouted traffic away from Cogent routes in certain locations to circumvent the problem. Consequently, if your usual routes tend to prefer passing through Cogent, you might notice slightly higher latency than usual as they may now be taking a different route, though it should not significantly affect accessibility. Additionally, please note that we cannot influence access to external services. If you are accessing external services such as hosted APIs or endpoints that rely on Cogent, these are beyond our control. On our end, we have made every effort to optimize our network to minimize disruptions.

UPDATE: It appears that Cogent has marked this as resolved as there is no longer any mention of it on their status page. Things also appear to be stabilized at this time. We will go ahead and close out this incident.