Upstream failure

Incident Report for Kuxo

Postmortem

All signs point to a power failure, although the supplier has refused to accept this as fact.

Since a full incident analysis cannot be completed in full capacity, this is being closed without a postmortem or analysis.

Posted Jul 17, 2022 - 04:33 UTC

Resolved

Access has been restored, services are live.
We will communicate with the datacenter to look into the origin of the issue.
Posted Jul 12, 2022 - 05:51 UTC

Update

We are still awaiting a response from the datacenter.
Posted Jul 12, 2022 - 04:59 UTC

Update

We are waiting on a response from the dadatacenter
Posted Jul 12, 2022 - 02:06 UTC

Identified

An issue with a core router after the power outage is affecting availability and we're working to restore this.
Posted Jul 12, 2022 - 01:49 UTC

Monitoring

We have identified this as a major power outage with the data center and are currently in communication to confirm this due to the operational damage caused.

Services have been restored and we are monitoring the situation.
Posted Jul 12, 2022 - 23:34 UTC

Identified

At the moment we have identified routers in AS212728 have failed, this is causing traffic to be dropped before it reaches us and viceversa, the supplier has been notified and will take action on the issue.
We are working to confirm the cause of the issue as upstream visibility is limited.
Posted Jul 11, 2022 - 22:13 UTC

Update

We are continuing to investigate this issue.
Posted Jul 11, 2022 - 21:16 UTC

Investigating

We are currently investigating an issue at uksouth-1 with our upstream causing a large-scale network outage
Posted Jul 11, 2022 - 20:51 UTC
This incident affected: Platforms (Bare Metal Cloud (BMC)), Sites (Kettering, UK (uksouth-1)), Bare Metal Cloud (BMC) (Metal Management Service, Metal Provisioning Service, Metal Networking Stack), and Billing System.