We are investigating...
about 50% or so of our BE lines are seeing a mix of low level packet loss, higher latency and latency spikes. Some lines are also seeing very frequent disconnects. We have informed BE.
BE have reported problems regarding 7 exchanges which may be related to this. They say: "We are currently experiencing an unplanned outage at our Crownhill (WWCRWN), Devonport (WWDPRT), Plympton (WWPTON), Saltash (WWSALT), St Budeaux (WWSBUD), Plymouth (WWPYTH) and Plymstock (WWPSTK) exchanges. Our engineers are informed and investigating. Apologies for the inconvenience caused."
Lines have been back to normal since about 10:05pm.
We suspect BE may have been re-routing traffic in their network as a result of the exchange problems which meant many other lines were being routed over congested links. We've not had this confirmed yet though.
(This was also affecting non-AA BE lines.)
We have a very small number of lines that have an increase in their minimum latency, the latency had jumped up by about 10-20ms. We have passed these lines on to BE for their comment.
An update from BE regarding their problems with the exchanges: "Crownhill (WWCRWN), Devonport (WWDPRT), Plympton (WWPTON), Saltash (WWSALT), St Budeaux (WWSBUD), Plymouth (WWPYTH) and Plymstock (WWPSTK): The links are back restoring all connectivity. Engineers are still investigating for the cause so future failures can be prevented. "
We've had an update from BE as they have been investigating this, here is the information "I have spoken to O2 throughout the day about this case and it is believed that the cause may have been due to the exchange outages in the south-west areas, as you suspected. The case has been elevated to O2's Network team who have since passed it to their Transport Services team that will be responsible for investigating the exact cause of the issue and also reaffirming that the problem has been resolved."
BE did have an outage affecting a number of exchanges (listed above) in the south of England. We suspect this also caused re-routing of traffic within their network which caused latency and loss on other lines.