The usage graphs were disrupted yesterday due to various work on core routers, and will be reset today starting shortly. This is to fix a problem where the ping monitoring was not working correctly. Service should otherwise be unaffected by this change.

There appears to be a major issue with fibre links in to our node in Maidenhead. This is also affecting out office which is why there was a delay posting this.

Staff are on the way to site now to investigate.

Staff are on the way. As ever, backup equipment was not set up right in our offices which had led to a delay updating customers on this. Incoming calls are being handle by very few staff at present which is also causing delays taking calls. We expect staff to be back on it in a few minutes.

From what we can see most things in the data centre are working, e.g phones. But the fact we are unable to call the data centre suggests they have something major going on. More to follow. Staff still not managed to get on site yet.

Ok still no proper news, but the data centre is apparently swamped with openreach, suggesting something major, but also suggesting that thy are on the case.

We can confirm that all the fibre links to our rack in Maidenhead are showing alarms at present, but that power and everything else is fine. BT vans are on site and they have a big hole in the ground, but the BT engineers claim nothing wrong and they did not touch anything.

We are reporting multiple fibre failures to BT now.

Oh, and no clue why data cent were not answering the phone. We'll worry about that later.

We have confirmation that other customers in the data centre have also lost fibre links. BT are on the case but no ETA yet.

Ok. Clarifying rumours here. BT vans on site, apparently on unrelated things. Hole in ground is near site and some sort of construction work. May be unrelated. Maybe not. We are waiting up updates from BT now.

It appears that BT had internal system problems meaning they have not actually logged the fault internally until just before 4pm!!

Whilst delays in BT are frustrating, the fact this appears to be a fibre break with other people affected means they should already be working on it. We hope. We are still awaiting an update.

Further checks confirm that our link from our office to Maidenhead is only half broken, which is surprising. I.e. one of the two fibres is apparently working. This suggests a damaged rather than cut fibre.

So far, all we have, is that BT are running diagnostics. No more information yet.

Now waiting for a call from the Openreach engineer who is running diagnostics...

There are two Openreach engineers on site investigating. They have said that they're investigating someone elses's fault on site so sounds like they might have found a common problem.

BT engineer turned up on site to find two other engineers already working on it. So it is in hand. No ETA.

BT found what looks to be a physical fibre fault (so a break somewhere - notes added at 19:24) and confirmed at 19:37 that they are still working on it.

BT are still working on the break. No ETA yet.

Still no news. BT should call back, hopefully with more information, soon.

Whilst we expect BT to sort this over night, we may not have status updates until early morning. All systems are set up to come back on line as soon as BT fix the fibre.

BT are trying to put in a temporary fix to the problem. No details as to the root cause, and no time scale for a fix yet, but they've said that if the temporary fix fails they'll have to replace the broken fibre. If that happens it is unlikely to be fixed tonight.

Most affected customers seem to have come back online in the last few minutes.

BT have confirmed that this is a "permanent fix", by which they mean that any further disruption should be a part of planned engineering works.

Our links for Ethernet services have been restored and our office is back on line. We're waiting for a full explanation from BT.

More details: http://aa.net.uk/news-2011-fibre-break.html

BT fixed broken fibre.

At just before 17:00 today a minor routing announcement change was made. This was a very minor change and was not run as a proper planned work because it was so minor, or so we thought. Even so the change was not done in the middle of the day.

At 17:36 we were advised by MSO text that customers on Ethernet links were seeing problems. By 17:44 we had identified the cause of the problem and resolved it.

Technical details: The problem is in fact related to exhaustion of IPv4 space. We are involved in arrangements to swap IP blocks around between ISPs to free up some additional space, and this meant fragmenting one of our larger IP blocks. In principle this is simple, we announce the smaller blocks and drop the larger block announcement later. Unfortunately one of the smaller blocks is routed to Ethernet connected customers and was included in the list annonced at Telehouse in London.

The impact was that some of the Internet was not accessable to Ethernet customers, depending on routing. This included all of our ADSL lines and many services peering on LINX in London which could not see Ethernet lines. Unfortunately, as a partial failure it was not picked up by any of our automated monitoring which could still see Ethernet customers.

This was human error, and resolved within 8 minutes of being reported on a bank holiday weekend out of hours.

We will consider ways to manage this better - it is a trade off as we could have put more in automated control which would have stopped this specific error but could lead to more radical errors being possible with a lot less effort. We're reviewing procedures for checking changes like this before hand in future.

Sorry for the inconvenience.