INITIAL

Looks like a lot of lines via BT dropped, and a few reconnected, but dropped again.

Looks like BT only, not specific to any one of our LNSs, started with most of Stepney based BRASs going off.

UPDATE

What is odd is it seemed to be BRASs toppling one after another. Very strange.

UPDATE

Looks to have stabalised with around half of BT lines being off.

UPDATE

No update from BT yet.

UPDATE

Some of our customers with access to other ISPs using BT backhaul have confirmed that this issue is not just A&A/BT lines but is more widespread.

UPDATE

We still cannot get hold of BT, and still have no update from BT...

UPDATE

Customers are getting a web page which suggests various possible reasons but does not list BT as a possible cause, not amused.

We also got "21CN Incident Report :21CN WBC : STEPNEY GREEN : LOSS OF SERVICE ( CANCELLED ) : Reference 23981 : Issue FINAL" which makes no sense as they seem to cancelled the "incident".

Still no reply trying to contact BT

UPDATE

Finally. BT have confirmed they did get our report, sent within 60 seconds of the incident starting. And they are aware of a major issue. Trying to get more details.

So far "seems to be quite wide-spread, with multiple CPs, not yet got to root cause as we have also lost some management layers"

UPDATE

It looks like around half of lines are down, but all of RADIUS is down, so anyone reconnecting now will not connect - if you are on-line and working then don't touch anything!

UPDATE

Finally we get "INITIAL REPORT for IMT 23989/13 : Advanced notice of potential loss of Broadband services."

Yes, they say "advanced notice" for something taht started 90 minutes ago, and "potential" for something that has wiped out half our lines!

"Geographical area     : Nationwide - Geographical region unknown at present"

UPDATE

We are seeing lines coming back.

UPDATE

I'll leave the status post open for now and will update when we get explanation from BT. Thanks for your patience everyone.

UPDATE

Please note, on an unrelated note, we are re-mapping LNSs. This was to be done over night but we changed the routing while lines were down to avoid a second blip over night. This means graphs are showing without any history earlier in the day for lots of lines that were off during the outage. The graphs will be re-instated over night as usual and you can then see the outage clearly.

UPDATE

Report from BT regarding this outage:

http://wiki.aaisp.org.uk/index.php/File:IMT23989.pdf