Since around 1am it looks liike custoemrs are having problems connecting. We have done an LNS upgrade which means around a third of customers are off line. This is being investigated now.

This looks related to the upgrade and seems to be related to platform RADIUS, but we are still investigating now.

Lines are coming back, but we are working to understand the issue as it appears to be RADIUS or BGP related.

The fix to this makes no sense, and will be reviewed later in the morning. The issue appears to relate to platform RADIUS from BT. We changed the way the endpoints are announced to BT and it all sprung in to life, but that makes no sense. The LNS upgrade does not explain it either. There may be some planned work as a result of this to investigate further.

The problem was not jus BT, but the fix we found was a result of changing BGP to BT which caused it to flip which RADIUS it was using. We have confirmed the issue is platform RADIUS and caused by LNS upgrades. The platform RADIUS is currently directed to a non-upgraded box and so BT, BE and data SIMs should all be able to connect now.

There will be s/w upgrades over the coming nights to correct this.

It looks like some issues with BE lines still, which we are working on

We have found the underlying issue, and will be re-jigging lines over night, sorry for any inconvenience.

This will me some loss of graphs.

The error has been corrected, and lines are being moved between the two affected LNSs now. Graphs for around 1/3 of customers will be missing before around 3:30 this morning.

Even with two working platform RADIUS servers, Be lines are still not connecting. We are not sure if their RADIUS have somehow blacklisted us because of lack of response, or what. Still working on that.

We have asked fluidata (who manage the BE connection) to check their RADIUS servers for us. This may come back anyway over night if it is a timeout/blacklist issue on their RADIUS.

We have just had a resonse from Fluidata and they seem to have given their end a kick and all is well now.

Graphs lost for some lines before around 03:30, and some other LNS updates over night during the week.

Not sure what yet

Routers and LNSs are all up and did not restart or anything, not clear what happened yet.

The issue lasted a few minutes and seems to be over with services back to normal. So far we can see that a number of BGP sessions internal to the Telehouse rack restarted. It almost looks like a switch restart, but we are still checking.

Checks confirm nothing rebooted, not even one of the network switches, so it is not clear what happened at this stage, we are still investigating, but the problem itself is over at present.

We have just seen a lot of 20CN and 21CN lines around scotland drop.

We are reporting this to BT now. 

Looks like the BT metro node in Scotland is down as it is affecting 20 and 21CN lines. 

BT operate are aware and currently investingating. 
BT will update us when they have more news. 

BT have raised a incident against Scotland, still awaiting an update as to what is happening.

Bt say they have lost service to 2 10Gb circuits. Diagnostics still ongoing.

BT engineers are on there way to site.

BT have an engineer on site and a second engineer en route to assist.
Should have further news within the hour.

BT are currently working to restoe service.

Lines coming back up.

From BT:

The affected router has had stopped process restarted and the controlled recovery of the device is being closely monitored. Service should be available within the next 30 minutes. The geographical area of impact is across Scotland, the North of England and Northern Ireland, on traffic that was terminating on this router. 

Some more info about this outage on the ISP Review website: http://www.ispreview.co.uk/index.php/2012/12/bt-20cn-and-21cn-broadband-isp-lines-down-in-scotland-and-north-england.html

More issues, this time broadband lines.

To summarise - a routine upgrade this morning resulted in a problem, but only when the last box was upgraded. We had upgraded routers during the week, as we don't upgrade everything all at once. However the nature of the problem was such that only when the last box was upgraded did problems happen.

The effect was a few seconds with no routing, and then a period where all worked. This repeated for around 6 minutes until fixes were applied.

We can see the cause, but this effect was not foreseen, so we are looking in to how we can avoid such issues in futre.

Sorry for any inconvenience.

Note: This would not have causes broadband lines to drop sync or lose connection at all - this was entirely a routing issue to the wider internet.

The issue is exactly the same as we had in Maidenhead so has been fixed quickly.

There have been reports that some parts of the internet damped flapping routes for some minutes after we applied a fix. These seem to be relatively few and far bewteen as overall traffic levels returned to normal levels immediately. Obviously there is not a lot we can do over such policies in individual ISPs.

One of our core links to BT droped affecting a third of our customers.

Lines are recovering and re-logging in.

It may take a few minutes for all lines to reconnect.

More details to follow

Most lines are back online. There are some which are not though and are 'flapping' (going up and down).

To restore service we are clearing lines off the LNS (core router) that the faulty BT link is on. This should then fix these remaining lines. - a side affect is that it will also mean that some BE lines will disconnect and reconnect. We apologise for this.

We are still seeing some lines 'flapping' we're investigating the cause of this.

More lines are now in this 'flapping' state, we are still working on this.

We have restarted the 'A' LNS in an attempt to stabalise connections.

Lines on the 'A' LNS are now looking stable. We're working on the other LNS's.

Having restarted our LNs's lines are reconnecting and are remaing stable.

(Graphs for today up to ~2:15pm would have been lost though.)

ADSL lines are looking stable now.

We are still in contact with BT about the link that is currently still down, and we will be reviewing how we can cope better with this type of outage in the future.

It is not clear exactly why the loss of a BT link has caused things to be come unstable. Restarting all of the LNSs had resolved this, and we are investigating this. We suspect there is an issue with the routing to BT when one of the links is down, and so may be doing some planned work in due course to make changes that could improve matters.

In the mean time we are trying to get the failed link back up and working.

BT have raised an incident and escalated it internally.
More update when we hear back from BT. 

We have seen a couple of knock on effects with the flapping lines - the LNSs are very fast, and so handled lines flapping way faster than the RADIUS accounting database can keep up. As a result, things like colours of lines shown on clueless control pages, and text updates for lines flapping, are a tad behind. It is all catching up.

We also experienced a knock on effect with database updates causing some secondary servers to get busy. This is something we are fixing properly in the long term, but it meant some issues with telephony, which were a bit unexpected. That has now been resolved, and in the longer term these secondary servers should no longer struggle when there are a lot ofg database updated.

Several people have reported some web pages being slow - it seems some of the sessios have come up with low MTUs and this is affecting a number of customers. We are clearing affected tunnels manually to rectify this so some people may see a PPP restart.

We can see how the timing of the LNS restart we did could have resulted in the MTU settings being wrong on the first few tunnels, and have made config changes for the future. I have cleared the affected sessions and they have reconnected cleanly.

A key thing here is ensuring that the niggles of today are permanently fixed - we are, of course, working on that. Most things are fixed for long term, and the one issue (the flapping) is still being investigated and we have an idea what to do for that as planned maintenance. Sorry for any inconvience.

Looking at the overall usage graphs for transit links, we can see that even though some lines were flapping a lot, and most had some issues, overall people were still able to use the internet.

We have normality, I repeat, we have normality. Anything you still can't cope with is therefore your own problem...

The accounting has all caught up at last, sorry for the delay, and the various delayed texts and emails.

Thank you all for your patience.

BT have an engineer on site investigating the fault. Customers are conected via the other fibre links we have though.

BT have fixed the broken fibre by replacing the service card.

This will be monitored.

Our monitoring is reporting that the fibre is not stable, and has been dropping. This fibre is not in use by customers so it's not affecting any customers. This has been passed on to BT.

BT have been testing this fibre today, they are not seeing a problem and the drops that we saw this morning have not happened since 9am. We'll continue to monitor it though.

BT confirm that they are seeing alarms on equipment, the investigation continues.

We are still working with BT to resolve this fibre issue. On of our staff was at the datacentre yesterday and BT have changed the service card yet again. However, we're still seeing the port flapping, and BT see alarms at their side. Further BT engineers are tasked to work on this again today.

In the meantime, we continue to use the other fibres that are part of this 'resilient' set so that service is unaffected.

BT replaced the service card on the NTE at our side of this circuit, and the fibre has remained stable.

We're seeing odd routing problems at the moment, this will affect internet access for DSL and Ethernet customers in London. Staff are investigating.

This looks to have recovered. We're still investigating the cause though.

We're investigating a packet loss / routing problem at the moemnt, this will be affecting broadband customers, more details to follow.

Looks like this was something quite major and country wide affecting various interconnects that the internet uses.

Things are geting back to normal now.

ADSL lines which did go down are re-connecting.

Service has been restored. This looks to have been caused by a problem within the LINX peering network which affected other networks and interconnects accross the country.

We've re-enabled our LINX peering.

At 08:06:58 all 20CN and 21CN lines routed via Manchester dropped and at the same time we stopped seeing any platform RADIUS requests from BT.

This suggests a major routing issue within BT. It means that nobody can re-connect any broadband line.

We are trying to get an answer out of BT on this.

BT have a string of open issues covering Manchester area. No ETA yet.

We have confirmation that this is not just affecting us (thanks Thomas). BT are investigating.

Whilst BT are being a tad inconsistent in telling various ISPs what is happening, we have heard from another ISP that "Engineer is onsite at Manchester with a spare card. Changeout is about to commence with the assistance of vendor  support. Next update will be at 10:20"

BT are not confirming the statement that a card is to be replaced, and in fact appear to be totally ignoring me now.

Looks to be working.

Manchester lines reconnecting

Platform RADIUS working

We are waiting for more detail on BT on this, and how it is they appear to depend entirely on a single card in Manchester for the entire BTW platform.

Lines on our "b.gormless" LNS dropped. We're investigating. Lines are coming back.

It looks like BT. Still investigating.

It looks like BT's link to b.gormless has gone down. We've taken b.gormless out of service.

BT fibre link is down - lines are moving to the backup LNS automatically.

BT have raised a fault and are investigating their end.
Update due by 13:15 

Most lines swicthed very quickly to the alternative LNS automatically.

The status of lines on the control pages too some time to catch up on the RADIUS accounting, but this has now happened.

Graphs may have gaps which will be repaired on the over night status.

Although broadband lines look fine this link is still down, meaning we have lost resilience on links to BT.

We are still chasing.

BT are arranging an engineer to go to the Datacentre.

Still not heard anything more.
Chasing again. 

Came back around 00:09.
It appears that this was a problem with a card BT side.
Need to confirm though. 

Cleared fault on Card.

One of the three live LNSs just restarted.

We are investigating.

Lines have reconnected cleanly as per the contingency plans. We have to find the cause, but I'll close this major issue for now.

(Graphs would have been lost for the day, sorry)

Lines came back shortly after, we'll chasing BT for information and reasons.

This was caused by a BT PEW.

One of the three LNSs in live use just restarted and we are investigating.

Lines are moving automatiucally to the backup LNS within seconds.

This is the first test of the three LNS plus fallback confiuguration.

By all accounts customers reconnected very quickly - within seconds.

We have a clue what triggered the problem and we are investigating now.

Something else went wrong this morning and we are investigating.

Another crash - this time somewhere completely different and equally impossible.

It looks like lines reconnected quickly over both LNSs as you would expect from a crash. This is an excellent test but obviously this sort of thing simply should not happen.

OK we are moving people that connected to D after it crashed over to C. It is about a 1/3 of customers that are being moved. The controlled LNS switch is deliberately slow taking one line at a time over and ensuring no load on BT or our RADIUS that could cause default accepts or other delays or problems.

We are installing new code with additional debugging on D, and will move some lines back to D shortly.

This means we will lose graphs for this morning.

Looks like a few dozen lines got a PPP kill twice. We have updated the scripts now, and we are moving 1000 lines over from C back to D now. The plan is to leave this for this afternoon to confirm all is working and stable.

We are moving some more lines over now.

We are moving the last of the lines over now.

All lines are on the new code on the "D" LNS.

The problem is we have not got to the underlying cause yet.

We'll leave service like this for the rest of the day.

Having been stable for over 24 hours I am closing this major issue but we will continue to monitor carefully and try and find the underlying cause.

There seems to have been some major issue over night with one of the LNSs in the new rack. Something odd with RADIUS from around 1am, and then lines dropping and reconnecting around 04:52. Graphs before 04:52 on one of the LNSs are lost as well. It seems problems persist with some BE lines though.

We are investigating.

We are switching lines to the right LNS now.

Stuck BE tunnels have been cleared and BE lines are all reconnecting now.

Looks like the issue here resulted in BT lines coming straight back, but BE lines not doing so. We now have all lines back.

We've just had a blip on some BT ADSL/FTTC lines, most are back online already. We'll investigate the cause.

BT lines on just one of the LNS's blipped - lines on C.Gormless blipped, lines on D.Gormless did not. 

Lines are almost all back online now.

This was affecting BT lines. BE lines were not affected.

The network switch port connected to one of our BT fibres was offline between 12:02:36 and 12:04:23.

We're investigating further.

Half of our BT circuits dropped at 12:02, and starting coming back online at 12:05, most lines were online within a few minutes.

This was caused by a drop in one of our links to BT. A fault has been logged to BT regarding this. We will move customers off this link over the weekend so as to enable us to investigate this with BT in more detail.

It looks like around half od the BE lines went off around 02:43 and the rest at 02:58.

We were not aware of planned work this morning, though various work has been planned this week.

We have tried routing traffic via our old rack as well, and no joy.

Obviously customers using BE+BT lines as a fallback arrangement are fine, using BT.

Obviously customers using 3G backup are working.

If you only have a BE line you may want to consider some of the fallback arrangements we can offer. Though it is rare for it to be BE that is the carrier that has failed, it is sensible to consider contingencies.

That said, we would hope they have this fixed during the night.

There were unplanned outtages on 14 exchanges at approximately those times, we're still waiting to find out why this occured.

O2 have confirmed that changes made by them to the network overnight resulted in loss of service to large parts of the network, including all wholesale services.

Their engineers are currently rebuilding configurations on all aggregation nodes on their network to reverse the changes. We are beginning to see sessions establish again, about 20% have reconnected in the past few minutes.

O2 engineers have restored configuration on about 40% of core devices. We are seeing roughly that % of sessions back up.

It appears that O2 engineers have now finished restoring configurations, as we can see all ISAMs on our monitoring again and we are seeing the majority of customers back online.

This was due to BE engineering work that failed. An initial report of the problem is available at http://aa.net.uk/news-2012-02-10-be.html

There was a routing problem affecting ADSL lines.

We initially thought this was just our office (hence the previous post)

Routing is now restored and we're investigating the cause.

There was, quite separate to any other issues, a problem at 02:06 which resulted in the main LNS resetting. All customers quickly reconnected with no problems.

We have detailed logs of the actual problem, but it is baffling us slightly at present. We will be investigating this later in the day. This does, however, appear to be completely unrelated to everything else that has been happening.

We are rebalancing some lines between LNSs over night after this incident.

Investigating

Something major with our LNS - investigating

No clear on cause but looks like our fault not BT this time.

Graphs lost, sorry

Looks like people coming back on line now

Not good - lost graphs - and cause unclear...

We will be investigating this in the morning. More details on my blog http://revk.www.me.uk/2012/02/ooops.html

We're looking in to a network problem affecting broadband customers - More details to follow...

Lines are coming back and we arte trying to find what happened.

We have engineers onsite in telehouse looking in to this.

However, lines and routing are recovering now.

Looks like some network issue in our core rack in telehouse - engineers on site investigating now. Lines mostly back up now.

We are still somewhat at a loss as to what happened, but it has meant all lines being dropped of the live LNS. We have switched the the backup anyway.

However, and possibly unrelated, we are seeing issues with BT 20CN lines coming up - i.e. they are not. This was happening before this more major issue, and means 20CN lines are still down. We are chasing With BT.

20CN lines now on - looks like BT did a config change early which resulted in problems. This would have been a non issue if there was not also some sort of major blip which we have yet to identify.

Two problems it seems.

1. BT did a configuration change earlier than expected. No idea why, and normally this would have affected a couple of customers at most. We were looking in to this already when...

2. Somehow there was a loss of contact between our live LNS and core network. We can see this must have happened and we are checking the switch logs to see if we can find why. Engineers were on site in the new rack (not connected, do not the cause!) and are checking this out now.

The effect was all lines went off. We have changed the system to be less sensitive in such cases now and not drop all sessions due to a loss of connectivity to the core network.

However, as all lines went off, the BT config change meant none of the 20CN lines came back until we discovered that what they had done.

Almost all lines on-line, and all should now be able to connect as normal.

All of the switches are being replaced as part of the upgrade anyway, but we obviously want to understand the underlying issue here.

Update: This is looking increasingly like a switch issue - the switch is working now, and new switches are in the new rack ready to be deployed over the next week or two.

Seems a major problem affecting all broadband lines - service si restoring automatically, but taking several minutes to get people back on line.

Lines are coming back but much more slowly than we would expect

Both LNSs are seeing people connect constantly and then disconnect, which makes little sense. We are still investigating the issue now.

We have reset equipment and switched LNSs, which has affected lines that were not previously affected. We are trying to see a pattern here.

Logs confirm that at 2am. suddenly things started working a lot better and many more lines coming back. Still investigating what is going on here.

We think, whatever it was, has magically fixed itself. We are going to clear all sessions to one LNS now, so a blip again for some people.

We are not entirely sure of the cause of the original problem, but it is such a long time since we have had a major issue like this it appears that we currently have a problem with the speed with which our systems can recover.

It seems our RADIUS server is struggling and once overloaded it is too slow to handle the connections and so the connections timeout and re-try causing more load and more delay. The result is that while everyone is trying to connect, nobody can, and it basically took well over an hour for lines to connect.

More investigation to follow.

I am closing the mahor incident on this now - but we are investigating the cause and the slow reconnect.

A significant percentage of our broadband lines blipped briefly, but appear to be coming back.

We're investigating.

A number of lines have ended up on the wrong LNS - we're rebalancing sessions now.

This appears to have been an over sensitive test on one of our systems that thought we had lost connectivity and shutdown sessions. The test is being changed and code changes to adjust timeouts to avoid this in future.

BT ADSL lines are down, Ethernet circuits are down too.

The seems to be a BT outage in Stepney Green where we (and others) interconnect in to BT.

Lines are coming back

BT have raised an Incident and are aware of a problem

Lines are still logging in.

If you're not up yet, please do wait a bit longer.

There should be no need to reboot ADSL routers, as they should reconnect. 

Lines are still reconnecting now - it will take a bit more time before all lines are back on.

This does not seem to have affected Ethernet Lines, just BT ADSL. (BE ADSL lines were not affected)

If customers are getting BT 'Service information' webpage coming up then please reboot your router.

-This will be caused by BT accepting your login but not passing it on to us. Rebooting will cause the router to relogin, and should get through to us.

It is worth noting that BT have had issues with Stepney all week. We have been relatively lucky up until now not to be affected, but we have been working with a number of ISPs connected to Stepney during the week who have suffered similar issues. We don't have any exact details from BT as to the cause of the problems as yet, but clearly they are working on them.

This incident is now closed. 

We've had confirmation from BT that this was affecting other ISPs too.

BT clearred the fault at 08:54.

By 09:05 we had about 50% of ADSL lines back online.

It was also reported by ThinkBroadband
http://www.thinkbroadband.com/news/4829-major-broadband-outage.html 

Updates to this incident have moved to: http://status.aa.nu/apost.cgi?incident=1271

Lines coming back - more info onthe other status message posted above.

It looks like most/all BE lines are down. We're investigating...

BE lines are reconnecting now.

From the look of the logs, we lost all the L2TP tunnels to BE between 19:24 and 19:31.

We'll post again when we find out what the cuase was.

The outage has been confirmed to be cuased by a power outage.

BT have informed us that an:

Emergency PEW is planned to carry out urgent maintenance work at Stepney Green. From 00:01 AM -02:00AM there will be a possible short impact to service while traffic is being rerouted within BT core to wbmc shared service at Stepney Green.
From 02:00 AM to 05:00 AM The impact will be:
Total loss of service to wbmc shared end users who are connected to the Stepney Green 21cn Bras's during the Pew window of 02:00 to 04:00.
Total loss of service to IPSC/wbmc end users who are connected to the Ilford 20cn Bras's during the Pew window of 02:00 to 04:00.
The customers will experience will be that they will be unable to connect to their CP's will the planned work in being actioned.

We think this will be little impact for most customers, but may be a period off line for people on Stepney BRASs directly.

the planned works are now over.

We see usage have occasional spikes, and they often have a reason, but this evening we have seen silly high usage since around 18:40, not only on 21CN but also 20CN. Something is clearly "up" and there is some "internet event" happening. No idea what yet, but has caught us unawares.

The system does the usual to manange the traffic - give more to the "premium" paying customers, and to generally give small packets for VoIP, DNS, and interactive traffic priority.

But we'll try and work out why and if necessary get more capacity in shortly.

P.S. it is not just us cuaght out by this - logs show some lines seeing loss and latency which is within BT's network (as we ensure the LCP echos get sent anyway).

This is worse than the world cup traffic!

Only clue is new Apple IOS5 stuff - if that is the cause I am impressed.

Usage has just reached unprecidented levels - we have not seen anothing like this...

Customers are even reporting issues on Be lines, what is this?

We have core links hitting 1Gb/s - something we have been planning for in 6 months time - this is really unprecidented usage levels.

One good thing from this is that it has provided the FireBrick team with a good benchmark of what happens when you push both the LNS and BGP routers to the full gigabit throughput. The answer is that it copes really well and is using CPU levels that are actaully very good. But what on earth is causing this unprecidented levels of usage is not entirely clear.

So far we thing there is a windows update and apple IOS5 release, but is that all it is?

We are guessing this was IOS5 release.

Whilst Thursday night was also somewhat excessively busy it was nothing like the previous nigh, and caused some congestion mostly on 21CN.

All our BE ADSL lines lost connection at 21:30 on Sunday night due to a power failure in our supplier's network. The outage was brief and most lines had reconnected by 21:35.

We are investigating a problem with short periods (seconds) of routing problems which will be affecting BT and BE broadband lines. Customers may be seeing no connectivity for short periods at a time. 

This has been now resolved. 

As the planned works notice is rather short, we are posting this as a major issue now. We expect all BT lines to go out of service over night from around 2am to 3am. This will not affect BE lines.

20CN lines had a short outage (half hour) and 21CN lines longer (90 minutes)

We're currently investigating a routing problem, this will be affecting internet access for customers. More information to follow shortly.

Routing has been restored, we're currently investigating what the cause was.

The routing problem, which lasted a couple of minutes, looks to have been caused by a netowrk loop at one of our peering points. -This also affected other network providers.

Sorry for the delay posting this, and thanks for all the MSO texts.

Last night a firewall config change was made on the BE side of our link to them. I am not sure we had a planned works notice. Anyway, the effect was to drop all sessions to us, and at least one other ISP. They reversed the change just under 20 minutes later and lines reconnected.

We are in discussions now as to what they were doing and why?

We're currently looking in to an outage that looks to have affected BT ADSL lines. More info to follow...

Lines are logging back in again now... it may take a while for them all to come up - but looks like there have been a couple of blips that took lines down - more details to follow...

This is one of the two main links we have to BT which is currently just not working - lines are coming up on the other LNS slowly.

Stats show that most lines went semsibly to the B server quite quickly.

The A server link is up and down like a yoyo which is not good for automated fall back, and manual intervention has ensured all lines are preferring the B server for now.

Note, whilst unaffected by this issue, we have switched *all* lines to the B server which includes the BE lines. Sorry that we are not really in a position to sensibly switch only the broken BT lines over. This would have beeb a momentary PPP restart on BE lines and loss of graphs for now.

Note line status on the control page is confused and will update over night. Some lines may PPP restart once on an hour over night.

Odd, and annoying, the A server BT link just stablised by itself.

This means no BT fault report will get anywhere and we will not get any sensinle explanation. Arrrg!

We'll ask anyway.

This has been reported to the BT, and they will be looking in to what they saw from their end during the outage. We'll update in the morning.

Some major BT issues.

Looks like relatively short blips with people coming back on line now.

23:10:36 All 20CN birmingham BRASs!

23:15 most BT lines.

Looking at the traffic levels this looks to have been mostly 20CN lines impacted.

I am pleased to say that one of the first FireBrick FB2700 users making use of a 3G backup for his DSL line reports it worked seamlessly for him.

At 00:27:10 all traffic on both BT link stopped, and BGP timed out. This is after some sort of blip affectiong lots of lines at around 00:00. This resulted in all sessions dropping shortly after. The connection resumed at 00:29:08.

This looks almost exactly the same as a few nights ago, which turned out to be a planned work that had been sneaked in. This was the same second on both lines this time, so not exactly the same.

Lines reconnected over several minutes. BE lines were unaffected.

The rate at which lines are reconnecting seems unusually slow.

Looks like a lot of people got a "BT default accept" during the outage which means they spend a while on an unconnected link. That then has to time out before they connect again. This could explain the slow connections.

A few people ended up on the wrong LNS briefly as well.

Connections are stable now, but low on numbers. If you are off line, try restarting your connection.

I'll leave this MSO open until we have more details in the morning.

Lines came back in batches after the incident, and there were a number of BRAS specific issues during the night as well.

We'll post any more information we get from BT when we get it.

It appeas that both of our links to BT dropped, at the same time (02:06:51), including BGP with BT. All lines went off line. This lasted several minutes until 02:14:37 before lines stated coming back.

BE lines unaffected, of course.

lines all back. We'll chase up for an explantion in the morning. very odd.

I bet it is some sort of planned work they failed to bring to our attention.

I was just about to do the LNS switch over, but that will happen slightly later this morning now.

Logs confirm this was not a clean shutdown of BGP.

Hmm, timing different on each link. On the backup it was 02:06:44 to 02:14:21

It looks like this was caused by a BT PEW. We are chasing BT to confirm this though.

BT confirm this was due a PEW.

Looks like most BT lines when off, possibly not all.

Lines coming back now. Taking a few minutes in some cases.

Some lines will need to be moved to the right LNS.

This means lines will now be on the new timeouts from now.

Status suggest that this was slightly more than half the BT lines. This is specific to BT and not BE lines.

Looking stable, but took a couple of attempts to steer sessions to the right LNS which is another oddity, and perhaps another clue in the mix.

Oddly, and almost certainly unrelated in any way, at 23:32 many of our BE lines dropped.

This does highlight the benefit of having a BE + BT line as they rarely break at the same time!

We'll let you know if we get anything out of BT

In the mean time, most lines will be on new timeouts now.

Looks like pretty much all BT lines blipped. Investigating.

As expected lines came back quickly, but we need to find why BT blipped.

Reports of one Be line "wobbling" for a few seconds but not losing connection may be a side effect as Be Lines seem generally unaffected by this. Look BT specific.

Lines come back spread between the LNSs so some lines will have had a second brief blip while moved to the other LNS.

During some planned maintenance things went wrong and lines have cleared unexpectedly. They will reconnect over the next few seconds and minutes. Sorry for inconvenience.

A chunk of graphs will be lost, sorry.

A diagnostic command caused and issue and this is being investigated so we do not have it happen again!

Sorry about that - checking how that happened now so we can avoid it ever happening again. Thanks for your patience. Lines all back on and on the right LNS apart from the odd stragler.

Some ADSL lines dropped at 12:52 - it does look like a BT problem, but we've not yet discovered any patters in the lines affected.

We'll update this shortly.

Update: We are sorry but it does actually apeear that this was all of our BT lines.
We will look into why our monitoing gave an inccorect percentage to begin with. 

Most lines have logged back in again now. Lines are still logging in and we expect the remaining to log back in during the next few minutes.

Some lines (about half the number of the first drop) have just dropped again, we are seeing these reconnect though.

Some customers have logged into the wrong LNS, we are bouncing them back to the correct LNS now.

Title of this post has been changed to reflect the problem better

from: BT Blip affecting some Customers to: Blip Affecting BT Lines

All remaining lines were back online by 13:30

We're still looking on to what casued this, and will be reporting back.

Sorry for delay - we should have more details on this incident shortly.

Note that the outage duration may be much shorted than shown on the graph due to many lines switching to the backup LNS. The graphs for while on the backup LNS will not show (i.e. show as purple) even though on line.

Interestingly, having seen some BT issues before, this looks to be some sort of glitch in the LNS on the link to BT and not actually a BT fault. Whilst it is remotely possible this was caused by some external factor we think it is a random hardware glitch on this occasion.

The logs suggest this issue lasted 4 to 5 seconds at most but clearly it must have had a knock on effect that lasted the 10 seconds that are normally the timeout. This caused sessions to time out. The result was lines dropping and reconnecting to both main and backup LNS.

Action: From now on, all new sessions will have a 20 second timeout rather than 10 by default. This should help both the BT issues we have seen and anything like this happening.

Whilst lines started reconnecting immediately (within seconds) the time taken for some lines to connect we several minutes.

Action: We are working on multiple authentication servers not just for normal backup usage but for load sharing to ensure lines reconnect much faster in future. This should speed up recovery in the event of any issues like this.

While staff started to clear the sessions from the backup LNS in a controlled way an unexpected issue caused the backup LNS to fail. This affected a number of lines a second time and the reconnected to the main LNS.

Action: The exact cause is being investigated still and we hope to understand this better soon.

Further actions: We are very concerned at the apparent random issue on the BT link port, and we will be replacing the hardware some time on the next couple of weeks as part of routine maintenance.

Sorry for the inconvenience.

We saw most links to BT fail at 02:30 and have since received initial notifications from BT of a major issue. We are trying to get more details.

This looks like it may be related to the previous issues in Slough, and many lines remain disconnected.

Following the BT incident our main LNS stopped responding and the working lines switched to the secondary LNS automatically. This is being investigated separately but the BT issue remains.

Looks like more connections around 3am means most customers are now back on line.

BT have confirmed the incidents closed. We'll try and get more details of what happened as ever.

Following the BT outage at 02:30:59, the main LNS stopped responding at 02:33:34. This is likely to be a result of the problems with the link to BT. This should obviously not happen.

The system automatically switched lines to the secondary LNS as designed, which meant some customers took everal minutes to reconnect.

We are investigating the cause and may be sending an engineer.

Both LNSs up and running now.

All lines on 100% 20CN-REGION-READING and 100% 21CN-REGION-SL dropped at 2011-03-30 17:35:17
We have advised BT
This is likely to have affected multiple internet providers using BT

Two of our ethernet services are down as well, we can only assume routed via the same place. Investigating.

It is very rare for ethernet and broadband services to be affected by a common fault and suggests something major in the openreach back-haul network which serves both services.

Latest is "looks like a major fibre cut near Slough"

No more news yet

engineers on site and that the next update will be at 21:30, apparently

Fault still ongoing, and no confirmed reason for outage yet. Engineers are still on site and trying to find out exactly what the problem is.

The outage has been identifed by BT as an issue with a 21CN core router in Slough. Engineers are trying to restore connections. Earlier information appears to be incorrect - it is not a cable failure.

Further outages for a number of lines on Guildfor BRASs at 01:04

Latest is "There are technical and manageent bridge calls going on", "engneers are on-site at Slugh exchange", and "they had called out CISCO"...

Some broadband lines are coming back on now.

And off they go again... Well, something is clearly happening...

Not wishing to get your hopes up, but some lines tricking back on again...

And off they go... Arrrrg

Many lines back , and seem to be more stable now.

Broadband looks pretty good now. Still two Ethernet circuits down.

Looking in more detail, still quite a few of the BRASes are not back yet, but those that are seem stable.

More lines coming back gradually.

Ethernet back briefly at 6am and off again (this is the 2 affected lines, all others have been fine).

Whle broadband lines have ben gradually coming on, some are still down. It could be that these are lines that need a router reboot, but we have not had confirmation from BT yet as to whether they think the fault is cleared or not. We suspect not quite.

Some broadband lines ent off and back on at 6am, and just now (06:33). BT have in fact now advised that they think the broadband side was fixed at 03:50.

Affected Ethernet lines back on now (06:35)

At this stage, we suggest that if you broadband is still down, try power cycling the router first, and if no luck contact support at 9am.

Closing the issue now as no longer service affecting since this morning. We'll post more details of what actually happened when we get them.

Thanks for the texts from customers over night, and sorry not posted sooner. Our monitoring systems have picked up some serious BT issues for lines going via Milton Keynes, both 20CN and 21CN.

This has been ongoing during the night, but lines are mostly back up around 5am.

Obviously we are trying to get an explanation from BT.

Whilst most lines are back, it seems a few are "stuck" in BT somewhere. If you are not able to get on-line please let support know.

A lot of 21CN lines just came on-line

So, summary, around midnight all lines via the Milton Keynes RASs went down. At around 5am the 20CN lines came back and at around 07:30 the 21CN lines came back. It looks like most lines have reconnected now.

Chasing BT for explanation...

Being investigated

The problem appears to be very similar to the previous one - all routing on the link within BT was lost for just long enough to drop everyone off line, and they reconnected over the following minutes.

Note that lines on the wrong LNS are being cleared back to the right LNS with a short PPP restart now.

We need to get to the bottom of this with BT, obviously. We thought it may be related to some other work they were doing last week, but seems not now. We are also planning changes to the way the system reacts to issues like this to ensure they are much less disruptive.

Currently BT ADSL lines are down - looks like our link to BT, we are investigating.

Be lines are OK.

Our LNS has not crashed, we're suspecting a BT blip/outage of some kind, but we'll another update very shortly.

We are seeing lines atempting to connect to us - however most connections are failing. We suspect a BT RADIUS problem of some sort.

We are seeing some line connecting successfully.

It may take some time before all lines are logged in again. There is probably no need to reboot ADSL routers, as the rouer will keept attempting to connect.

About half of the lines are back online now. reconnections are happening all the time.

Some routers are getting a default accept from BT - meaning the routers log in to BT rather than through to us.

If you're still not online, or your router is getting a BT IP address, then please reboot the router and it should then re-connect through to us as normal. If that fails, turn the router off for 10 minutes and try again.

We are still investigating what caused this outage, we'll update this post with further details shortly.

Details about the cause is on this status post: http://status.aaisp.net.uk/apost.cgi?incident=891

The live LNS just restarted causing all lines to blip. Lines are coming back.

Lines are coming back on both LNS's as expected, and we will clear lines from B back to the A server shortly. Most lines are back now.

Actually, the bottleneck may be the authentication server - we'll look in to that when all is stable in case we can speed things up in future.

Yes, definitely some sort of bottleneck on authentication servers here.

Additional authentication server added and lines now reconnecting much faster.

Apologies for the slower than usual recovery from the issue.

We are investigating the cause of the problem now. This does not look related to the upgrade at the weekend and so we are continuing with the A server and the current version while we investigate the cause.

A lot of lines, seemingly on 20CN and 21CN BT lines, all blipped at around 22:00:21.

It is not clear why at this stage.

Note that separately the planned work LNS upgrade was not started until a little before 22:30.

Major issue with primary LNS - checking now

Ok LNS issue. Investigating. Lines connecting as fast as they can. Some a few seconds, but some a few minutes.

Graphs lost for today, sorry.

Cause identified. Unexpected packet from some PPP or other non ethernet conection. Fix already issued by FireBrick.

New code release to be deployed on next update.

Packer loss on  21CN-BRAS-RED8-L-NWS and 21CN-BRAS-RED9-L-NWS since lastnights PEW.  BT have reported it to BTO and waiting for an engineer to be task to be raised.

This is also affecting:
RED2, 3, 7 and 10 

BT Operate are still investigating this issue, possibly a faulty card (equipment in the exchange). This is mainly impacting Users in the North London, London, Essex areas.

 BT Operate tech support are still investigating.

BTO rerouted traffic to a different VLAN.