The 'per IP stats' usages pages are not working very well at the moment. Some customers are not seeing any usage, and some are seeing very little usage, and others are seeing correct amounts of usage!

This is being looked in to.

This does not affect the 'Line' usage or any billing based usage records. 

We are sorry for how long this has been ongoing.
We are still trying to resolve the issue, however it is taking a lot longer than we expected.

We do not currently have an ETA however we will update this status once we have further news.

02 have informed us that there is currently an outage at the Shepherds Bush (LWSHE), Kensington Gardens (WRKGDN) and Acton (LWACT) exchanges.

Engineers are currently investigating and 02 expect to have an update within the next 2 hours. 

This is now resolved, we have not been given any information as to the cause or fix for this outage.

This is now resolved, we have not been given any information as to the cause or fix for this outage.

Due to human error, we have managed to clear PPP on around a 1/3 of lines (B.gormless) today.

We are looking in to how procedures can be tightened up to avoid this in future.

This was a controlled restart of the LNS not a crash, and as such the PPP restarts were even quicker as devices did not have to wait for LCP timeout to restart.

And yes, we are seriously looking at the best way to avoid this sort of error happening again.

And yes, the human concerned was me - RevK - so I owe a few people a pint...

The answer is css - the control pages of the various machines currently all look identical apart from the host name. Using some css we can make all "live" boxes stand out very clearly.

Ater chasing BT on the cause for this outage, it would appear that this was caused by a Planned Works at London Stepney Green. 

The aim of these planned works was to increase capacity. 

Ater chasing BT on the cause for this outage, it would appear that this was caused by a Planned Works at London Stepney Green. 

The aim of these planned works was to increase capacity. 

We are trying to get one of our LNSs working on latest code, and this is proving a struggle as there are some major upgrades. We hope to finish this as per the planned work during this evening. This may mean loss of graphs and some PPP restarts for some customers.

This is being an issue, and we will be working on this over night we expect.

We think working over night on this may have paid off - we're monitoring now, but we hope we have cracked the issue at last.

c.gormless is now running at normal load (1/3 of customers) and showing no leaked RADIUS slots or stuck sessions or processes. This is good news.

It looks like the work last night as paid off. This evening we have the "D" LNS working on the newest code, and are letting lines trickle over to it when they reconnect for any reason. We plan to switch over more lines over night.

But so far it is looking like it is working as planned. We are checking the performance of the new code as well to make sure this was all worth it

Thank you all for your patience on this.

We have switched traffic to d.gormless over night as planned.

The LNSs are looking stable - we are still seeing a few sessions from the previous few days which have not had proper accounting (so free usage) which are being cleared up.

We are also seeing the odd BT issue from time to time, and it is nice to say that a blip is not, in fact, my fault - more "situation normal".

We're still looking in to why people are getting the the odd up/down emails and tweets as it is all related. Now that the LNSs are being stable we can investigate this somewhat more easily.

The scheduled LNS switch overs to move on to the new code base seem to be working as planned. Tonight's has gone through without any apparent issues. We will be rolling out the change over the next couple of nights to cover all LNSs.

Once again, thank you all for your patience and understanding.

There is packet loss on BE lines thoughtout the country, we have reported this BE and they are going to investigate.

We've had reports that BE Retail customers are also seeing a similar problem, so this seems like a general BE network/backhaul problem.

We have reported this to BE.

BE are aware of a possible core network problem and they are investigating.

BE have said that this is due to a technical fault on their network which they are investigating.

This was resolved just before 9pm - repairs to fibres were made within the BE network.

It looks like we have a blip in RADIUS around 5pm which has resulted in lines showing "pink" or "salmon" on the management pages and some people getting odd status update emails.

Services are working - this is an accounting issue.

We're not sure what happened exactly, but this is a scenario that the systems are designed to cope with relatively sensibly. Some usage may not be metered for the next few hours and some lines will be PPP restarted over night.

We are also doing an LNS switch over tonight, but there are issues with that which means it is likely to all happen later than usual, i.e. around 7am.

As it happens we are working on some updates to RADIUS which is hoped to be phased in over the next few weeks. Incidents like this, whilst minor, and not service affecting, are a nuisance, and they are being incorporated in to the new design to make such issues less likely.

We're clearing the pink lines now, ppp restart.

Some customers on C or D gormless will have gaps in their graphs this afternoon and this evening. The lines have been connected through these gaps but have has some PPP restarts.

Some customers will have had PPP restarts around 08:30 and possibly moving from the D to the C LNS.

Sorry for any inconvenience.

We seem to have an issue with RADIUS accounting this morning. We have restarted things and we are investigating the cause.

There have been some odd tweets and emails about the issue, but it lists the previous "AdminReset" from a previous LNS switch over from earlier in the week. This seems to be an odd side effect of the problem which we also need to investigate.

P.S. As usual, this also means some free usage, and will mean some PPP restarts later today and over night.

It also means the lines show "salmon pink" on the control pages. But the lines are working, of course.

We are manually clearing some of the remaining sessions for which there is no accounting. This means a brief PPP restart for some customers.

There is definitely something up with RADIUS which we are stil chasing.

Two thirds of lines are all sorted and accounting cleanly now.

One third is still being working on.

We think it is all sorted for now

We have noticed that for the last few days there have been increased latency on quite a few of our BE lines.
Initially the latency was occurring between 18:00 - 00:00, however today we have seen the latency start at 09:00.  

BE have been informed and this has been passed to O2 to investigate possible network issues.

 

Looks like the increased latency suddenly stopped around 1:30.
Neither BE or O2 could see a cause for this.

We will montior over the weekend and early next week to make sure the issue is actaully resolved. 

The increased latency has not returned as of yet, however we are still chasing BE as to an explaination as to what the issue was.

We finally have an update as to what the cause of this issue was:

"I can confirm that the underlying cause had been narrowed down to a network card in our provider's core network. The card was found to be just looping and generating high amounts of traffic. As a countermeasure, an engineer physically rebooted the device and thereby resolved the issue. The card itself has remained stable since and no further concerns have been registered."

We finally have an update as to what the cause of this issue was:

"I can confirm that the underlying cause had been narrowed down to a network card in our provider's core network. The card was found to be just looping and generating high amounts of traffic. As a countermeasure, an engineer physically rebooted the device and thereby resolved the issue. The card itself has remained stable since and no further concerns have been registered."

Card Reboot 

This appears to have been caused by a BT PEW.

This appears to have been caused by a BT PEW.

One of our LNSs had an issue today, lines blipped and reconnected automatically. Graphs are lost from before the incident

This is being investigated.

This only affected one of the three live LNSs.

This happened again, and we are investigating the cause. Thankfully the fallback systems are very quick and efficient these days, but this should not happen!

Packet loss is back on lines on the MANOR PARK exchange.  We have reported this to BT and BTO are going to investigate.

BT's IP department are currently investigating and monitoring this issue.

BT's IP department are currently investigating and monitoring this issue.

Lines: 100% 21CN-REGION-21CN-BRAS-RED10-L-NWS dropped at 2013-03-27 02:24:36
We have advised BT
This is likely to have affected multiple internet providers using BT

An issue was reported this evening with packet loss at LINX. We have suspended LINX peering for the time being until this was resolved.

We'd like to thank customers for alerting us to this with correct use of our MSO text system.

Not quite an April fool's joke, sadly, but some Home::1 users have seen their lines restricted this morning. This appears to be due to a delay between bill for this month being issued and DD being scheduled. This is silly! It has been fixed for next month.

Quite separately, something is not right with RADIUS accounting which means a lot of users are not seeing any metered usage, i.e. download is free for many customers this morning, including Home::1 users. This too i sbeing fixed now.

All affected lines have had the restrictionlifted.

One of our LNSs had an issue today, lines blipped and reconnected automatically. Graphs are lost from before the incident, and usage for the hour up to the incident will not have been metered/billed.

We have some clear ideas what caused this and it is being investigated.

This only affected one of the three live LNSs.

Note we cleared some of the affected lines back to the original LNS at 9pm causing a PPP reconnect.

For the last few nights there appears to have been packet loss on all 21CN BRADFORD-ON-AVON line.
The packet loss seems to last between 17:00 - 00:00.

We are currently reporting this to BT. 

BT have raised a works request.

BT operate are still investigating, we should have an update tomorrow morning.

This looks to be all fixed.

There is  packet loss on lines on the MANOR PARK exchange. We have reported this to BT and BTO are going to investigate.

Service was fully restored this morning after loop testing BT performed cleared alarms.

Since 02:06 today we have all lines that are connecting though the BT BRAS '21CN-BRAS-RED10-SL' and '21CN-BRAS-RED9-SL' are down.

This is due to BT planned work on the Slough BRAS.

This will be affecting other ISPs too, and is affecting about 0.6% of our customers. 

Lines came back online at around 3am.

Following loss of one external link at 4:31 this morning we can see that there are some areas which are not quite falling back as planned. This means some of the wholesale customer links have needed some manual reconfiguration. We believe this is all resolved now, but as it is running on only one link the wholesale interconnects are at-risk until the link is fixed. We are chasing this with the suppliers.

This should have no impact on our broadband customers generally, though you may find you switch LNS if you reconnect at all today as part of the work around for this has been a change of active LNSs. The impact of this is loss of graphs during the day, but should be correct historically from tomorrow.

In the longer term we will be contacting wholesale customers to review the fallback arrangements in their network and ours to ensure such a link failure in future would fall-back seamlessly.

There is packet loss on FTTP lines on the BRADWELL ABBEY exchange, we have reported this to BT.

The packet loss appears to have disappeared at around 22:00 last night. We're awaiting further updates from BT later on today.

This is still a problem, and we're seeing low levels of packet loss which will affect speed.

The latest from BT is that they have planned work on 6th Feb to increase capacity.

We will try to get more details and a clarification as to wether this upgrade will fix the problems that our monitoring and customers are seeing. 

BTs  planned capacity increase on 6th Feb appears to have resolved the issue.

Similar to the problem at the weekend, one of our ports to Datahop is flapping - we're disabling the port and moving the traffic over to our other port.

More details to follow

Datacentre staff are investigating the faulty cable, but service was restored in about 15 minutes of the fault happening.

Service restored, the cable has sinces be connected to a different port at the far end.

The main www.aa.net.uk web page normally has a graphic on the home page for Home::1 and quota limited SIM users.

This is not working at present for most customers as a result of a link failure.

Once we have sorted the link failure we are going to be making a few changes so this does not happen again.

It also means the top-up screen for users on Home::1 is missing, please text support if you need topup over the weekend.

Resolved by a reconfiguration for now

Nagios has gone mental telling me about a port blip at 04:31 in London, and I can see that a handfull of customers had an issue at the same time. I'm investigating what is going on, but overall things look fine, traffic is flowing, and not a peep out of customers on irc.

This port started flapping again on 30th Jan 13:10 more info here: http://status.aa.nu/apost.cgi?incident=1726

The issue relates to a specific link, and the lines that blipped werte mobile data SIMs. The link has been shut down (thanks Paul). We're investigating this further, but panic over for now.

The "B" LNS restarted unexpectedly - lines reconnected to other LNSs as expected, but graphs are lost.

Lines on the Hampton exchange have been down since around 18:04 this is linked to the Southampton 7750 router/Multi-Service Interconnect Link failing, an enigneer is currently working to replace the on-board Flash card.

Service was restored at 11:15:55

Some lines on the GISBURN exchange dropped at 15:56:52

A card is failing on MSIP equipment at Preston. A remote reset has been attempted but did not restore service. A BT  engineer is being tasked to perform a reseat or change hardware.

BT Incident Reference:2357. Service was restored at 18:21 following the change of a failed card at Preston.

One of Be's LACs appears to have restarted affecting a hadful of customers. Lines reconnected.

The "D" LNS restarted affecting a 1/3 of customers - it is the same cause code as before and we are looking to see if the extra diagnostics have provided more clues and how we can add even more to track this next time. This is very infreqent and we have failed to reproduce this in the lab, so it is taking time to track it down and fix it. Lines reconnected immediately as they are supposed to and graphs are lost before the restart.

Again we have run in to a snag with Direct Debits and Home::1 which has meant there is a race condition where several Home::1 accounts show as not being paid before the Direct Debit notices are sent out. We believe we have found the underlying cause of this at last, and we are getting lines back on line now. Sorry for the inconvenience.

The start of month bill run took longer than expected which created the race condition, we have adjusted the logic to try and avoid this issue in future.

There was some sort of blip that caused problems with 3 of our 4 external routers. It is not clear what exactly at the stage. They all recovered automatically.

Some Be lines just blipped, they have come back though. More info to follow.

This looks like a(n) LAC within the Be network, we're contacting Be for more details, howevr lines came back online within a couple of minutes.

Another blip happened at 15:02, we are chasing Be.

Be are aware of the fault and suspect a router on the Be network - they are investigaating the cause.

The fault has passed now and we've asked Be for an update as to what they have done.

This was caused be a router restart within Be.

We've just had an LNS reboot which has meant about a third of our lines dropped. 

They will reconnect in the next minute or so.

This is similar to the problem yesterday which affected a different LNS.

We are investigating.

Lines came back online within a minute or two, we are investigating the cause of these blips.

The Redruth Exchange is experiencing packet loss in the evenings from around 7pm to 11pm. It has been reported to BT

BT say they have resolved this problem and that their links last night were showing good levels of traffic without problems.

We'll continue to monitor, but hopfully the latency problem has now been fixed.

BT say they have resolved this problem and that their links last night were showing good levels of traffic without problems.

We'll continue to monitor, but hopfully the latency problem has now been fixed.

Sorry for lack of post earlier, it should have been automatic.

BT clearly have some major issues with their Stepney node which has meant lines have been up and down all night.

BT say this is fixed, and lines all look stable

One of our links to BT 'blipped' causing around a third of our customers to disconnect. Lines are reconnecting now though.

Most lines are back online now.

The issue was indeed the link to BT, and we are awaiting an explanation from BT as to what happened. It is possible they will not know either.

One of our LNS's just rebooted causing DSL lines connected to it to blip.

We are looking in to the cause, but line will reconnect within a minute or two.

Lines did reconnect immediately - if nothing else this is a good test of the fall-back and redundant systems. Only a third of customers were effected and the main problem is loss of the monitoring graphs. We do apologise for any inconvenience.

This is indeed the same issue we saw last time, and is still being investigated.

As part of the preparation for tomorrow's planned work, we are running on two LNSs today rather than three. This allows the switch change to go ahead tomorrow.

Normally an LNS change some means people will see their graph start in the early hours, but graph history is correct and shows the graphs join up the next day.

Today is slightly special and some people will change LNS if they re-connect at all during the day. The result is a graph that starts from that first reconnect. Again, over night, the historical graphs will glue it all together. It just means some people with line issues will not see the start of day today on their graphs today.

Anyone with multiple lines bonded that reconnects may change LNS too, in which case the other lines will be moved within a couple of minutes to ensure bonding continues normally.

We are switching back to three LNSs tonight - but the issue with graphs changing on a reconnect will still apply until we have cleared lines to new LNSs over the next few nights.

One of our LNSs restarted unexpectedly (c.gormless) and we are looking in to it now. Lines reconnecting automatically as expected.

The diagnostics are not that clear and this will take a bit longer to get to the bottom of this. The immediate issue is over with an automatic restart, and all four LNSs are in operation.

We are adding more diagnostics and will be including this in LNS updates over coming weeks. This should help track what happened if this ever happens again.

There appear to have been a couple of blips with BT 20CN lines in Manchester. No word from BT.

We have identified a reason some upgrades have causes a few seconds down time when not expetced to cause any. We have just done upgrades to rectify this before tomorrows switch change over.

Some customers may have seen a few seconds of issues but hopefully we have managed to sort this once and for all now.

Sorry for any inconvenience.

Since 7pm there has been notably high latency on many BT 21CN lines on Guildford BRASs. We are investigating with BT.

Our analysis shows this to be a fualty LAG in BT core network somewhere near the Guildford BRASs. We are chasing this with BT.

We have managed to manually juggle IP addresses of LNS L2TP enpoints to work around this on most affected lines now.

Looks to be fixed at 21:00

Here is some more information about this particualr issue (originally posted to Facebook)

I do think we should write up more about some of this stuff. At 7pm last night we had over 100 customer's Internet start being unusable with massive latency. See graph below, shows it.

Within 20 minutes we hadidentified which lines were affected, and were on to BT (as they were all BT lines). Customers were updates by status pages and on irc.

We managed to work out that there was a pattern in the affected lines.

Not only were they all BT lines, on a specific area metro node (Guildford), and all 21CN lines, there was a pattern where sets of lines on combinations of LNS at our end and and BRASs at BTs end had issues.

We tested changing a line to a different IP on the same LNS our end, and as we expected, the problem went away. We were also able to confirm with affected customers on irc that they saw the fix work as well.

We were able to tell BT that the problem must be in a link aggregation group (LAG) somewhere in the back-haul from Guildford. Basically, this means multiple links are used to handle the total capacity. The equipment picks the link based on a hash from things like IP addresses, so the IP address of the BRAS and LNS together decide the link. One of the links was ill and so affected a set of lines.

We then proceeded to manually change affected lines to a different IP for the LNS (not a change to the customer IP, this is internal). This caused their service to go back to normal.

BT did fix the issue (still waiting for a detailed report from them) at 9pm.

From experience, this is the sort of issue BT have no way to detect themselves. Maybe they have better monitoring now, but when we reported it they did not know about it, and other ISPs had not spotted it.

It is worth bearing in mind, this fault will have affected every ISP using BT 21CN lines, not just us. The fact we got BT on to it so quickly helped all of the affected lines on all affected ISPs. You're welcome!

It is only because we have this detailed monitoring and graphing of every line that we can identify and diagnose issues like this at all, let alone so quickly.

Temporary loss of service followed by packet loss on the Shoreditch Exchange. BT currently investigating the issue.

IPMB card switchover carried out under ALS 500428 at 12:53, BT fixed fault.

IPMB card switchover carried out under ALS 500428 at 12:53

At roughly 3pm a few guildford RASs went down.

We are rpeorting this to BT at the moment. 

The RASs that are affected are:

21CN-BRAS-RED9-GI-B

21CN-BRAS-RED10-GI-B

21CN-BRAS-RED12-GI-B

BT have raised an incident.

Trying to get more information. 

BT say diagnostics are still ongoing update should be around 16:00

Most lines are back up now.

fault fixed, Missing vlan configuration was reapplied.

Missing vlan configuration was reapplied

We carried out some major "behind the scenes" updates to the main control systems over the weekend, and whilst most of these have gone seamlessly, a few have not quite gone to plan.

As such, some of the controls and buttons and text in the information packs are not quite right.

Feel free to point out any issues on irc and we'll catch them all.

Sorry for any inconvenience.

Home::1 users were offline for about 5 minutes from 11:00 this morning.

It's now fixed.

We've given all Home::1 users a 1 gigabyte extra quota this month by way of apology.

At around 8:30 all 21CN lines at the SOUTHAMPTON exchange went down.

We are reporting this to BT. 

BT have raised a task to investigate this.

Please see http://status.aa.net.uk/apost.cgi?incident=1655

Fixed

http://status.aa.net.uk/apost.cgi?incident=1655

All 21CN lines at WANDSWORTH exchange and the 21CN-BRAS-RED3-SL RAS went down at roughly 9:30.

We are reporting this to BT. 

BT are investigating

Please see http://status.aa.net.uk/apost.cgi?incident=1655

Fixed

http://status.aa.net.uk/apost.cgi?incident=1655

At around 09:19:57 21CN line on UPPER HOLLOWAY on the  21CN-BRAS-RED12-L-WAT lost PPP. We are reporting this to BT. 

Please see http://status.aa.net.uk/apost.cgi?incident=1655

Fixed

http://status.aa.net.uk/apost.cgi?incident=1655