Last 10 major outages posts, sorted by last updated
Order posts by limited to posts

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Feb 27, 12:00 PM (1½ months ago)
CLOSED
Feb 27, 12:31 PM (1½ months ago)
DESCRIPTION
A number of lines dropped and reconnected at around noon. We're investigating.
Resolution:

The X.Witless LNS hung and restarted which caused customers to disconnect and reconnect.

This incident is related to https://aastatus.net/42608 X.Witless had been running without incident for 104 days. However, it is not fitted with an NVMe drive and was running software that pre-dates our NVMe drive fixes. We suspect the hang was caused by these two factors.

Further work on our LNSs is being planned and updates will be posted to the status page in due course.


Closed: 1½ months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 20, 10:20 AM (1¼ years ago)
CLOSED
Oct 20, 10:45 AM (1¼ years ago)
DESCRIPTION
Lines connected on the witless LNS dropped and are in process of reconnecting.
Resolution: The Witless LNSs suffered a crash which was caused by a bug in our CQM graphing code. This bug has been fixed and is in the process of being tested and we will then roll out the new software to the the Witless LNSs overnight as 'emergency' planned work.

Closed: 1¼ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Jul 20, 01:00 AM (1½ years ago)
CLOSED
Jul 20, 07:29 AM (1½ years ago)
DESCRIPTION
We're investigating a problem causing some lines to be reconnecting.
Resolution: A bug has been found in the LNS code which erroneously limited the number of connections that would successfully connect. This has been fixed. We will update the software on our test LNS (B.Doubtless) overnight. We'll then reschedule an upgrade of the production LNS (X.Witless) and bulk-move of customers for later in the week.

Closed: 1½ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
May 19, 12:51 PM (2¾ years ago)
CLOSED
May 19, 01:21 PM (2¾ years ago)
DESCRIPTION
We're seeing a high number of broadband lines dropping and reconnecting, we're investigating
Resolution: Line drops were caused when staff bought up an interconnect between our two London datacentres. The configuration of this circuit was intended to be separate from the rest of our network but somehow it did carry some traffic and cause this outage for some customers. We continue to investigate the cause in more detail.

Closed: 2¾ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 29, 09:00 AM (3¼ years ago)
CLOSED
Oct 29, 10:00 AM (3¼ years ago)
DESCRIPTION
We had overnight problems with a faulty switch in London. details here: https://aastatus.net/37744 Bringing the replacement switch back in to service is causing some disruption to services though.
Resolution:

Closed: 3¼ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (3½ years ago)
CLOSED
Sep 15, 04:00 PM (3½ years ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


Closed: 3½ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (3½ years ago)
CLOSED
Sep 06, 08:00 PM (3½ years ago)
DESCRIPTION
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Resolution: We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

Closed: 3½ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 12, 04:30 AM (3½ years ago)
CLOSED
Aug 12, 05:15 PM (3½ years ago)
DESCRIPTION
We have a number of DSL lines down in and around Scotland. BT say: "Due to the severe weather in Scotland early this morning there was a major outage at Edinburgh Exchange causing loss of 20CN & 21CN / ETHERNET, MEAS & BROADBAND SERVICES. Engineers are currently on site recovering services. We're sorry for any inconvenience that this has caused."
Resolution:

Closed: 3½ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Jan 22, 11:10 PM (4 years ago)
CLOSED
Jan 23, 09:05 AM (4 years ago)
DESCRIPTION
We're investigating a problem with some broadband lines showing 50% packetloss from just after 11PM.
Resolution: The problem was resolved just after 9AM. the reason for this outage is in https://aastatus.net/33308 "Networks have investigated and advised they found a number of errors on a card in LOH which was erroring and causing back plain errors."

Closed: 4 years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Nov 19, 12:01 AM (4¼ years ago)
CLOSED
Nov 19, 01:00 AM (4¼ years ago)
DESCRIPTION
We are seeing line drops on some TalkTalk lines since midnight. We're investigating...
Resolution: This was a fault within TalkTalk. The 'routing engine' in one of their routers failed.

Closed: 4¼ years ago
View full details