Order posts by limited to posts

Yesterday 15:06:09
Details
11 Oct 2013 17:12:54
We've fixed a problem with line up/down SMS/email/tweet notifications. This may result in customers receiving old notifications, so please disregard any such historical notifications that you receive.
Update
12 Oct 2013 13:24:43
It looks like the notification system is still a bit broken, with some customers receiving many duplicate notices. We've disabled notifications entirely until we've fixed this. Sorry for any inconvenience.
Update
14 Oct 2013 10:52:12
The notification problem should be fixed now, but please do let support know if you receive any erroneous updates.
Started 11 Oct 2013 17:09:34 by AAISP Staff

31 Mar 17:00:00
Details
31 Mar 15:10:58
There are some router upgrades during this week. We'll roll out changes gradually and disruption should be minimal (few seconds maybe but ideally no outage at all). We'll roll out LNS upgrades as well over night.
Started 31 Mar 17:00:00
Previously expected 7 Apr

9 Apr 23:00:00
Details
9 Apr 20:02:43

Question: Does the Heartbleed bug affect any AAISP servers?

The answer is that no servers are affected that hold customer data or our aa.net.uk SSL certificate secret key. The control and billing pages, email servers and our ticketing system are all running an unaffected version of openssl.

This doesn't mean that we're running out of date software; we still apply backported security patches to those boxes and plan suitable upgrades in the long term.

Unfortunately, however, we had a single test box that was both affected by the bug and held the CAcert signed certificate that we use for our email services. We are therefore going to revoke that certificate and replace the secret key.

The chances of the key having been leaked are tiny, but we think it is worth this measure as a precaution.

Customers who do not have the CAcert root cert installed may see warnings when they connect to our email services. There is more information here: http://aa.net.uk/cacert.html

Please contact support if you have any questions.

Started 9 Apr 19:40:20 by AAISP Staff
Closed 9 Apr 23:00:00

13 Feb 16:00:00
Details
13 Feb 15:14:54
We are investigating a potential problem with our Maidenhead routers. This should be considered an "at risk" period. Fibre links based in Maidenhead will already have blipped briefly at about 14:55, but we will try to ensure that they do not lose connectivity again.
Started 13 Feb 14:55:00
Closed 13 Feb 16:00:00
Previously expected 13 Feb 16:00:00

3 Apr 15:04:47
Details
1 Apr 08:59:54
As you may be aware the office suffered a break-in in mid March, and as a result we've installed new computers. In a change of direction for A&A we have opted for Windows 8, and are pleased to report that going forward we will only be able to support Windows based devices. Linux and Apple support will be dropped from today. We are also replacing all of our routers with Cisco IOS 14.04.01 based devices so we can deploy Carrier Grade NAT.
Update
1 Apr 13:00:00
Happy April 1st! smiley
Started 1 Apr

14 Mar 13:45:03
Details
14 Mar 08:25:52
Our offices suffered a break-in last night. Because of this, we may be slow to answer phones, etc, this morning. If it is taking a long time to get through by phone, please email instead.
Update
14 Mar 13:45:42
We're back in the office and mostly back to normal! Thank you for your patience this morning.
Closed 14 Mar 13:45:03

22 Feb 13:00:00
Details
22 Feb 08:03:44
A disk server has failed, it is impacting all web sites we host and email. Engineers are working on this now.
Update
22 Feb 10:08:38
There is a major issue with one of the disk servers, and we are planning to switch to a backup, but that is likely to involve an engineer visit to the data centre.
Update
22 Feb 10:16:44
Engineer is on his way to the data centre now.
Update
22 Feb 11:15:40
This is looking more complex than expected - we have switched the secondary controller, but there are issues with one of the disk arrays as well. Engineer still on site.
Update
22 Feb 11:17:48
Disk array is rebuilding now. We should have email working shortly and then web pages once the disk array rebuilds.
Update
22 Feb 11:57:48
Web space up, and mail servers being reconnected to disk array now.
Update
22 Feb 12:11:56
Issues with web pages again, investigating.
Update
22 Feb 12:14:17
The secondary disk server is now showing problems too. We are working on it.
Update
22 Feb 12:32:03
This is proving to be quite a serious issue - we appear to have issues with two separate disk controllers and with some of the RAID disks and with the file system on one of the disks. This is a very odd multiple failure, especially given that all of this is monitored constantly and was not showing any issues yesterday. We do have daily backups, so if all else fails there are ways to get service restored with backups and some loss of recent emails or changes. At this stage we are working to repair the failed file systems before considering that move.
Update
22 Feb 12:35:22
It looks like we have the mail file store repaired and mail should be back on line shortly.
Update
22 Feb 12:38:27
Web pages back.
Update
22 Feb 12:40:20
Incoming email should now be working again.
Update
22 Feb 12:44:44
We are checking all mail and web servers now to confirm all is well again.
Resolution Obviously this sort of multiple failure is somewhat unexpected. We do have plans for new disk servers anyway, and this type of failure will be considered as part of that system design.
Started 22 Feb 00:39:00
Closed 22 Feb 13:00:00
Previously expected 22 Feb 13:00:00

25 Sep 2013 16:20:00
Details
25 Sep 2013 16:09:44
We're currently investigating a problem with our disk storage server that runs our email and web space. We are investigating.
Update
25 Sep 2013 16:18:26
The server is being restarted at the moment.
Update
25 Sep 2013 16:20:41
Ser server is back on line, we are doing post-boot up checks etc to being the services back online.
Update
25 Sep 2013 16:20:58
Web services are now back online.
Started 25 Sep 2013 16:00:00
Closed 25 Sep 2013 16:20:00

13 Jan 11:39:45
Details
13 Jan 11:39:45
We've updated SSL certificates for our customer facing servers this morning (email, webmail). Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html
Started 13 Jan 11:38:46

13 Jan 09:00:00
Details
9 Jan 14:41:09
We'll be doing some router upgrades over the weekend - this should have little or no impact.
Started 11 Jan
Closed 13 Jan 09:00:00
Previously expected 12 Jan 23:59:59

03 Dec 2013
Details
02 Dec 2013 16:46:51
We are suffering some unexplained blips with some ports flapping occasionally during the day - and this has been most noticable in effects on Ethernet customers and VoIP services. We believe it is not affecting normal broadband servcies. We have been trying to track this down all day and continue to work on this.
Update
03 Dec 2013 09:27:41
The main issues stopped around 7pm, but we are still looking in to this.
Started 02 Dec 2013 10:00:00
Closed 03 Dec 2013
Previously expected 03 Dec 2013

24 Dec 2013 11:59:17
Details
24 Dec 2013 11:59:17
Just in case any one is concerned, we have renewed the ssl certificate for *.aa.net.uk today. This is used on various systems.

17 Dec 2013 14:54:33
Details
17 Dec 2013 11:14:24

FUD (Fear Uncertainty and Doubt) seems to abound at the moment with a report claiming that there is a secret back-door in BT FTTC modems allowing US DoD / NSA / GCHQ access.

The BT FTTC modems do have a management VLAN for TR069 allowing them to be upgraded, and allowing BT to do things like line tests. This is pretty standard practice and no surprise to anyone.

The hype appears to stem from the fact that BT have used an un-routed block of IP addresses for that management LAN, a block that is assigned to US Department of Defence, 30.x.x.x. This is not uncommon for private management LANs in large networks and does not mean there is any link to the US DoD. It is just a number they picked, and being a block that does not clash with anything in their network or any normal private addresses makes this a reasonably sensible choice.

The paper in question presents no evidence that BT modems have secret spy back doors. Yes, BT could change the firmware in the modem, and could add all sorts if they wanted, just as they could add something to the DSLAM or BRAS or any part of their network. If they did, then they would most likely be found out and a real paper on the issue with actual evidence would be published.

We see no evidence of any deliberate tampering with any packets across the BTW back-haul network, either in the modem or elsewhere.

Happy for comments - this was originally posted on our Facebook page.

Update
17 Dec 2013 14:54:21
Some more links:
  • http://www.ispreview.co.uk/index.php/2013/12/confusion-alleged-gchq-nsa-backdoor-bt-fttc-modems.html
  • http://revk.www.me.uk/2013/12/paraniod-ravings.html

  • 10 Dec 2013 11:10:02
    Details
    10 Dec 2013 10:19:02

    Here are the opening times for the Christmas holiday period:

    • Tuesday 24th December - Open with fewer staff
    • Wednesday 25th December - Closed ***
    • Thursday 26th December - Closed ***
    • Friday 27th December - Skeleton Staff, use email/IRC/SMS to contact us *
    • Saturday 28th December - Closed **
    • Sunday 29th December - Closed **
    • Monday 30th December - Skeleton Staff, use email/IRC/SMS to contact us *
    • Tuesday 31st December - Skeleton Staff, use email/IRC/SMS to contact us *
    • Wednesday 1st January - Closed ***
    • Thursday 2nd January - Open as usual (***)

    Special Holiday Rates

    We are pleased to confirm that customers on our standard units based tariffs will again benefit from a Christmas special rate between Christmas and New Year:

    * The three normal working days between Christmas and New Year will be a Christmas special rate which consumes no units regardless of the amount of usage. That is 27th Dec, 30th Dec, and 31st Dec.
    ** The 28th Dec and 29th Dec being normal Weekend rates.
    *** The 25th Dec, 26th Dec, 1st Jan (and for lines in Scotland, 2nd Jan) are also at Holiday rate.
    The special 'Holiday' rate is the same as weekend rate.


    Merry Christmas and a Happy New Year from all at A&A
    Started 10 Dec 2013 10:00:00

    06 Dec 2013 07:26:39
    Details
    04 Dec 2013 08:49:13
    We will be shutting down 'Clueless' on Friday morning for some maintenance. It will be offline for about 15 minutes.
    During this time access to the control pages won't be possible.
    Update
    06 Dec 2013 07:09:59
    This work will be starting shortly.
    Update
    06 Dec 2013 07:18:34
    Clueless is being shut down now, a few minutes earlier than planned. We expect it to be back online in about 15 minutes or so.
    Update
    06 Dec 2013 07:26:26
    Works completed.
    Resolution Works completed. RAM upgraded and a new Disk for the CQM Graphs archive added.
    Started 06 Dec 2013 07:30:00
    Closed 06 Dec 2013 07:26:39

    01 Dec 2013 09:20:39
    Details
    01 Dec 2013 08:10:32
    Our main database/control server known as clueless is suffering some issues this morning. This could start to have some knock on effects on other services and delay some billing this morning. We hope to have this resolved soon.
    Update
    01 Dec 2013 09:20:55
    Clueless is now back to normal.
    Started 01 Dec 2013 05:00:00
    Closed 01 Dec 2013 09:20:39
    Previously expected 01 Dec 2013 09:00:00

    01 Nov 2013 08:43:14
    Details
    05 Jul 2013 15:42:28

    We have added a feature to our Control Pages which will list IPs that we have found to reply to DNS queries. This is to aid customers in securing any open DNS servers that they may have.

    We will rescan IPs on a weekly basis, and we will be contacting customers by email when we find vulnerable DNS servers shortly.

    There is more information about Open DNS resolvers on our wiki:
      http://wiki.aa.org.uk/Category:Open_DNS_Resolvers

    Our control pages can be accessed at:
      http://aa.net.uk/login-clueless.html

    We are happy for customers to run their own recursive or authoritative DNS resolvers, this tool should help where there are DNS servers that are potentially miss-configured and vulnerable to amplification attacks.

    Started 05 Jul 2013 15:39:00

    24 Sep 2013 16:04:14
    Details
    24 Sep 2013 16:04:14
    Customer who have a PSTN line provided by us are now able to run an Openreach 'Copper Integrated Demand Test' on their line. This can help diagnose a PSTN problem before speaking to AAISP Support. More information on the wiki at: http://wiki.aa.org.uk/Line_Diagnostics#Run_PSTN_Test
    Started 24 Sep 2013 16:00:00

    08 Sep 2013 20:20:24
    Details
    06 Sep 2013 11:49:48
    We will be doing some router upgrades over the weekend. This is a prelude to some minor internal network reconfiguration. This plan is to work through these changes very carefully and fall back if any issues. Generally, a router upgrade is likely to have almost no impact on traffic, but been seen to cause a blip for a few seconds some times.
    Update
    08 Sep 2013 20:22:54
    Router upgrades complete, and new transit feed on-line, including more IPv6 transit. We will be rolling out upgrades to LNSs over night during the week. There was minimal disruption to most customers during this work but it was not quite 100% seamless. Well done to the Paul and Cliff and other staff working on this over the weekend.
    Started 06 Sep 2013 17:00:00
    Closed 08 Sep 2013 20:20:24
    Previously expected 09 Sep 2013

    05 Sep 2013 19:04:46
    Details
    05 Sep 2013 18:07:46
    We are seeing some internal routing issues which are being looked in to now.
    Update
    05 Sep 2013 18:46:07
    This is still being worked on.
    Update
    05 Sep 2013 18:58:21
    Things are getting back to normal now.
    Update
    05 Sep 2013 19:21:46
    Customers may need to reboot their DSL routers if they are still having problems with connectivity.
    Started 05 Sep 2013 18:05:00
    Closed 05 Sep 2013 19:04:46

    17 Aug 2013 17:30:46
    Details
    17 Aug 2013 17:30:46
    Customers who have UK domains with us are getting emails from Nominet, confirming the domains they have, and very much worded as if you have just signed up for a domain. This appears to be a side effect of some housekeeping we are doing to bring our domain management pages back up to date. The domains have not been re-registered or anything and have the same registration date, etc. Once again our control pages for UK domains allows contact details to be updated, NS records to be updated and DNS sec DS records to be updated to Nominet.
    Started 17 Aug 2013 17:27:06

    22 Jul 2013 12:28:57
    Details
    30 Aug 2011 10:01:46

    We've updated SSL certificates for our customer facing servers this morning (email, control pages).

    Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html

    Started 20 Feb 2012 11:22:47 by AAISP Staff

    20 Jul 2013 07:43:44
    Details
    18 Jul 2013 16:10:20
    We are doing router upgrades over the next few days. As usualy these should have little or no impact, but have been seen to cause a few seconds of impact on routing on some occasions and more so on IPv6. Last time we did upgrades there was a PPP restart on many TalkTalk lines. We think we know the cause of that and so will be doing upgrades in a slightly different sequence to see if that can be avoided. We do not do all routers on the same day, and so this will be over a number of evenings. There will be LNS upgrades over night as well to bring LNSs up to the latest release of code as well.
    Update
    20 Jul 2013 07:44:28
    The main routers have all been upgraded. There still some over night LNS switch overs to be done over the weekend to complete the upgrades.
    Started 18 Jul 2013 17:30:00
    Closed 20 Jul 2013 07:43:44
    Previously expected 22 Jul 2013

    09 Jul 2013
    Details
    24 Jun 2013 09:43:19

    Our office telephones are currently not working, but we expect them to be working very soon.

    Sorry for the inconvenience

    Resolution

    Calls are now working again.

    Closed 09 Jul 2013

    09 Jul 2013
    Details
    08 Jul 2013 11:16:11
    There have been three power blips this morning in the Bracknell area that has affected our offices. They are short, and typically affecting desktop PCs. Telephones are unaffected by this as we have various UPSs on routers etc, but please be aware that there may be a few minutes delay whilst desktop PCs restart.
    Started 08 Jul 2013 11:10:49
    Closed 09 Jul 2013

    14 Jul 2013 18:00:00
    Details
    09 Jul 2013 16:06:12
    We are going to be doing some upgrades on various routers in the evenings. These typically take a few seconds, and whilsy they should have no impact on routing they have been seen to cause a few seconds of disruption on occasion. We will also be doing some over night LNS switch overs which will be based on your selected time of night, as usual.
    Update
    09 Jul 2013 17:44:39
    One router had a minor flap when upgrading, we are trying again.
    Update
    09 Jul 2013 17:54:26
    First batch of router upgrades done, and one LNS switch scheduled for over night.
    Update
    11 Jul 2013 19:02:08
    Final batch of core upgrades complete. Oddly this caused a PPP restart on many TalkTalk lines, which we need to look in to as that was not entirely expected. LNS upgrades still to roll out over next couple of nights.
    Update
    12 Jul 2013 06:01:42
    The LNS switches have not quite worked as planned over night, and this is being worked on manually now.
    Started 09 Jul 2013 17:00:00
    Closed 14 Jul 2013 18:00:00
    Previously expected 14 Jul 2013 18:00:00

    01 Jun 2013 15:30:00
    Details
    22 May 2013 09:14:44

    We will be rebooting one of our core switches in THN on Saturday the first of June at midday. Before the reboot we will move sessions to route broadband traffic away from this switch to minimise disruption to service.

    We will also be carrying out some upgrades that will make the control pages unavailable for a few minutes.

    These are seperate works and will not be done simultaneously.

    Update
    01 Jun 2013 12:20:58

    The work on the switch reboot is about to start - Most of the services and connections that use this switch have already been moved over, we expect the affect of the reboot to be minimal.

    Update
    01 Jun 2013 12:24:23

    Switch reboot is happening now.

    Update
    01 Jun 2013 12:25:23

    The switch is now back online.

    Update
    01 Jun 2013 12:45:38

    The switch reboot has been successfull. Work on The Control Pages will begin shortly.

    Update
    01 Jun 2013 13:17:57

    Work on the Control Pages is being started, there will be a few minutes where access to the control pages is not possible as we will also be rebooting the machine.

    Update
    01 Jun 2013 14:14:31

    The Control Page server (Clueless) is being rebooted.

    Update
    01 Jun 2013 14:16:23

    The server is back online, we're checking that all is ok.

    Update
    01 Jun 2013 14:33:49

    This work is now complete, so please contact support as normal if you see any problems.

    Started 01 Jun 2013 12:00:00 by AAISP Staff
    Closed 01 Jun 2013 15:30:00
    Previously expected 01 Jun 2013 15:30:00

    30 May 2013 20:00:00
    Details
    30 May 2013 12:50:46

    We will be upgrading a router in Maidenhead which should cause little or no impact.

    If that works we'll be upgrading some other routers in Telehouse shortly afterwards.

    The changes are to do with the recent TCP SYN flood attacks affecting customer routers and some additional steps to mitigate the impact of these attacks on customers.

    Started 30 May 2013 17:00:00
    Closed 30 May 2013 20:00:00
    Previously expected 30 May 2013 20:00:00

    26 Apr 2013 09:40:38
    Details
    23 Apr 2013 11:25:28

    We apologise in advance to customers calling in and experiencing what sounds like loud drilling. We'll do our best to cope, but may need to call you back at times.

    In other news, we'll having air-conditioning installed and working by the end of the week.

    Started 23 Apr 2013 11:22:09

    26 Apr 2013 09:40:00
    Details
    26 Apr 2013 09:23:14

    We have a problem with our office telephones that is being investigated at the moment. We do apologise. 

    Resolution

    This has now been fixed, phones are rining again!

    Started 26 Apr 2013 09:20:00
    Closed 26 Apr 2013 09:40:00

    30 Mar 2013 08:31:00
    Details
    29 Mar 2013 20:49:52

    A small number customers with regular payments of Direct Debits on the 9th of each month have, this month, had a collection going through on the 2nd and not the 9th. This is not in accordance with the notice previously given for the regular amount as it stated the 9th. Obviously this is wrong and we do apologise for any inconvenience caused. We have re-written the whole of our Direct Debit system from scratch this month, and this is the only issues that has come to light. We have already made changes to ensure this does not happen again.

    Under the scheme rules customers are entitled to reclaim the Direct Debit from their bank as not having had correct notice. Normally bouncing or reclaiming a Direct Debit would incur an admin fee, but our accounts department have been instructed to cancel these as it is our mistake. Anyone doing this will have to ensure they do pay by the 9th though, so it is unlikely to be sensible for people to do a claw back the payment and then send it back to us a few days later. We understand some people may, however, bounce the payment or have insufficient funds on the 2nd. Customers that have incurred any loss as a result of this, such as bank charges, should contact the accounts department.

    Once again, sorry for any inconvenience.

    P.S. Part of the rework meant writing a specification for how the DDs are collected, and customers may find this useful. http://aa.net.uk/legal-dd.html

    Update
    29 Mar 2013 22:25:02

    We have cleared all records of "regular" payment notifications from the system and added new checks in place to ensure we track these better in future. This means that your next DD collection will have 5 working days notice. This will then re-establish regular payments for those that pay the same each month.

    Once again, sorry for any inconvenience.

    Resolution

    We have notified the affected customers individually now.

    Started 29 Mar 2013
    Closed 30 Mar 2013 08:31:00

    22 Mar 2013 10:11:18
    Details
    22 Mar 2013 10:05:51

     

    Thinkbroadband have conducted a massive survey which asks a lot of important questions about the way people view their broadband. Well worth a read.

    http://www.thinkbroadband.com/broadband-survey.html

    There are some impressive stats relating to AAISP, e.g. pages 28 (value), 29 (speed), 31 (reliability), 32 (customer service), even 34 (social media use), 38 (loyalty).

    However, the survey covers much more than just rating ISPs, such as people's views on parental controls for Internet access, how long they spend accessing the Internet per day, and what things are most important to them...

    Well done everyone at Thinkbroadband - it is an impressive survey. And thank you to our customers who completed the survey

     

    Started 22 Mar 2013 10:00:00

    01 Apr 2013 10:34:30
    Details
    17 Mar 2013 09:21:47

    We have slightly changed the system that sends the emailed Direct Debit notices.

    These are now plain text only, no PDF attachment. And, temporarily, they are not PGP signed. We are however working on more general way to handle signing and even encrypting of emails to customers.

    This is part of a general tidy up of DD collections making the current, somewhat complex, processes for working out when a collection is made in to a more maintainable and concsistent system.

    Customers should not see any significant changes, but as ever, if you have any questions, please do contact accounts.

    Update
    20 Mar 2013 10:33:56

    Please note that the notices are now again PGP signed, but they do not currently have a PDF attachment. We are looking in to ways to provide the PDF again in due course.

    Update
    21 Mar 2013 19:02:55

    We are now including a link to a PDF for the confirmation in the signed email.

    Update
    24 Mar 2013 08:58:37

    We are now sending signed plaintext and attached PDF as before.

    Sorry for the time this has taken to update.

    Any questions, please contact the accounts dept.

    Update
    24 Mar 2013 09:46:01

    There were minor issues with duplicate notices sent this morning, but these seem to be resolved now.

    Started 17 Mar 2013
    Closed 01 Apr 2013 10:34:30
    Previously expected 02 Apr 2013

    02 Mar 2013 16:15:08
    Details
    01 Mar 2013 09:19:34

    Invoices issued today have some missing items - such as VoIP numbers, calls, extra services etc. Invoices for these missing items will be raised later today.

    We apologise for the confusion this has caused.

    Resolution

    The missing items were VoIP and SIM charges and have been issued as a separate invoice later in the day - sorry for any confusion caused.

    Started 01 Mar 2013 09:15:00
    Closed 02 Mar 2013 16:15:08

    27 Feb 2013 15:00:00
    Details
    15 Feb 2013 12:55:48

     

    We will be physically moving the router that runs our unofficial L2TP service to a different data centre. During this time L2TP connections will be offline. -This affects customers connecting to l2tp.aa.net.uk and does not affect our wholesale customers.

    This will happen on Wednesday 27th February in the afternoon, and it expected to take about an hour.

     

    Update
    27 Feb 2013 12:09:16

    This work will start a little later on this afternoon - We'll post updates before the work starts.

    Update
    27 Feb 2013 13:34:40

    This work is about to begin. We estimate service will be restored in about 60-90 minutes time. 

    Update
    27 Feb 2013 14:26:06

    L2TP should be back now.

    Resolution

    Work completed successfully

    Started 27 Feb 2013 13:00:00
    Closed 27 Feb 2013 15:00:00

    27 Feb 2013 12:05:43
    Details
    15 Feb 2013 12:21:08

    We will be physically moving the server that runs our secondary DNS service to a different data centre. This is to be considered an at-risk period for customer domains.

    This will happen on Wednesday 27th February in the afternoon, and it expected to take about an hour.

    The IP address will remain the same.

    The primary nameserver will be untouched.

    Update
    27 Feb 2013 11:23:20

    This work will commence shortly, a little earlier than expected.

    Update
    27 Feb 2013 11:29:32

    This is happening now, it should be back in about 30 minutes.

    Update
    27 Feb 2013 11:40:23

    The server has been physically moved and it's networking is currently being reconfigured.

    Update
    27 Feb 2013 11:50:13

    Networking is configured, we're foing a final reboot to check all is in order.

    Update
    27 Feb 2013 12:06:54

    This is work has completed. Staff will be monitoring.

    (We'll be doing planned work L2TP (status.aa.net.uk/1734) on a bit later this afternoon.)

    Resolution

    This is work has completed. Staff will be monitoring.

    (We'll be doing planned work L2TP (status.aa.net.uk/1734) on a bit later this afternoon.)

    Started 27 Feb 2013 12:00:00
    Closed 27 Feb 2013 12:05:43

    25 Feb 2013 14:42:11
    Details
    25 Feb 2013 14:19:53

    Our office is currently offline, this will mean phones will not work.

    We should be back up in a few minutes

    Update
    25 Feb 2013 14:23:14

    We're now back online!

    Resolution

    This was due to a router upgrade

    Started 25 Feb 2013 14:17:33
    Closed 25 Feb 2013 14:42:11

    25 Feb 2013
    Details
    22 Feb 2013 11:34:48

    We will be upgrading some routers over the weekend - not all on the same day, and starting a routine upgrade of LNSs over several days. This should not have an impact.

    Update
    23 Feb 2013 09:16:30

    The first batch of upgrades are complete.

    Started 23 Feb 2013
    Closed 25 Feb 2013
    Previously expected 25 Feb 2013

    17 Feb 2013 07:57:04
    Details
    17 Feb 2013 06:51:33

    Our web site, and hosted customer web sites are not responding. We are looking in to this.

    Update
    17 Feb 2013 07:32:53

    This issue is the same as the one affecting email, and is related to a disk server.

    Update
    17 Feb 2013 07:43:06

    We are working on disk server now

    Resolution

    We sites now working.

    Started 17 Feb 2013 06:22:30
    Closed 17 Feb 2013 07:57:04

    11 Feb 2013 12:53:12
    Details
    11 Feb 2013 12:40:11

     

    We've just had a blip causing ADSL lines to drop of and reconnect, this also affected routing for about a minute.

    More info to follow shortly.

     

    Update
    11 Feb 2013 12:53:12

    The issue affected both BGP links and LNS traffic, cauing lines to reconnect. Not all lines and services were affected. The lines have reconnected quickly, and the problem seems to have cleared without any need for any intervention from staff.

    We are investigating the cause - this looks to be something external that has triggered this. We are trying to find exactly how this has happened so we can avoid the issue in future.

    General Users Affected 33%
    Closed 11 Feb 2013 12:53:12

    08 Feb 2013 23:59:59
    Details
    29 Jan 2013 11:11:19

    As those following my blog may have noticed, we are moving staff around in the office - we have a new room all set up, and the heating finally fixed, and new carpet tiles and so on.

    During the rest of this week we will be moving people in to the new office, which means there may be some disruption to call handling. We are obviously working to ensure everything runs smoothly with minimal impact on customer services.

    Please do bear with us if specific staff are not available for short periods while they move desks.

    Started 04 Feb 2013
    Closed 08 Feb 2013 23:59:59
    Previously expected 08 Feb 2013 23:59:59

    26 Jan 2013 04:59:58
    Details
    26 Jan 2013 04:56:46

    We have lost access to everything in HEX 6/7 which means L2TP only customers are off line. This also affects some of our general servers. This is being investigated now.

    Update
    26 Jan 2013 04:57:41

    The issue is a result of a port/link failure earlier. We are re-configuring access via an alternative route now.

    Resolution

    Access has been restored

    Started 26 Jan 2013 04:31:00
    Closed 26 Jan 2013 04:59:58

    17 Jan 2013 10:53:10
    Details
    17 Jan 2013 10:44:52

    As you know there are some warnings for bad weather which may mean we do not have the full complement of staff in the offices on some dasys - as usual we are arranging that staff can, where sensible, work from home, so as to try and ensure the usual levels of cover.

    Please bear with us, especially right at the start of the day, if there are delays answering calls and emails, as staff discover whether they are able to get to the office or not.

    Started 17 Jan 2013 10:42:58

    16 Jan 2013 16:20:20
    Details
    16 Jan 2013 16:17:45

    In an effort to move our services to use a single domain (aa.net.uk), we are now running our billing system from https://accounts.aa.net.uk/aal/

    Started 16 Jan 2013 16:16:16

    13 Jan 2013 16:29:45
    Details
    13 Jan 2013 16:29:45

    We have finally created a new World of Warcraft Guild for the Horde on Thunderhorn: AAISP Darkside

    Brand new guild so no rank yet but customers welcome to join.

    Started 13 Jan 2013 16:28:46

    13 Jan 2013 15:39:02
    Details
    13 Jan 2013 15:39:02

    As some of you may know, we have SSL certificates on many of our servers and services that make use of CACert as a certificate authority. Some time ago we changed the certificates for our accounts pages and ordering pages, and now we have also obtained certificates for many of our web servers under aa.net.uk. These use a "known" CA and so no longer cause the same warning in browsers if you do not have the CACert root certificate. They are no more secure as a result of this change, and in some ways are slightly less secure, but this avoids the warnings and so we have bowed to the "protection racket" that is certificate signing. The new certificates do not cover use of aaisp.net.uk based domains.

    Ironically this change has caused some concern amongst customers. However, it should generally mean less queries on the matter.

    As a result we are gradually changing URLs used for some things, e.g. ordering is now on order.aa.net.uk not www.s-s-l.co.uk. We will soon be changing the accounts pages over as well, probably to accounts.aa.net.uk, but wil be keeping the old URL as well for the time being.

    At the moment SSL based email servers (web and TLS on SMTP, IMAP, etc) are not changing, and will continue with CACert based certificates.

    in the long term we expect DANE to take over from CA certificates, and we plan to have proper DANE support for our secure web sites soon.

    Sorry for any concern caused.

    Started 11 Jan 2013

    17 Dec 2012 16:08:16
    Details
    17 Dec 2012 16:08:16

    We have been automatically posting updates from our Status page to our facebook page, however, we've decided to separate out our Status updates from the other posts on Facebook. So, from now on Status posts will be on a separate Facebook page, AAISPStatus:

    http://www.facebook.com/AAISPStatus

    Do 'Like' the page if you want to be kept up to date with Status posts whilst on Facebook.


    08 Dec 2012 08:40:18
    Details
    08 Dec 2012 08:05:18

    We updated a couple of routers this morning - this is routing work, and usually managed without even dropping packet. However, we are seeing some issues with IPv6 at our office at present. It is not clear if this is affecting IPv6 for services in Maidenhead or not yet, and we are working on this now.

    Update
    08 Dec 2012 08:13:58

    It looks like this is affecting IPv4 as well, at least to our offices.

    Update
    08 Dec 2012 08:16:37

    This is looking very much like it is our offices that at suffering and not general services out of Maidenhead, but we are still trying to track down the problem.

    Update
    08 Dec 2012 08:40:13

    This looks to have affected some Ethernet customers out of Maidenhead as well.

    Update
    08 Dec 2012 08:41:21

    We have found the cause - BGP announcements for some prefixes out of Maidenhead were flapping. This has been resolved now, and we understand how it happened. We are looking in to the logic more carefully to work out if this was something we should have been able to predict.

    Started 08 Dec 2012 08:00:00
    Closed 08 Dec 2012 08:40:18

    06 Dec 2012 15:08:28
    Details
    06 Dec 2012 13:39:18

    We are currently experiencing problems in the office meaning or connection to the internet is down.

    Update
    06 Dec 2012 13:52:16

    We are back.


    01 Dec 2012 16:18:18
    Details
    01 Dec 2012 16:18:18

    It seems the issue with odd DD notices was not as simple we we expected and a number of customers have DD notices for collections which are not on the right dates. We are clearing and re-issuing all DD notices today.

    I do apologise for any inconvenience caused.

    Update
    01 Dec 2012 16:58:49

    A small number of people recieved a blank Direct Debit notice when they do not in fact pay by Direct Debit. Obviously this is a mistake and payment will not be collected from such people. Pay as normal.

    Started 01 Dec 2012