Please see http://status.aa.net.uk/1750 for information regarding planned work that will affect web hosting on the evening of March 7th.

We will be performing a swap of the core network switches in our Maidenhead datacentre on the afternoon of Sunday 10th February.

The work is expected to take up to half an hour, during which there will be periods of time when service is affected.

The affected services will be:

Email, both incoming and outgoing
VoIP calls
Ethernet services terminated in Maidenhead
Web page hosting
Control panel and Billing system access
Customer hosted servers

General internet connectivity over DSL and Ethernet terminated in London will be unaffected by this work.

This staus page will be updated during the work.

This work will be done a little earlier than initially scheduled. The work will take place between 12:30pm and 2pm, but actual outage is only expected to be a couple of minutes.

One of our carriers that provide us connectivity in Maidenhead have planned (short notice) maintenance on parts of their network that provide us connectivity.

This will happen between midnight and 2pm.

We are not expecting this to impact customers, but is to be considered 'at-risk'.

We'll be rebooting one of our database servers shortly after 5pm today. This is to replace a faulty disk drive. 

There will be a period of a few minutes where some usage/logs won't be viewable from our control pages.

This is happening now, it should take about 10 minutes.

The server has booted up ok.

Apologies for some minor distruption with IPv6 today in relation to services we run in the Maidenhead Data Centre. There have been a few minutes during today where routing has not been quite right for IPv6 for some of the services.

We have renumbered our office on to a separate /48 to improve native IPv6 routing. (A /48 is visible to more of the Internet).

Over the next few days or weeks we will be renumbering services in the data centre from 2001:8b0::/48 to 2001:8b0:1::/48 addresses. Thanks to a customer (Lawrence) for allowing us to change is allocation to free that up.

If you want to trust an IPv6 as being from A&A infrastructure you should be able to trust 2001:8b0::/47 as this includes the /48 was have in London and the /48 we have in Maidenhead and our offices.

At present there may be a handful of machines which are not technically A&A infrastructure within this space that will be renumbered this week as well. A&A infrastructure should have reverse DNS under aa.net.uk as well.

We aim to parallel run the old and new IPv6 during DNS changes so that there are no issues with any services. If anyone has any issues please contact support.

We are not planning any IPv4 changes.

Note, customers that have IPv6 on fibre from Maidenhead can now request a /48 if you need, routed via your existing /64 link.

We are pleased to confirm the hosting in the Maidenhead Data Centre now has full 1500 byte MTU native IPv6 transit in operation. This improves the MTU to our various servers, to customer servers, and to Ethernet transit customers. If you have any issues, please do contact support, but this should be completely seamless.

We may also be making some changes to the IPv6 addresses for A&A servers in Maidenhead in due course. We'll post more details when that is all decided. This should not have any impact as we will, of course, overlap assignments and make necessary changes to DNS records.

This is ongoing. We're investigating.

This should now be fixed. Please let support know if you see any problems, or have any questions.

This was resolved yesterday afternoon.

We have a switch upgrade planned for Sunday - but it seems that somehow it knows!

There was some sort of issue yesterday at 9am and again at 11:20pm. Monitoring suggests something not quite right since 9am yesterday and seems to point to the switch which is going to be upgraded on Sunday.

We are monitoring carefully and if there are any issues during the day we will have staff carry out the upgrade as an emergency during the day.

This could affect several services, including VoIP. Hopefully this will not be an issue, and the upgrade will go ahead as planned on Sunday.

Work is progressing, some servers are already up and working on the new switch.

Everything seem sto be working ok - Customers may have seen an outage of around 2 minutes.

Some customer hosted servers are offline at the moment, checking the configuration for them at the moment.

The swap over has been sucsessful. Most services had a couple of minutes of outage, and some hosted servers were a longer due to a configuration issue.

There's a connectivty problem in Maidenhead. We're investigating.

Connectivity looks normal again now. We're still investigating the cause of the problem.

The problem was resolved by power cycling some hardware. Although we're still not sure exactly what caused this, we are suspicious of one of our switches. We already have planned work to replace the switch on Sunday.

Disk server playing uo affecting web pages and email.

Staff are working on this.

All sorted

We are just rebooting one of our Disk storage servers due to a problem affecting email and web page hosting services.

We expect services to be restored in a few minutes

Service restored.

service restored within a few minutes, appologies for this outage.

We're seeing packet loss affecting our offices at the moment.

We are investigating and have an engineer on their way to site, and so this should be considered an at-risk period for connectivity in Maidenhead until we isolate the problem.

This has now been resolved.

There is currently a problem with IPv6 routing in our maidenhead datacentre. This will be affecting Ethernet customers as well as access to some of our server over IPv6.

IPv6 is now routing again.

This looks like one or our transit links was announcing IPv6, but not routing it. We've taken the link down for the time being whilst we investigate further.

Transit provider confirm that they are investigating.

There was a problem with a link to a transit provider, they have now resolved this, are are looking to how this problem can be prevented in the future.

One of our stoarge servers used for email and web pages is needing to be rebooted. This is happening now.

Web pages and email access will be affected for a couple of minutes.

The Disk server is booting back up now, service will be restored very shortly.

Web and email servers are now back online.

This problem was caused by some kind of file system problem which we will investigate.

IT looks like a few severs rebooted, so suggests a power problem. Trying to get some more information now.

IT would affect web and email hosted services as well as VOIP.

It has been confirmed that this was a problem with power. It is being worked on.

There are a few minor problems, like one of our spam checking servers is offline (the other servers should cope), one of our customer facing SMTP relays died (no longer in DNS so not customer affecting) and customised voicemail messages are not working on our VoIP service.

The accounts system web interface is down too (although the accounts system itself is fine).

These are not major problems, and we'll continue to work on them in the morning.

It looks like there are some ongoing connectivity problems with Palsant's transit feed. They are manifesting themselves as brief outages, but it doesn't look too serious.

We'll post details when we know more.

Power went off at 22:07:47 and was restored at 22:10:59

Engineers are on site this morning to resolve remaining issues with some servers.

Transit seems to be stable now.

Most services are working correctly - some issues with voicemail/announcement messages at present, but this is being worked on. One of the outgoing mail servers has not come back to life and will be worked on this morning so that the mail queue can be sent ASAP.

Pulsant are carrying out work on transit in Maidenhead.

This work will affect all services we host in Maidenhead, such as email, web and VOIP.

The work is scheduled for between:

23:00 30/05/2012 and 04:00 31/05/2012

A 10 minute disruption to service is expected.

Not sure what happened, but both core routers reported a power outage.

We'll try and find what happened.

We are are seeing intermittent packet loss affecting services in Maidenhead - this will be most noticeable on VoIP calls where they may be short periods of silence during a call.

The loss is sporadic, it lasts a few seconds and there are long periods of time where the loss is not present, as yet there is no clear pattern to pinpoint the cause -a tricky one to track down!

We are investigating and working with the Datacentre to fix this.

We had put in a temporary fix whilst we get to the bottom of the problem.

There's a problem with web and email disk storage in Maidenhead.

We're investigating.

Now back. We're investigating the cause of this.

There was a rather odd issue with routing in Maindenhead affecting several of our servers there. We think this may be an issue with a switch, but we have managed to get things working again. There may be some planned maintenance as a result later.

Email and web servers will have about 30 minutes of down time as we move the file server that they use.

During this time access to email and web pages will not be available.

Date: Thursday 29th
Starting time: 8pm
Reason: Moving to secondary file server to allow for software updates 

This work will be starting shortly.

Web and incoming email (pop3/imap) servers will be offline for about 30 minutes form now.

Our webserver is now back online.

Email services are back up now.

The move has been completed as planned, services were restored within about 11 minutes.

There is a network problem in our Maidenhead datacentre at the moment, this will be affecting some of our services, mainly VoIP, email (sending and receiving) our web servers and hosted servers.

VoIP customers will be most affected and may have some calls with audio problems (breaking up)

We are tracking down the source of the problem

We have tracked the source and are in the process of shutting down the compromised customer machine.

The compromised machines have been shutdown.

Compromised customer servers have been shut down.

Our main disk storage server is having problems at the moment, this is affecting email and web server storage.

Engineers are investigating.

One of our disk server is being rebooted.

POP/IMAP services back

WWW Service back

We believe we know what caused this. We will do planned work to apply an update.

Please see http://status.aa.net.uk/apost.cgi?incident=1450 for planned works

We have been informed by the Maidenhead datacentre that they have some planned work on their UPS systems.

They will be carrying out a replacement exercise on end-of-life capacitors in the UPS.

The unit will be taken off-line for the duration of the work however due to the configuration of the installation there will be no impact to the electrical supply or resilience to equipment 

The work will be carried out on Monday 16th January, starting at 21:00 and should last 5 hours.

This work is factory advised and forms part of their robust Planned and Preventative maintenance regime. 

We're investigating reports of strange routing problems involvng access to/from our Maidenhead data centre

We are seeing a major issue in Midenhead - high levels of transit packet loss that will be affecting VoIP, email, and Ethernet customers (as well out our offices).

Looking like some sort of denial of service attack.

Sorry for the delay posting more details. This affects our links directly and making it difficult. The problem appears to be a huge denial of service attack invovling tens of thousands of sessions and filling gigabit links.

We have identified the target and disconnected it, black holed the target and even tried to divert traffic but to no avail as yet.

We are still working on this.

Just an update to say that this is still being worked on...

Still working on this

The problem is now only affeting Ethernet customers on the same block as the address being DDOS'd. We are still working on the issue. Most other services will be working fine now.

Some side effects on other services from Maidenhead, but we are still working on narrowing down the issue.

DOS attacks are, thankfully, rare. This has to be the biggest we have seen.

We will, of course, be talking to the customer who is being DOSed to fine what could have provoked such a major attack. There is usually a reason.

The blackhole for the target machine was removed at 1pm today, however the traffic was still being sent and affected VoIP, email and Ethernet services.

The block is now in place again, and we'll continue to investigate.

Similar to last night, access to servers and services in Maidenhead have high packet loss.

This will affect email, voip and Ethernet services.

Update to follow shortly.

Datacentre staff are working to blackhole the IP address that is the target of this attack.

The target IP address has been blackhole'd and service has been restored.

We'll update the initial post from yesterday with further updates to this. http://status.aa.net.uk/apost.cgi?incident=1364

We'll be moving some servers around between racks in the Maidenhead datacentre on 19th January. These servers are mainly related to email services, but we are not expecting customers to notice anyting whilst the moves take place.

The servers moving will be:

tertiary-mx.co.uk (the backup email relay)
A couple of the Spam checking servers (there are many, so one being down at a time won't cause a problem)
One of the outgoing SMTP servers (which is offline at the moment)

This work has been completed.

We have problems with Web and Email services at the moment, please see this post:

http://status.aa.nu/apost.cgi?incident=1298

On Sunday morning we'll be moving over the storage server that the email and web services use. The work should take less than 30 minutes, but as the work is carried out access to email and web sites will be unavailable.

We aim to make this move as quickly as possible.

FTP & RSYNC access to our web server has been stopped. 

The work of moving over the storage servers will begin shortly. We anticipate this to take less than 30 minutes.

The Work is starting now, access to websites we host and email will be unavailable whilst this is carried out.

Web pages are now being served again.

Email is back up

FTP and rsync access to our web server is now running.

This work is almost complete, customers should be back on email and their web pages being served again.

There appears to have been a severe network glitch affecting both diverse routes out of the Maidenhead data centre. Routing is recovering now, but this would have affected Ethernet customers and VoIP customers the most. Some authentication of DSL lines may have been delayed. Access to our email and web servers and other hosted services would also have been affected.

The incident appears to have lasted a few minutes. We are trying to get more details.

The carriers have confirmed they had an outage and should send an explanation shortly.

Carriers explain the fault as:

The cause of this incident was traced to events on the network which caused high CPU load on the transit routers. This then resulted in router protocol instability which affected transit services.

We have since stabilised the network and are developing solutions to be implemented which should reduce the impact of such events in the future.

Loss of connectivity was detected at 11:24 with service restored by 11:27.

We'll be moving a couple of servers between datacentres. Their IP addresses will not be changing, it's just a physical move.

The servers are:

A.Hopeless - One of our legacy email servers
-Access to receiving email during this time will not be possible.

Primary-dns.co.uk -  One of our internet facing DNS resolvers
We don't anticipate that service will be affected as the secondary DNS server will be available.

Customer Wiki
Access to wiki.aaisp.org.uk will be unavailable during this time

A couple of other internal use servers will be moved at the same time.

Start Time: 17:30
Duration: 1 hour 

This work is nearly complete - just waiting for various email services to start up.

Our routers in the Maidenhead data centre had some issues over night. These may have caused a few seconds outage in some services and Ethernet access. Broadband services not affected.

We are again upgrading a router this evening - it should have little or not impact as the system should fall back to the secondary router. There is, as always, a risk.

These upgrades, when complete on both routers (not both done together!) will mean faster fallback in the event of a failure in future as we are upgrading to support VRRP3 with sub second timing.

Arrrg, why is this never simple.

Abandoned for now - maybe later.

All sorted

We will be upgrading one of the main routers this evening to latest release. Sorry for short notice. As usual this hsould have little or no impact on services.

Not playing the game quite - so may have a few seconds disruption...

Completed

Please be aware that testing of the redundant power ATS (Automatic Transfer Switch) equipment at the maidenhead datacentre bill be carried out on Thursday 10th March at 19:00 and conclude by 23:00 on the same day.

Whilst the work taking place is non-intrusive and power redundancy will still be available at all times, customers should treat this as an at risk period.

We have lost comms with Maidenhead and we have an engineer going to site now, we are not sure what the issue is but it may be power related.

Email, VOIP and some others services will be affected.

This is also affectig Ethernet customers and hosted servers in Maidenhaed

There appears to haver been a fire alarm that has gone off and data center has been evacuated. No evidence of a fire though but power is down

Staff are just approaching the data centre now.

Power is being restored now

Our engineers are on site and power has been restored, servers of ours are coming back on line, further updates will be posted when we get them

Not all power has been resotred yet. Some services (control pages, VOIP, web) are still down. They should be restored shortly.

VoIP and control pages are back. Email and web should be back soon.

The A viop server is still down.

Email servers are mostly back, and web services are back. We've still got some voip problems and are working on it.

The A voip server has a database problem, and won't let customers register.

There is now a database problem on C SIP server too. Investigating.

Database fixed on C.

Database problems fixed on A and C servers.

Most services are back up now, we have had a number of hardware fail as part of the power outage incident. 

Currently the main problem is our email ticketing server - this is affecting emails to support/sales/accounts etc - and so is causing a delay in email replies.

There are also problems with:

The online ordering system
ADSL usage reporting
ADSL line status on Clueless

Other servers still have problems which we are working through, but other servers are managing with the load (may services have multiple servers).

The odd effect with lines not showing as on-line properly on clueless is fixed, and lines will clear properly over night as a result. PPP restarts of lines are needed but this is done automatically in stages to minimise disruption.

On-line ordering restored a little while ago.

I would just like to say that I am very pleased with how my staff have handled this today - tackling the issues in a sensible priority and updating status pages. This is a major issue with not just a power outage, but issues with access to the building, and possibly even a power surges as several pieces of equipment have failed totally. The backup arrangements for critical systems have worked as expected as has the maintenance of broadband internet access, DNS, and RADIUS authentication. Well done everyone. We'll try and get a more detailed explanation from the data centre in due course. Staff are working on the last of the issues now.

thankless (ticketing) still down and being rebuilt now.

We have now got our email ticketing system back online - we do apologise for the time this has taken, and the delay this has caused to email to support, sales and accounts.

We'll close this incident for now - but will add the official response fron BlueSquare when they have let us know.

This is the official report from BlueSquare (Our racks are in the building called BS2)

This is a Reason for Outage Report with details regarding the power supply in BS2/3 with BlueSquare Data Services Ltd.

At 10:06 on Thursday 17th March one of the six UPS modules located in BlueSquare 2/3 suffered a critical component failure which resulted in a dead short on the output side (critical load side) of the UPS. This failure also caused an amount of smoke to be released by the failed UPS system which resulted in the fire alarm activating and the fire service attending. Once the fire service was happy with the situation we were able to restore power to the site via the generators with the UPS system bypassed whilst we investigated the fault further.

Due to the short circuit occurring on the output side of the UPS this meant the other UPS’s immediately went into an overload condition which then switched all modules into bypass mode, as per the design of the system. This overload then transferred to the raw mains and tripped the main incomer to the site. This caused the overload condition to cease and power was lost to the site. The UPS manufactures then worked to check all the remaining UPS modules to ensure the same component was within specification, and to fully test each UPS system, replacing some components where necessary. No further faults were found on the remaining UPS modules, and load was then switched back to full UPS protection at approx 02:15 and building load was transferred back from the generators to utility mains at approx 02:25.

Due to the size of the failure we have commissioned an independent organisation to forensically examine the failed UPS module. This work is scheduled to be completed next week and we will provide further details once we receive their report. This was an extremely unusual type of failure and the manufactures have not experienced such a problem before, despite over 3,000 similar UPS units being deployed. This suggests there isn’t an inherent design problem in the units but we will not reach any conclusions until the forensic examination is complete.

The failed UPS module will be replaced within the next 4 weeks and until that time we will remain on ‘N’ redundancy level at BlueSquare 2 & 3. Further updates will be provided before this replacement work takes place.

A number of customers have asked as to why this failure could occur when we operate an N+1 UPS architecture. The reason for this is that all of the six UPS modules in BlueSquare 2/3 are paralleled together as one large UPS system. BlueSquare 2/3 only requires 5 modules to hold the critical load to the site, however we have an additional unit which provides the redundancy in the event of a UPS module failure. However, as this failure was on the common critical load side of the UPS (the same output that feeds the distribution boards which then in turn feed the racks) and all the UPS systems are paralleled together, this had the effect of causing all UPS modules to go down.

As an example, in a N+N configuration, such as in our Tier IV Milton Keynes site, a failure of this nature would not be possible as two banks of independent UPS systems operate providing true A&B feeds to each rack.

We are expecting to do some work on routers in Maidenhead. This should have minimal impact as we can work on one at a time.

Sorry for the very short notice.

We are actually doing another restart now.

We will be doing some work on the storage servers that are used by our  Web Server on Sunday. This will mean a couple of hours where FTP and RSYNC access will be disabled.

This work is currently being started. FTP and rsync access to our webserver will be unavalable for the next couple of hours.

This work has been completed. The webserver is now using new storage severs.

Please report any problems to support.

We currently have routing problems to our datacentre in Maidenhead, this will be affecting access to:

• Email - incoming and outgoing
• VoIP
• Hosted server
• Control Pages (Clueless)

We have engineers looking in to this at the moment, and will post anohter update shortly.

This is now working. It seems to be some routing/peering problem outside of our and BlueSquare's network - If we get any more details we'll post an update.

We are restarting one of the routers in Maidenhead this evening. It should be seamless in terms of routing though we have seen blips on IPv6 when doing this in the past so there is a risk of disruption for a few seconds.

Maindenhead handles ethernet customers, our offices, and access to many of our servers including VoIP for call set up and recorded calls.

We are just about to reboot a switch in our Maidenhead datacentre. This will affect a few hosted customer servers and some AAISP servers. It's expected to only mean a few minutes of downtime. Sorry for the short notice.

Not the switch that we rebooted earlier this evening, but another switch is currently being rebooted. This was not planned and not related to work we've been doing this evening with email.

This will affect some hosted customers for a few minutes, we do apologise.

this will also affect outgoing mail (sending mail via smtp.aaisp.net.uk)

It's expected to be resolved in a few minutes though

This took rather longer than expected, but has now been done and servers are now visible again.

Some minor router reconfiguration work this evening to add additional routing resilience resulted in an unexpected side effect that took a few minutes to rectify.

This affected specifically the routing between London and Maidenhead meaning broadband customers lost access to email and VoIP, and Ethernet customers lost access to A&A servers in London and broadband lines.