We have a switch upgrade planned for Sunday - but it seems that somehow it knows!

There was some sort of issue yesterday at 9am and again at 11:20pm. Monitoring suggests something not quite right since 9am yesterday and seems to point to the switch which is going to be upgraded on Sunday.

We are monitoring carefully and if there are any issues during the day we will have staff carry out the upgrade as an emergency during the day.

This could affect several services, including VoIP. Hopefully this will not be an issue, and the upgrade will go ahead as planned on Sunday.

Work is progressing, some servers are already up and working on the new switch.

Everything seem sto be working ok - Customers may have seen an outage of around 2 minutes.

Some customer hosted servers are offline at the moment, checking the configuration for them at the moment.

The swap over has been sucsessful. Most services had a couple of minutes of outage, and some hosted servers were a longer due to a configuration issue.

We are just rebooting one of our Disk storage servers due to a problem affecting email and web page hosting services.

We expect services to be restored in a few minutes

Service restored.

service restored within a few minutes, appologies for this outage.

We're seeing packet loss affecting our offices at the moment.

We are investigating and have an engineer on their way to site, and so this should be considered an at-risk period for connectivity in Maidenhead until we isolate the problem.

This has now been resolved.

There is currently a problem with IPv6 routing in our maidenhead datacentre. This will be affecting Ethernet customers as well as access to some of our server over IPv6.

IPv6 is now routing again.

This looks like one or our transit links was announcing IPv6, but not routing it. We've taken the link down for the time being whilst we investigate further.

Transit provider confirm that they are investigating.

There was a problem with a link to a transit provider, they have now resolved this, are are looking to how this problem can be prevented in the future.

One of our stoarge servers used for email and web pages is needing to be rebooted. This is happening now.

Web pages and email access will be affected for a couple of minutes.

The Disk server is booting back up now, service will be restored very shortly.

Web and email servers are now back online.

This problem was caused by some kind of file system problem which we will investigate.

IT looks like a few severs rebooted, so suggests a power problem. Trying to get some more information now.

IT would affect web and email hosted services as well as VOIP.

It has been confirmed that this was a problem with power. It is being worked on.

There are a few minor problems, like one of our spam checking servers is offline (the other servers should cope), one of our customer facing SMTP relays died (no longer in DNS so not customer affecting) and customised voicemail messages are not working on our VoIP service.

The accounts system web interface is down too (although the accounts system itself is fine).

These are not major problems, and we'll continue to work on them in the morning.

It looks like there are some ongoing connectivity problems with Palsant's transit feed. They are manifesting themselves as brief outages, but it doesn't look too serious.

We'll post details when we know more.

Power went off at 22:07:47 and was restored at 22:10:59

Engineers are on site this morning to resolve remaining issues with some servers.

Transit seems to be stable now.

Most services are working correctly - some issues with voicemail/announcement messages at present, but this is being worked on. One of the outgoing mail servers has not come back to life and will be worked on this morning so that the mail queue can be sent ASAP.

Not sure what happened, but both core routers reported a power outage.

We'll try and find what happened.

We are are seeing intermittent packet loss affecting services in Maidenhead - this will be most noticeable on VoIP calls where they may be short periods of silence during a call.

The loss is sporadic, it lasts a few seconds and there are long periods of time where the loss is not present, as yet there is no clear pattern to pinpoint the cause -a tricky one to track down!

We are investigating and working with the Datacentre to fix this.

We had put in a temporary fix whilst we get to the bottom of the problem.

There was a rather odd issue with routing in Maindenhead affecting several of our servers there. We think this may be an issue with a switch, but we have managed to get things working again. There may be some planned maintenance as a result later.

There is a network problem in our Maidenhead datacentre at the moment, this will be affecting some of our services, mainly VoIP, email (sending and receiving) our web servers and hosted servers.

VoIP customers will be most affected and may have some calls with audio problems (breaking up)

We are tracking down the source of the problem

We have tracked the source and are in the process of shutting down the compromised customer machine.

The compromised machines have been shutdown.

Compromised customer servers have been shut down.

Our main disk storage server is having problems at the moment, this is affecting email and web server storage.

Engineers are investigating.

One of our disk server is being rebooted.

POP/IMAP services back

WWW Service back

We believe we know what caused this. We will do planned work to apply an update.

Please see http://status.aa.net.uk/apost.cgi?incident=1450 for planned works

We're investigating reports of strange routing problems involvng access to/from our Maidenhead data centre

Our routers in the Maidenhead data centre had some issues over night. These may have caused a few seconds outage in some services and Ethernet access. Broadband services not affected.

Not the switch that we rebooted earlier this evening, but another switch is currently being rebooted. This was not planned and not related to work we've been doing this evening with email.

This will affect some hosted customers for a few minutes, we do apologise.

this will also affect outgoing mail (sending mail via smtp.aaisp.net.uk)

It's expected to be resolved in a few minutes though

This took rather longer than expected, but has now been done and servers are now visible again.

Some minor router reconfiguration work this evening to add additional routing resilience resulted in an unexpected side effect that took a few minutes to rectify.

This affected specifically the routing between London and Maidenhead meaning broadband customers lost access to email and VoIP, and Ethernet customers lost access to A&A servers in London and broadband lines.