One of our webmail servers is being rebooted, this is similar to the problem yesterday. Once it's back up we'll be considering what to do with it so as to stop this problem happening again.

This will affect some Squirrelmail users, and all Roundcube users.

It's back up.

There have been some times recently where customers have had their email accounts compromised and used by a third party to send out lots of spam through our servers. This results in some of our servers being added to third party black lists which causes problems.

Whilst we have monitoring in place and do act when this happens we have implemented some extra spam checking on our outgoing email servers.

This mainly affects authenticated email, and will result in IP addresses being fire-walled for a period of time if it seen sending multiple spam messages.

We'd appreciate feedback if legitimate email is wrongly classified by mistake.

We're in the process of over-hauling our spam checking servers. This is involving new servers and updated software. These new servers will be put in to service over the next few weeks.

We expect this to further improve our spam checking accuracy.

We welcome feedback, especially if spam classification is incorrect. The new spam servers are called G.spamless, H.spamless, I.spamless and J.spamless - these names will appear in the headers of the email.

Since noon on Wednesday 19th we've been spam checking incoming email with the new servers. We are monitoring and would appreciate any feedback people may have with respects to whether some spam emails are not being marked as spam.

The RoundCube webmail has been updated. 

One of the things you'll soon notice is the new 'skin' - you can change back to the old style if you prefer by going to: Settings - User Interface, and set the Skin to 'Classic'

There is a problem at the moment with the initial login page not working and needing to re-enter your password on the next page - this is being investigated.

Logging in to roundcube is now working as expected from the initial webmail page - sorry for the inconvenience.

We'll be doing some work on our spam servers w/c 9th May. Including updating the spamassassin software and reviewing the various plugins and extra rulesets that we use.

We don't expect any downtime - we have multiple spam checking servers and we wil start this process using internal testing servers.

If you have any feedback on the spam checking capabaility, please do email support

We have changed the level of email that accounts for 'light' usage from 1,500 to 5,000 messages per month.

For more information please see: http://www.aaisp.net.uk/domains-emailhigh.html

To help reduce the spam we relay from infected of hacked computers, we run messages through spam and phishing signatures which check for known virus, phishing and spam messages.

We now also checking the IP address connecting to us against 'DNS Blacklists'.

(If you find your IP is on a blacklist then the MX Toolbox site is a good place to test your IP address against lots of blacklists.)

If we deny access then a suitable error message with details is given. You can speak to Support who can help diagnose a problem with you.

We will be monitoring this new feature and will be reviewing, do pass on feedback if you have any.

We've disabled the blacklists checks for the moment.

(More customers than expected where being blacklisted, we'll re-visit this next week)

-Just to calrify, we're doing these extra checks to help protect our mail server from being blacklisted themselves due to customers relaying junk email through us. -the goal is to ensure we provide a reliable email service without causing delays and spreading junk email accross the Internet.

As some of you will have seen, and as we announced recently, we have started to put email usage in to one of three tiers. They are Light, Standard and High.

The tiers are based on number of emails processed (including spam) so are a reflection not just of how much email you get but how much you put your email address about (e.g. on forum, web pages, news groups, etc). This is also affected by how many aliases you have or whether you have a wildcard alias. Emails to invalid addresses do not count.

We sent out emails this month to people who were not on the Light level of usage. The emails were not worded very well, for which I do apologise. I think you will find the ones sent next month to be a lot clearer.

We are only changing people off a lower tariff to a higher one if you have high usage for three months in a row. Even then the price change does not come in until the next bill. So this is lots of warning.

Are we trying to rip off customers? Of course not! Email is however an expensive business to run, and as you can see by the slow down last week it is not easy to keep on top of it. We had already ordered a new, and very fast, storage server which we hope to install this week. We have a lot of machines doing email (incoming, spam checking, webmail, imap & pop, outgoing all have at least 2 machines). This is more equipment, power, bandwidth, and rack space than any other service - yet for over half of the domains we run we include email free of charge with the broadband and the rest only cost £1/month.

We are keen to include a domain, email and web space with broadband still, but it is not sensible for someone to use the free domain to handle 250,000 emails a month for a company with hundreds of mailboxes and "pay" the same as someone that has a home email address and 100 emails a month, is it?

90% of domains on which we run email are in the Light category which is free with broadband (one domain per site) or £1/month+VAT. A cheap service.

For 9% of domains we handle over 1,000 emails a month and and this is the Standard email service. It is only £5/month even if you have broadband, which is not an expensive email service. Remember it is still unlimited mailboxes.

For 1% of domains we handle over 10,000 emails a month, and these are an handful of companies using our email with, typically, hundreds of staff mailboxes. These will be charged £100/month.

The exact proprtions will change, I am sure, as some people will take email elsewhere and some will stop wildcard mailboxes and so on.

We expect this change will either provide more money to help maintain and run a good quality of service, or move some of the high users again making the service better.

I hope that makes sense, but do feel free to ask me questions in irc.

We have updated the level of email included in the 'Light' tariff to be 1,500.

There are more details on the email tariffs on the following page:
http://www.aaisp.net.uk/broadband-emailhigh.html

Just a reminder for people using IMAP that we have an 'Archive' option. This is set on the Mailbox options on the Domain Control Pages.

The idea being that old messages can be moved in to an archive folder and sorted in to year and month sub folders.

This helps keep your Inbox and Sent Itmes folders optimised as they will have less messages in, meaning opening the Inbox would be quicker.

There are various settings. Please see the Email Archive page on our web site for more information. Do let us know if you have any ideas, send them in to support.

Collecting email by POP3 or IMAP on our 'C' email servers is slow today.

This is something we're aware of and working to resolve. The bottleneck is the storage servers, we are will be replacing this as soon as possible with a new, faster, solution.

(More information has been posted to this post regarding email.)

Background

Due to the recent problems we have with our outgoing mail servers being blacklisted (eg Hotmail), we've taken some steps to reduce the junk email that passes through them.

We've already added extra phish and spam checking.

One reason our servers get blacklisted is due to customers own email servers relaying bounce messages through us. These tend to be messages that the customer server has accepted from the sender, but then decides that it's spam and wants to bounce it back. Which results in our servers relaying bounces (containing spam), to innocent email addresses.

The Change

Over the next day or so, we'll be making a change to our outgoing email servers (smtp.aaisp.net.uk) so that they won't accept email that look like bounces. (ie messages that are from a Null sender, and with a subject line that looks like a bounce)

Workarounds

We would recommend that customers using their own email server a) send email directly using MX records, rather then relay the message on to smtp.aaisp.net.uk and b) don't accept spam and then bounce it.

Do contact Support if you have any questions or concerns.

We're looking into problems with our 'H' mail server.

IMAP and POP3 are unaffected but this will be affecting some webmail access.

A reboot of the server fixed it.

One of our mail servers is down. This will be affecting webmail (specifically RoundCube).

We're working on it.

Back up. Checking that services are running as expected.

All looks normal now.

One of our webmail servers is a little unhappy, and customer are seeing timeouts and long loading times when using webmail.

This is being looked in to.

New hardware has been ordered to replace this server.

We've altered the webmail server so that customers who normally get redirected the to 'G' webmail server are now being directed to the 'H' webmail server.

Customers do not need to change anything at their end.

Do let us know if this improves webmail for you for the time being.

We'll be upgrading the software on our IMAP and POP3 servers on Wednesday Evening from 9pm. We expect this to take an hour. During this period there will be times when collecting email from our servers will not be possible.

We'll update this status page during the work.

Work has begun on this. We hope to keep the server restarts to a minimum, there may be a periods of few minutes where email is not accessible over the next couple of hours.

This work has completed.

Our web mail servers are currently offline, this is being looked in to at the moment.

The service is back up after a reboot, we'll monitor and investigate the cause of this outage.

We'll be performing some work on our disk storange that is used for our email and web hosting services. 

This will mean an outage of about 15-30 minutes at 9pm on Thursday 7th March.

This work will commence at 9pm, in a few minutes.

Services are now coming back online.

Work completed

We will be taking email offline for maintenance, which will affect all inbound email access (except legacy servers) on Wednesday the 27th of February.

This work is not expected to take longer than an hour.

We will carry out the work early in the morning, between 6 and 7, to minimise disruption.

Outbound email services will not be affected.

We chose this time based on load on the mail servers; it's a good quiet time.

This has been completed successfully.

Our monitoring systems had flagged up a problem with email access. We are looking in to this.

The issue appears to be a problem with one of the disk servers.

Working on disk server now

The disk servers are now operational, and web sites are working. We are checking email servers now, but they may be busy catching up for a while.

Some mail servers back, still working on this. It may take a while to get all email services fully restored this morning.

Further mail servers back on line. We expect most customers are working now.

All looks back to normal now.

There is planned work to investigate the underlying cause on the 27th of February.

Our backup SMTP relay tertiary-mx.co.uk is currently not accepting email. This is only a minor problem as it should only be used if it's the backup MX for your own server and your own server is down.

We're investigating.

Now sorted, and our monitoring's been improved to make problems on tertiary-mx.co.uk easier to spot in future.

There is currently a problem on our customer facing 'H' mail server, affecting access to webmail (and possibly some IMAP/POP3 depending on client settings).

We are investigating.

Sorry, this affects roundcube everywhere! It's running again now on the H server.

We're still working on G.

Sorry, working again now but rebooting the server in a few minutes.

It's all back now. Please contact support if you are still seeing problems.

There was a minor problem fixed on Saturday. No further reports of problems, so all fixed now.

One of our webmail servers is currently being rebooted due to a faulty disk. Sorry for the inconvenience.

The server is back up.

Looks like some issues affecting email and web services. Being investigated.

It looks like it is the main disk storage which handles email and web pages.

Disk server restarted - sorry for delay - thanks Paul

Two of the servers that are part of our outgoing mail relay (smtp.aa.net.uk or smtp.aaisp.net.uk) appear to be on a blacklist.

We are investigating the cause of this and working to get this resolved soon 

We have found the cause and stopped email from the mailbox in question.

We are now in the process of removing any queued mail from this user and will then request the servers are removed from the blacklist. 

We've taken the affected mail servers out of DNS in the mean time.

The servers are no longer being black listed and are back in DNS.

We're restarting the 'H' webmail server as it's not very happy at the moment.

service restored

One of the servers that is part of our out going mail relay (smtp.aa.net.uk or smtp.aaisp.net.uk) has just suffered from a disk problem.

We've taken this server out of the pool, and will take 10 minutes for the DNS to update.

If you have problems sending email, please try again in 10 minutes. If still not working, then it maybe that your email program has remembered the old server and may need to be restarted.

The server is rebuilding it's RAID. We'll leave it out of service for the time being whilst we investigate.

Email in the queue is being processed, no email has been lost.

Email service should now be OK.

Our main mail servers are not available due to an issue with the disk storage. This is being investigated now.

This is affecting our web sites too

Service restored - sorry for any inconvenience.

Now back. We're investigating the cause of this.

Our outbound mail servers are occasionally refusing to send email with the error "Sender verify failed". This doesn't appear to be affecting many customers. We are investigating.

We're making changes to try and identify/resolve the underlying problem.

Still doesn't seem to be affecting many customers.

We've tracked this down to a peculiar DNS problem on one of our resolvers in the Maidenhead datacentre. 

We will carry on monitoring this, but it should be 'resolved'.

The 'D' email server will have one of its disks changed on Friday afternoon Monday at 5pm and will require a reboot. We expect this to take around 10 minutes.

This is being moved to Monday

This work is about to start.

The server is back online.

And is off again. Working on it.

After changing the disk the server was rebooted a second time and has failed to boot up. Datacentre staff have been tasked to investigate.

The Server is booting up now.

The server is rebuilding its RAID array, but is up and functioning. We're sorry for the inconvenience this caused.

Rebuild complete - it should be fine now.

One of our POP3/IMAP servers is experiences high load, engineers are looking in to it, but some customers may be seeing error messages in their email programs.

Load is down to a sensible level now, staff will monitor the server

We will be carrying out maintenance work on our legacy mail servers (A and B hopeless) on Wednesday the 14th of March. Mail services on these servers will be unavailable for periods of time while the work is carried out.

As the work involves hardware changes, they may be unavailable for relatively long periods of time.

We will be emailing customers who appear to be using these mail servers to notify them of this work and to urge them to upgrade to our newer mail system.

These servers are very old, and not very efficient, and so we are deprecating their use. As of the 14th of March 2012 they will no longer be officially supported. If they break they will be a low priority to fix.

If you have any questions, please email support.

The A mail server work is pretty much complete. Please let us know if anything's not working

Because of the time left in the day, we'll do the B server tomorrow instead. Sorry.

A mail server work complete - POP3 and SMTP relay, etc, tested and working.

B mail server done. Please let us know of any problems.

There was a problem on one of the outgoing email servers this morning. This has been resolved and we're looking in to the cause.

Rumours that we had in fact moved the mail servers to Translybonia to reduce expenses are greatly exaggerated. - (It was actually a database problem.)

Hotmail have decided to black list one of our email server IP addresses. We've taken it off line for the moment.

customer who have had bounce back messages from hotmail/live.com should resend them in an hours time.

One of the servers has been taken out of the 'pool'.

One of our webmail servers is off line at the moment, this will affect some of our email users from accessing it's web pages.

We expect service to resume shortly though and will update this post again.

This server (The G Mail server) is being rebooted.

Server rebooted, and back online.

We've have a few incidents this week where Hotmail have been blacklisting some of our IPs.

This has meant that some email has been bounced back.

We believe this is resolved, for now, so please do resend messages that have been bounced back,

We'll carry on monitoring the situation

We will be performing some routine updates on the email servers on the evening of 29th September.

This will happen between 8pm and 10pm, during this period there will be a few minutes where access to POP3/IMAP mailboxes will be unavailable.

This work has been completed.

We're looking in to problems with POP3 and IMAP collections, this is also affecting the webserver as they use the same storage systems

We have engineers working on the storage systems.

One of the storage servers has been restarted, and we're working on re-mounting the filesystems etc

Filesystems are being remounted etc at the moment  - service should be resumed shortly

Email and Web systems are back up, we do appologise for this outage.

We have a backlog of email that is currently being processed. This would mean that some email over the past 24 hours may be delayed.

There is an issue on the A mail server. We're investigating.

If you have any questions please contact support - it would be best to move to our newer mail system.

Mail services back up now. It looks like some email on the A server may have been lost.

Apologies for any inconvenience.

We strongly recommend moving to the C mail servers; A is a legacy server and will be turned off at some point in the future.

Our legacy email server, B.Hopeless, will be moved from our London datacentre to Maidenhead this Friday evening.

This will mean down time of a few hours between 7pm and 11pm.

It's IP address will be changing too, but our DNS records will be updated.

Customers can move on the new email servers, do contact support to discuss.

This work is nearly completed

We'll be upgrading the roundcube webmail today - this will mean some moments when it's not available - however, this should only be for a short amount of time. 

SquirellMail will be unaffected.

The upgrade has been completed. The new features include:

SMTP delivery Status notifications (under the 'Setting' icon when composing a message)
Threaded Messages option ('Options' icon on next to the column titles)
Contact Groups (under the Address book, you can now create groups)

We are needing to reboot the legacy email server, A.Hopeless, now.

Sorry for the short notice, it should be back within 10 minutes.

Server has been rebooted, and is back online.

At about 6:20am our legacy email server, B.Hopeless, stopped responding to our monitoring. We are investigating what the problem is and will update this post shortly.

Customers on this server will be unable to collect email at the moment.

A remote power cycle has not brought the server back. Data centre staff have been tasked to the server.

The server hardware does not look well. We are attempting to move the disks to another machine.

B.Hopeless is booting up on new hardware. It will take a little while longer for the smtp and pop3 services to be started.

Services have started, and the server is running. Staff will monitor the server.

Please do contact support to have your domain moved on to the new email servers.

Original hardware had died. Disks moved to new hardware and booted up.

Our legacy, A, email server has a file system problem. Access to email on the server won't be possible for the time being.

We hope to have this back up as soon as  possible, but no eta at the moment.

We strongly encourage customers on this server ask support to move them on the the 'C' email server.

This is just affecting POP3 collection on the A email server. Outgoing email is unaffected. 

Staff in the datacentre have been unable to boot the server due to file system errors.

Our plan is to build a new server, from backup, on new hardware in the Maidenhead datacentre.

This will take some time to do, but we are able to move customer on to the new email servers - please contact support who will be able to assist.

We'll update this post every couple of hours with progress, we hope to have the server running by the end of the day though.

We do appologise for this, most customers have been moved off this server in the past few years since we introduced the new email servers.

Are my emails lost? 

(Short answer: probably not. Long answer follows...)

Emails to your domain are normally in 1 of 2 places:

a) On our server, waiting for you to collect (or to be relayed on to your server)
b) In your email program, as you've already downloaded them from the server.

As the server is broken, new emails sent to you will be on either:
a) The broken server
b) On the server of the person sending you the email (as they have been unable to send them on to our broken server)

Emails on our server are backed up overnight, and we are restoring these at the moment.

Looking at the logs we have, it looks like the server stopped accepting emails sometime before 11pm, 9th March. Our latest backup completed at 3am, 10th March. So emails should not be lost. Once the server has been rebuilt, emails will be accessible again.

Emails that are on the senders server will be queued, and they will wait until the broken server is back up, or support have moved you to the new email server.

The ideal way forward for you is to move your domain on to the new email servers.

Staff are still working on building the replacement server. So far the progress has been good - in respect to copying files and databases etc.

There is still work to be done, but we hope to have the server running by the end of the day.

We are close to getting this working. We are sorting out problems caused by the age of the email server software and getting it to run on a new operating system.

The server appears to be working now. POP3 collections have resumed, queued email on the server will be retried. - it is being closly monitored.

Please do contact support if you have any problems, and please contact support to move your domain off this legacy server.

This has been completed.

The old server had disk and file system faults - it was decided that it would be quicker to build a new server (new hardware, new operating system) and data from backup, and also locate the new server in our Maidenhead datacentre.

The new server has new IP addresses as follows:

81.187.30.15
2001:8b0:0:30::51bb:1ef

(However, this information shouldn't be needed as DNS names should be used)

One of our webmail servers is currently being rebooted, we'll expect it to be back on line shortly, after we investigate what happened to the server to stop it responding.

The server is now back online.

One of our legacy email servers will be rebooted at lunchtime as a disk is being replaced.

The server will be shut down at 12:30, and will be down for 10-15 minutes.

This work has been completed.

We'll be upgrading the software on some of our email servers this weekend. As we have multiple servers this should not be noticed by customers.

This work has been completed.

We will upgrade the 'SquirrelMail' webmail on Friday. This is to 'address' the address book problem and also to bring the software up to the latest version.

The 'Roundcube' webmail will still be available though.

Note: The Address book problem has been fixed now, we'll still be upgrading the software on Friday though.

This has been postponed until further notice.

Our email servers and their storage servers are on dual power supplies, we will be moving where the power is connected to on 1 of the PSUs on each of our disk storage servers.

We don't anticipate a problem, as the servers will happily run on a single power supply.

This work should only take 5 minutes.

This work is being carried out now

We will be doing some minor work on the Email Storage servers - this is to finalise the move of domains on to our new disk storage servers.

Almost all domains are now using the new storage servers and this work is to unmount the old storage servers from the mail servers.

The impact on end users is expected to be minimal, at most there may be a few minutes where collecting email will not be possible.

This work is being started now.

This has been completed.