Since around 1am it looks liike custoemrs are having problems connecting. We have done an LNS upgrade which means around a third of customers are off line. This is being investigated now.
This looks related to the upgrade and seems to be related to platform RADIUS, but we are still investigating now.
Lines are coming back, but we are working to understand the issue as it appears to be RADIUS or BGP related.
The fix to this makes no sense, and will be reviewed later in the morning. The issue appears to relate to platform RADIUS from BT. We changed the way the endpoints are announced to BT and it all sprung in to life, but that makes no sense. The LNS upgrade does not explain it either. There may be some planned work as a result of this to investigate further.
The problem was not jus BT, but the fix we found was a result of changing BGP to BT which caused it to flip which RADIUS it was using. We have confirmed the issue is platform RADIUS and caused by LNS upgrades. The platform RADIUS is currently directed to a non-upgraded box and so BT, BE and data SIMs should all be able to connect now.
There will be s/w upgrades over the coming nights to correct this.
It looks like some issues with BE lines still, which we are working on
We have found the underlying issue, and will be re-jigging lines over night, sorry for any inconvenience.
This will me some loss of graphs.
The error has been corrected, and lines are being moved between the two affected LNSs now. Graphs for around 1/3 of customers will be missing before around 3:30 this morning.
Even with two working platform RADIUS servers, Be lines are still not connecting. We are not sure if their RADIUS have somehow blacklisted us because of lack of response, or what. Still working on that.
We have asked fluidata (who manage the BE connection) to check their RADIUS servers for us. This may come back anyway over night if it is a timeout/blacklist issue on their RADIUS.
We have just had a resonse from Fluidata and they seem to have given their end a kick and all is well now.
Graphs lost for some lines before around 03:30, and some other LNS updates over night during the week.