Connection issues (still affecting some BE lines)
MAJOR Closed Broadband and Ethernet
STATUS
Closed
CREATED
Mar 25, 02:04 AM (11 years ago)
AFFECTED
Broadband and Ethernet
STARTED
Mar 25, 12:50 AM (11 years ago)
CLOSED
Mar 25, 04:41 AM (11 years ago)
REFERENCE
1760 / AA1760
INFORMATION
  • INITIAL
    11 years ago by Adrian

    Since around 1am it looks liike custoemrs are having problems connecting. We have done an LNS upgrade which means around a third of customers are off line. This is being investigated now.

  • UPDATE
    11 years ago by Adrian

    This looks related to the upgrade and seems to be related to platform RADIUS, but we are still investigating now.

  • UPDATE
    11 years ago by Adrian

    Lines are coming back, but we are working to understand the issue as it appears to be RADIUS or BGP related.

  • UPDATE
    11 years ago by Adrian

    The fix to this makes no sense, and will be reviewed later in the morning. The issue appears to relate to platform RADIUS from BT. We changed the way the endpoints are announced to BT and it all sprung in to life, but that makes no sense. The LNS upgrade does not explain it either. There may be some planned work as a result of this to investigate further.

  • UPDATE
    11 years ago by Adrian

    The problem was not jus BT, but the fix we found was a result of changing BGP to BT which caused it to flip which RADIUS it was using. We have confirmed the issue is platform RADIUS and caused by LNS upgrades. The platform RADIUS is currently directed to a non-upgraded box and so BT, BE and data SIMs should all be able to connect now.

    There will be s/w upgrades over the coming nights to correct this.

  • UPDATE
    11 years ago by Adrian

    It looks like some issues with BE lines still, which we are working on

  • UPDATE
    11 years ago by Adrian

    We have found the underlying issue, and will be re-jigging lines over night, sorry for any inconvenience.

    This will me some loss of graphs.

  • UPDATE
    11 years ago by Adrian

    The error has been corrected, and lines are being moved between the two affected LNSs now. Graphs for around 1/3 of customers will be missing before around 3:30 this morning.

  • UPDATE
    11 years ago by Adrian

    Even with two working platform RADIUS servers, Be lines are still not connecting. We are not sure if their RADIUS have somehow blacklisted us because of lack of response, or what. Still working on that.

  • UPDATE
    11 years ago by Adrian

    We have asked fluidata (who manage the BE connection) to check their RADIUS servers for us. This may come back anyway over night if it is a timeout/blacklist issue on their RADIUS.

  • UPDATE
    11 years ago by Adrian

    We have just had a resonse from Fluidata and they seem to have given their end a kick and all is well now.

    Graphs lost for some lines before around 03:30, and some other LNS updates over night during the week.

  • Closed