Major Incident – INC0107444 – Multiple circuit outage
22nd July 2022 12:25:43
Updates:
Description: Major Incident – INC0107444 – Multiple circuit outage
Current Impact: Impacting a number of Leased Line, Broadband and Consumer circuits. Exact numbers unknown
Incident Start Time: 22/07/2022 11:57:59
Major Incident Raised: 22/07/2022 12:25:43
Major Incident Manager: Sam Abberley
Location/City: Nationwide
Update Number: 2
Completed Actions:
– Reports of circuit impact into the CityFibre TSC
– CityFibre TSC engaged CityFibre NOC for initial investigations
– CityFibre NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled
– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
– Further impact to Consumer circuits discovered and acknowledged
– NOC investigations determined an issue within the core network emanating from a specific location
– NOC contacted hardware supplier and raised a Priority 1 case
– All logs provided to hardware supplier for analysis
– Internal Bridge call convened
Current Action Plan:
CityFibre NOC have been investigating further and have determined that the issue is emanating from a specific location with the core network. As such, this device has been checked and the hardware supplier has been contacted directly for a Priority 1 case to be raised with them. During this time, CityFibre NOC have attempted to alleviate some issues across the core network by performing a workaround to reroute traffic to avoid the suspected affected device.
All of the requisite logs have been provided and CityFibre NOC are working alongside the hardware supplier to determine our next course of action. As of yet, it is still unclear as to when full service can be restored however we will endeavour to provide this as soon as we have it available.
We apologise for the disruption and hope to have a further update soon.
Next Update:
17:30
Description: Major Incident – INC0107444 – Multiple circuit outage
Current Impact: Impacting Leased Lines as well as Broadband. Unknown number as of yet
Incident Start Time: 22/07/2022 11:57:59
Major Incident Raised: 22/07/2022 12:25:43
Major Incident Manager: Sam Abberley
Location/City: Nationwide
Update Number: 1
Completed Actions:
– Reports of circuit impact into the CityFibre TSC
– CityFibre TSC engaged CityFibre NOC for initial investigations
– CityFibre NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled
– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
Current Action Plan:
At this juncture, it has been determined that several tunnels are currently flapping following initial investigations by CityFibre NOC.
We are working to determine as to why the tunnels are flapping in order for restoration works to begin. At this point, we are unable to provide an estimated time of repair however once one is available we will provide it as soon as possible.
We apologise for the disruption and hope to have a further update soon.
Next Update:
15:30
22/07/2022 13:38
Description: Major Incident – INC0107444 – Multiple circuit outage
Current Impact: Impacting Leased Lines as well as Broadband. Unknown number as of yet
Incident Start Time: 22/07/2022 11:57:59
Major Incident Raised: 22/07/2022 12:25:43
Major Incident Manager: Sam Abberley
Location/City: Nationwide
Completed Actions:
– Reports of circuit impact into the CityFibre TSC
– CityFibre TSC engaged CityFibre NOC for initial investigations
– CityFibre NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled
Current Action Plan:
A Major Incident has been declared as of 12:25 following the confirmation of an issue affecting our core network. Initial impact assessment has determined that both leased line circuits and standard broadband circuits are affected.
Currently, we are investigating the issue alongside the CityFibre NOC team and are working to restore service as quickly as possible. An internal bridge call is being held at 12:45 to discuss the current impact and determine a plan of action for restoring service.
We apologise for the disruption and hope to have a further update soon.
Next Update:
13:30
22/07/2022 12:39
We are aware of an ongoing incident affecting a number of services. Our Engineers are conducting preliminary investigations to determine the comprehensive Service Impact Analysis.
We apologise for the inconvenience caused. Our main priority is to deliver the levels of service that our customers deserve and as such we have invoked our Major Incident Process. CityFibre are working to affect a full restoration as soon as possible.
We will continue to issue regular updates until all services have been restored. Further details will be issued within 15 minutes.
22nd July 2022 12:25:43