INCIDENT: SysEleven STACK issues in region CBK Wednesday 16th August 2023 19:37:00


Affected Components: SysEleven Stack, region CBK

Incident Start: 16th Aug. 2023/23:37


Description:


Customer Impact:

Update: 17th Aug. 2023/07:25 am

We identified a problem with the gateway and right now no external traffic is possible, we are still working on the restoration

Update: 17th Aug. 2023/07:55 am

We have been observing network problems in the Cloud Region CBK in relation with the underlying SDN (Midonet). So far, we have not been able to restore the functionality of the Cloud Region.

Systems in the CBK cloud region are currently not accessible from public networks, which means that it is also not possible to transfer data to and from the systems. Authentication to MetaKube clusters and individual other systems (SysEleven Login, Open ID Connect, code.syseleven.de) is currently restricted. The systems in the CBK region are basically running, so we do not currently expect any data loss.

We are working at full speed to resolve the issues.

Update: 17th Aug. 2023/08:35 am

Cluster creation in all regions is currently taking longer than usual.

Update: 17th Aug. 2023/08:45 am

Network Connectivity in region CBK is slowly recovering

Update: 17th Aug. 2023/09:10 am

The cause of the Cloud Region CBK network issues has been identified and resolved, Cloud Region CBK functionality has been restored to most systems, or should be shortly.

If you continue to observe problems, please contact us.

Update: 17th Aug. 2023/09:30 am

We are running scripts to automatically redeploy loadbalancers in the cloud region CBK. If you experience further network related problems this might be related and should be fixed shortly.

Update: 17th Aug. 2023/10:40 am

Some loadbalancers are still causing issues, we are looking into the issues, while most loadbalancers should be back online and functional.

Update: 17th Aug. 2023/11:10 am

We consider the incident as resolved. Please contact us if you still experience problems.

Update: 21th Aug. 2023/08:15 pm

Reason for Outage (RfO)