Past Incidents

Incident Start: 20th Aug. 2023/05:58 pm

We saw a problem with the network gateway for our cloud region cbk. We are still monitoring and evaluating what caused the issues.

Update: 20th Aug. 2023/06:13 pm

We consider the incident as resolved. Please contact us if you still experience problems.

[We have edited the wording for clarity on August 22nd (Service Management)]

Incident Start: 19th Aug. 2023/01:20 pm

We saw a problem with the network gateway for our cloud region cbk. We are still monitoring and evaluating what caused the issues.

Update: 19th Aug. 2023/01:30 pm

We consider the incident as resolved. Please contact us if you still experience problems.

[We have edited the wording for clarity on August 22nd (Service Management)]

Affected Components: SysEleven Stack, region CBK

Incident Start: 16th Aug. 2023/23:37

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• Connectivity is restricted

Update: 17th Aug. 2023/07:25 am

We identified a problem with the gateway and right now no external traffic is possible, we are still working on the restoration

Update: 17th Aug. 2023/07:55 am

We have been observing network problems in the Cloud Region CBK in relation with the underlying SDN (Midonet). So far, we have not been able to restore the functionality of the Cloud Region.

Systems in the CBK cloud region are currently not accessible from public networks, which means that it is also not possible to transfer data to and from the systems. Authentication to MetaKube clusters and individual other systems (SysEleven Login, Open ID Connect, code.syseleven.de) is currently restricted. The systems in the CBK region are basically running, so we do not currently expect any data loss.

We are working at full speed to resolve the issues.

Update: 17th Aug. 2023/08:35 am

Cluster creation in all regions is currently taking longer than usual.

Update: 17th Aug. 2023/08:45 am

Network Connectivity in region CBK is slowly recovering

Update: 17th Aug. 2023/09:10 am

The cause of the Cloud Region CBK network issues has been identified and resolved, Cloud Region CBK functionality has been restored to most systems, or should be shortly.

If you continue to observe problems, please contact us.

Update: 17th Aug. 2023/09:30 am

We are running scripts to automatically redeploy loadbalancers in the cloud region CBK. If you experience further network related problems this might be related and should be fixed shortly.

Update: 17th Aug. 2023/10:40 am

Some loadbalancers are still causing issues, we are looking into the issues, while most loadbalancers should be back online and functional.

Update: 17th Aug. 2023/11:10 am

We consider the incident as resolved. Please contact us if you still experience problems.

Update: 21th Aug. 2023/08:15 pm

Reason for Outage (RfO)

Affected Components: SysEleven Stack LBaaS API (Octavia), region DBL

Incident Start: 9 Aug 2023 15:00

Description:

• Creating octavia load balancers failed in region DBL. Updating existing load balancers brought them into an immutable state ("PENDING_UPDATE") after the update failed.

Customer Impact:

• Creating or updating octavia load balancers failed

Update: 18:30

Creating and updating load balancers is possible again. Some load balancers that became stuck in the PENDING_UPDATE state will be repaired now.

Update: 19:45

The load balancers that were stuck in PENDING_UPDATE have been repaired. The octavia LBaaS service is now fully functional again.

Affected Components: SysEleven Stack, region CBK

Incident Start: 08 Aug. 2023 19:45

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• Connectivity is restricted

Update: 20:00

The SDN of the CBK region had a sudden load spike, it was a short term disruption. We are investigating the root cause.