Past Incidents

Affected Components: SysEleven Stack in region FES

Scheduled Start: 17 Oct 2023, 22:00 CEST

Scheduled End: 18 Oct 2023, 02:00 CEST

State: ABORTED

Description:

This is a maintenance announcement for region FES. We will update the operating system on the controller servers.

Customer Impact during the Maintenance:

• OpenStack API is expected to be down.

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack in region CBK

Scheduled Start: 19 Sept 2023, 22:00 CEST

Scheduled End: 20 Sept 2023, 02:00 CEST

State: COMPLETED

Description:

This is a maintenance announcement for region CBK. We will update the operating system on the controller servers.

Customer Impact during the Maintenance:

• OpenStack API is expected to be down.

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack in region DBL

Scheduled Start: 12 Sept 2023, 22:00 CEST

Scheduled End: 13 Sept 2023, 02:00 CEST

State: COMPLETED

Description:

This is a maintenance announcement for region DBL. We will update the operating system on the controller servers.

Customer Impact during the Maintenance:

• OpenStack API is expected to be down.

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: MetaKube Platform, all regions

Scheduled Start: 05. Sept. 2023 / 18:00 CEST

Scheduled End: 05. Sept. 2023 / 21:00 CEST

State: DONE

Description:

• Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the Maintenance:

It is possible that various functions of the control plane are unavailable for brief periods.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack API, region DBL

Incident Start: 31th Aug. 2023/14:15

Incident End: 31th Aug. 2023/15:00

Description:

• We are seeing problems with the Cinder Volume services as well as Nova API services. The Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update:15:00

We brought back the failed services, control over resources is back, we are observing the situation and search for the root cause

Incident Start: 24th Aug. 2023/10:10 am

We saw a problem with the network gateway for our cloud region cbk. We are still monitoring and evaluating what caused the issues.

Update: 24th Aug. 2023/10:21 pm

We consider the incident as resolved. Please contact us if you still experience problems.

Affected Components: MetaKube control planes (all regions)

Scheduled Start: 22. Aug. 2023/06:00 pm Scheduled End: 22. Aug. 2023/09:00 pm

State: COMPLETED

Description:

We will carry out an important update of our infrastructure.

Customer Impact during the Maintenance:

It is possible that various functions of the control plane are unavailable for brief periods.

Incident Start: 20th Aug. 2023/05:58 pm

We saw a problem with the network gateway for our cloud region cbk. We are still monitoring and evaluating what caused the issues.

Update: 20th Aug. 2023/06:13 pm

We consider the incident as resolved. Please contact us if you still experience problems.

[We have edited the wording for clarity on August 22nd (Service Management)]

Incident Start: 19th Aug. 2023/01:20 pm

We saw a problem with the network gateway for our cloud region cbk. We are still monitoring and evaluating what caused the issues.

Update: 19th Aug. 2023/01:30 pm

We consider the incident as resolved. Please contact us if you still experience problems.

[We have edited the wording for clarity on August 22nd (Service Management)]

Affected Components: SysEleven Stack, region CBK

Incident Start: 16th Aug. 2023/23:37

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• Connectivity is restricted

Update: 17th Aug. 2023/07:25 am

We identified a problem with the gateway and right now no external traffic is possible, we are still working on the restoration

Update: 17th Aug. 2023/07:55 am

We have been observing network problems in the Cloud Region CBK in relation with the underlying SDN (Midonet). So far, we have not been able to restore the functionality of the Cloud Region.

Systems in the CBK cloud region are currently not accessible from public networks, which means that it is also not possible to transfer data to and from the systems. Authentication to MetaKube clusters and individual other systems (SysEleven Login, Open ID Connect, code.syseleven.de) is currently restricted. The systems in the CBK region are basically running, so we do not currently expect any data loss.

We are working at full speed to resolve the issues.

Update: 17th Aug. 2023/08:35 am

Cluster creation in all regions is currently taking longer than usual.

Update: 17th Aug. 2023/08:45 am

Network Connectivity in region CBK is slowly recovering

Update: 17th Aug. 2023/09:10 am

The cause of the Cloud Region CBK network issues has been identified and resolved, Cloud Region CBK functionality has been restored to most systems, or should be shortly.

If you continue to observe problems, please contact us.

Update: 17th Aug. 2023/09:30 am

We are running scripts to automatically redeploy loadbalancers in the cloud region CBK. If you experience further network related problems this might be related and should be fixed shortly.

Update: 17th Aug. 2023/10:40 am

Some loadbalancers are still causing issues, we are looking into the issues, while most loadbalancers should be back online and functional.

Update: 17th Aug. 2023/11:10 am

We consider the incident as resolved. Please contact us if you still experience problems.

Update: 21th Aug. 2023/08:15 pm

Reason for Outage (RfO)

Affected Components: SysEleven Stack Networking in region CBK

Scheduled Start: 15 Aug 2023, 22:00 CEST

Scheduled End: 16 Aug 2023, 01:00 CEST

State: COMPLETED

Description:

We will perform an emergency maintenance on networking gateways.

Customer Impact during the Maintenance:

• Interruption of external connectivity of virtual machines of up to 2 minutes
• VPN connections could be interrupted for up to 2 minutes

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack LBaaS API (Octavia), region DBL

Incident Start: 9 Aug 2023 15:00

Description:

• Creating octavia load balancers failed in region DBL. Updating existing load balancers brought them into an immutable state ("PENDING_UPDATE") after the update failed.

Customer Impact:

• Creating or updating octavia load balancers failed

Update: 18:30

Creating and updating load balancers is possible again. Some load balancers that became stuck in the PENDING_UPDATE state will be repaired now.

Update: 19:45

The load balancers that were stuck in PENDING_UPDATE have been repaired. The octavia LBaaS service is now fully functional again.