Past Incidents

Affected Components: SysEleven Stack, identity service (Keystone)

Scheduled Start: 11. Jan. 2023/22:00

Scheduled End: 12. Jan. 2023/00:00

State: COMPLETED

Description:

• Scheduled maintenance of our SysEleven Stack identity service (Keystone)
• We will update Keystone to a new version during this maintenance

We do not expect customer impact during the maintenance.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update:

11:15 PM: We completed the maintenance, we did not encounter any obstacles.

Affected Components: NFS, Datacenter BLU

Scheduled Start: 9. Jan. 2023/11:00 pm

Scheduled End: 10. Jan. 2023/02:00 am

State: SCHEDULED

Description:

In order to ensure the stability of the NFS Service we will carry out maintenance work on the storage appliances at the location Berlin Lützowstraße (BLU) on January 10th 2023 from 11:00 pm to 2:00 am..

Customer Impact during the Maintenance:

• Scheduled maintenance of our NFS service (datacenter Lützowstrße)
• During the maintenance work, the cluster nodes will be individually removed from the network and restarted. Existing client connections are swivelled onto the active cluster nodes
• NFS Service in Datacenter Lützowstraße (Blue) might be interrupted due to connection timeouts

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack API, region DBL

Incident Start: 21th Dec. 2022/09:25 am Incident End: 21th Dec. 2022/09:41 am

Description:

• Performance of the SysEleven Stack API is affected

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 09:50 am

The API issues already resolved, we are still investigating what was causing them in the first place

Update: 10:25 am

We identified the root cause

Affected Components: metakube control plane, all regions

Incident Start: 20th of December 2022/12:00 pm

Incident End: 20th of December 2022/2:15 pm

Description:

We are currently experiencing a major outage of the kubernetes control plane across all regions.

Update 01:35 pm We have successfully identified the issue and a fix is currently rolled out.

Update 02:15 pm We have successfully fixed the issue

Customer Impact during the Maintenance:

Examples:

• Services will be impeded
• Connectivity is impaired

Customer Actions:

• Please inform us if you notice any further irregularities

Affected Components: Hardware node failure, region BKI

Incident Start: 18th Dec. 2022/11:25 am Incident End: 18th Dec. 2022/11:55 am

Description:

• Malfunction of a hardware node
• Restart of the hardware node was necessary
• Stage only, no production instances affected

Customer Impact: Stage only, no production instances affected

• During the period of restart, there will be a short interruption in the availability of the systems.
• Affected customers were notified via E-Mail.
• Please check the affected systems for their full functionality.

Affected Components: SysEleven Stack, region DBL

Incident Start: 12th Dec. 2022/13:40 Incident End: 12th Dec. 2022/14:10

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• Connectivity is restricted

Update:14:10

We were able to pin down the root cause and rolled out a change to fix the problem

Update:15:00

We are still in the process of cleaning up follow up problems for Octavia load balancers

Update:15:47

The Octavia load balancers are repaired now too. With that all issues are fixed.