Past Incidents

Affected Components: SysEleven Stack, identity service (Keystone)

Scheduled Start: 25. Jan. 2023/22:00

Scheduled End: 26. Jan. 2023/00:00

State: COMPLETED

Description:

• Scheduled maintenance of our SysEleven Stack identity service (Keystone)
• We will update Keystone to a new version during this maintenance

We do not expect customer impact during the maintenance.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update: 23:15 We completed the maintenance, observed a minor Keystone slowdown from 22:15 to 22:25.

Affected Components: SysEleven Stack, region CBK

Incident Start: 13 Jan 2023, 09:43 Incident End: 13 Jan 2023, 09:50

Description:

• Block storage I/O was slow in region CBK between 9:43 and 9:50.

Customer Impact:

• I/O performance for block storage of VMs (shared-storage main disks or volumes) was degraded which could lead to stalled system operations.

Affected Components: SysEleven Stack, identity service (Keystone)

Scheduled Start: 11. Jan. 2023/22:00

Scheduled End: 12. Jan. 2023/00:00

State: COMPLETED

Description:

• Scheduled maintenance of our SysEleven Stack identity service (Keystone)
• We will update Keystone to a new version during this maintenance

We do not expect customer impact during the maintenance.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update:

11:15 PM: We completed the maintenance, we did not encounter any obstacles.

Affected Components: NFS, Datacenter BLU

Scheduled Start: 9. Jan. 2023/11:00 pm

Scheduled End: 10. Jan. 2023/02:00 am

State: SCHEDULED

Description:

In order to ensure the stability of the NFS Service we will carry out maintenance work on the storage appliances at the location Berlin Lützowstraße (BLU) on January 10th 2023 from 11:00 pm to 2:00 am..

Customer Impact during the Maintenance:

• Scheduled maintenance of our NFS service (datacenter Lützowstrße)
• During the maintenance work, the cluster nodes will be individually removed from the network and restarted. Existing client connections are swivelled onto the active cluster nodes
• NFS Service in Datacenter Lützowstraße (Blue) might be interrupted due to connection timeouts

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack API, region DBL

Incident Start: 21th Dec. 2022/09:25 am Incident End: 21th Dec. 2022/09:41 am

Description:

• Performance of the SysEleven Stack API is affected

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 09:50 am

The API issues already resolved, we are still investigating what was causing them in the first place

Update: 10:25 am

We identified the root cause

Affected Components: metakube control plane, all regions

Incident Start: 20th of December 2022/12:00 pm

Incident End: 20th of December 2022/2:15 pm

Description:

We are currently experiencing a major outage of the kubernetes control plane across all regions.

Update 01:35 pm We have successfully identified the issue and a fix is currently rolled out.

Update 02:15 pm We have successfully fixed the issue

Customer Impact during the Maintenance:

Examples:

• Services will be impeded
• Connectivity is impaired

Customer Actions:

• Please inform us if you notice any further irregularities