Past Incidents

Affected Components: SysEleven Stack in region CBK

Scheduled Start: 9 Feb 2023, 22:00 CET

Scheduled End: 10 Feb 2023, 01:00 CET

State: COMPLETED

Description:

We will roll out security updates for API servers and kernels.

Customer Impact during the Maintenance:

• Interruption of connectivity of virtual machines of up to 2 minutes
• OpenStack APIs may be unreachable for a short time (a few minutes)

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: code.syseleven.de

Scheduled Start: 07. Feb. 2023/10:00 pm

Scheduled End: 08. Feb. 2023/01:00 am

State: SCHEDULED

Description:

We will implement changes to the storage type used by GitLab. We do not expect any restrictions on the accessibility of code.syseleven.de for the duration of the maintenance work, but it will not be possible to upload images to the container registry during this time. Read access should be possible throughout.

Customer Impact during the Maintenance:

• read Access to code.syseleven.de will not be affected
• it will not be possible to upload images to the container registry

Customer Actions:

• No customer actions needed

Affected Components: SysEleven Stack Networking in region FES

Scheduled Start: 6 Feb 2023, 22:00 CET

Scheduled End: 7 Feb 2023, 01:00 CET

State: SCHEDULED

Description:

We will update the networking software (SDN).

Customer Impact during the Maintenance:

• interruption of connectivity of virtual machines of up to 2 minutes
• VPN connections could be interrupted for up to 2 minutes

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack, region DBL

Incident Start: 02 Feb. 2023/12:20

Description:

• Occurring errors are currently being investigated.
• It is currently not possible to create/attach/detach/delete cinder volumes

Customer Impact:

• API functionality is currently limited

Update:15:22

We identified the root cause and rolled provided a fix for the affected systems, we are now currently observing the situation

Affected Components: F5 Loadbalancer, BKI

Incident Start: 1st Feb. 2023/15:35 Incident End: 1st Feb. 2023/16:15

Description:

• Malfunction of our F5 Cluster in BKI caused by http2

Customer Impact:

• Affected customers will see downtimes and/or flapping availabilty for their websites
• Affected customers were notified via E-Mail
• HTTP2 functions will be unavailable

Update 15:42: A fix was applied to prevent the cluster from crashing, HTTP2 has been disabled for all customers on the F5 Loadbalancer in BKI

Update 15:55: The cluster is running stable with the deployed fix. HTTP2 is still disabled.

Update 16:15: The cluster is still stable, so we declare this incident resolved. HTTP2 will stay disabled until further notice.

Affected Components: SysEleven Stack, identity service (Keystone)

Scheduled Start: 25. Jan. 2023/22:00

Scheduled End: 26. Jan. 2023/00:00

State: COMPLETED

Description:

• Scheduled maintenance of our SysEleven Stack identity service (Keystone)
• We will update Keystone to a new version during this maintenance

We do not expect customer impact during the maintenance.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update: 23:15 We completed the maintenance, observed a minor Keystone slowdown from 22:15 to 22:25.

Affected Components: SysEleven Stack, region CBK

Incident Start: 13 Jan 2023, 09:43 Incident End: 13 Jan 2023, 09:50

Description:

• Block storage I/O was slow in region CBK between 9:43 and 9:50.

Customer Impact:

• I/O performance for block storage of VMs (shared-storage main disks or volumes) was degraded which could lead to stalled system operations.