Past Incidents

Affected Components: Keycloak platform (login.syseleven.de), all regions

Scheduled Start: 21. Feb. 2023/07:00 am

Scheduled End: 21. Feb. 2023/08:00 am

State: SCHEDULED

Description:

we will perform a scheduled update of our Keycloak platform (login.syseleven.de) to version 20.0.3.

Customer Impact during the Maintenance: During the update, the service will need to be briefly paused and restarted.

Access to the following services may be limited during this maintenance:

• Metakube Dashboard
• Harbor (scr.syseleven.de)
• Gitlab (code.syseleven.de)
• Managed Grafana (*.grafana.syseleven.de)
• the Kubernetes API of customer clusters (if set up by the customer)

There will be no impact on running Metakube clusters, however, changes to clusters may not be possible if the above services are used.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: code.syseleven.de

Scheduled Start: 17. Feb. 2023/13:30 am

Scheduled End: 17. Feb. 2023/14:00 am

State: SCHEDULED

Description:

We are performing urgent and security-relevant maintenance work on code.syseleven.de, the system on which SysEleven provides Gitlab repositories for customers, between 1:30 - 2:00 pm. This involves updating the Gitlab software, which may lead to a brief interruption in the accessibility of the service.

Customer Impact during the Maintenance:

• Service code.syseleven.de will be briefly interrupted for up to 30 min

Customer Actions:

• No customer actions needed

Affected Components: Regular/emergency hotline

Incident Start: 10th February 2022/10:32 am Incident End: 10th February 2022/12:20 am

Description:

We are currently experiencing an outage at our external phone service provider. We have tightened the focus on our ticket system, please open a ticket instead of calling us in case of an emergency.

Update:

Our external service provider informed us that the incident has been mitigated. All systems are operational again.

Affected Components: SysEleven Stack in region CBK

Scheduled Start: 9 Feb 2023, 22:00 CET

Scheduled End: 10 Feb 2023, 01:00 CET

State: COMPLETED

Description:

We will roll out security updates for API servers and kernels.

Customer Impact during the Maintenance:

• Interruption of connectivity of virtual machines of up to 2 minutes
• OpenStack APIs may be unreachable for a short time (a few minutes)

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: code.syseleven.de

Scheduled Start: 07. Feb. 2023/10:00 pm

Scheduled End: 08. Feb. 2023/01:00 am

State: SCHEDULED

Description:

We will implement changes to the storage type used by GitLab. We do not expect any restrictions on the accessibility of code.syseleven.de for the duration of the maintenance work, but it will not be possible to upload images to the container registry during this time. Read access should be possible throughout.

Customer Impact during the Maintenance:

• read Access to code.syseleven.de will not be affected
• it will not be possible to upload images to the container registry

Customer Actions:

• No customer actions needed

Affected Components: SysEleven Stack Networking in region FES

Scheduled Start: 6 Feb 2023, 22:00 CET

Scheduled End: 7 Feb 2023, 01:00 CET

State: SCHEDULED

Description:

We will update the networking software (SDN).

Customer Impact during the Maintenance:

• interruption of connectivity of virtual machines of up to 2 minutes
• VPN connections could be interrupted for up to 2 minutes

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack, region DBL

Incident Start: 02 Feb. 2023/12:20

Description:

• Occurring errors are currently being investigated.
• It is currently not possible to create/attach/detach/delete cinder volumes

Customer Impact:

• API functionality is currently limited

Update:15:22

We identified the root cause and rolled provided a fix for the affected systems, we are now currently observing the situation

Affected Components: F5 Loadbalancer, BKI

Incident Start: 1st Feb. 2023/15:35 Incident End: 1st Feb. 2023/16:15

Description:

• Malfunction of our F5 Cluster in BKI caused by http2

Customer Impact:

• Affected customers will see downtimes and/or flapping availabilty for their websites
• Affected customers were notified via E-Mail
• HTTP2 functions will be unavailable

Update 15:42: A fix was applied to prevent the cluster from crashing, HTTP2 has been disabled for all customers on the F5 Loadbalancer in BKI

Update 15:55: The cluster is running stable with the deployed fix. HTTP2 is still disabled.

Update 16:15: The cluster is still stable, so we declare this incident resolved. HTTP2 will stay disabled until further notice.