Past Incidents

Affected Components: SysEleven Stack API, region FES

Incident Start: 02nd February. 2023/11:00 Incident End: 02nd February. 2023/12:30

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update:11:45

We identified the issue and fixed it. We are currently watching the situation and repairing follow up problems.

Update:12:05

We see additional problems and failing requests again, we are further investigating

Update:16:15

The observed API problems were fixed around 12:30, we kept an eye on the situation until 16:15

Affected Components: SysEleven Login, all regions

Incident start: 21. Feb. 2022/08:30 am Incident end: 21. Feb. 2022/09:03 am

Description:

We are currently experiencing a major outage of our login services and are working on mitigating the situation.

Update: The situation has been resolved.

Affected Components: Regular/emergency hotline

Incident Start: 10th February 2022/10:32 am Incident End: 10th February 2022/12:20 am

Description:

We are currently experiencing an outage at our external phone service provider. We have tightened the focus on our ticket system, please open a ticket instead of calling us in case of an emergency.

Update:

Our external service provider informed us that the incident has been mitigated. All systems are operational again.

Affected Components: https://code.syseleven.de

Incident Start: 8th Feb. 2023/23:45 Incident End: **9th Feb. 2023/06:55"

Description:

• unexpected problems during planned maintenance
• rollback failed
• restore of backup data was necessary

Customer Impact:

• code.syseleven.de unavailable until restore is completed
• loss of all changes since 18:00 (this affects everything such as issues, commits, pipelines, artifacts and registry images)

Update 06:55:

• Backup was restored successfully
• site is available again

Affected Components: SysEleven Metakube control plane services, region FES

Incident Start: 6th Sept. 2022/23:35

Description:

• Accessibility of the SysEleven Metakube API is unstable

Customer Impact:

• Cluster apiserver access disrupted or slow

Update 02:10:

• We found the issue and were able to mitigate the problem. All services recovered and are reachable again.

Affected Components: SysEleven Stack, region DBL

Incident Start: 02 Feb. 2023/12:20

Description:

• Occurring errors are currently being investigated.
• It is currently not possible to create/attach/detach/delete cinder volumes

Customer Impact:

• API functionality is currently limited

Update:15:22

We identified the root cause and rolled provided a fix for the affected systems, we are now currently observing the situation

Affected Components: F5 Loadbalancer, BKI

Incident Start: 1st Feb. 2023/15:35 Incident End: 1st Feb. 2023/16:15

Description:

• Malfunction of our F5 Cluster in BKI caused by http2

Customer Impact:

• Affected customers will see downtimes and/or flapping availabilty for their websites
• Affected customers were notified via E-Mail
• HTTP2 functions will be unavailable

Update 15:42: A fix was applied to prevent the cluster from crashing, HTTP2 has been disabled for all customers on the F5 Loadbalancer in BKI

Update 15:55: The cluster is running stable with the deployed fix. HTTP2 is still disabled.

Update 16:15: The cluster is still stable, so we declare this incident resolved. HTTP2 will stay disabled until further notice.