Past Incidents

Affected Components: Control Planes in Regions FES, DBL, CBK

Incident Start: 2024-12-09 18:30 UTC+01:00 (CET)

Description:

Some cluster control planes are not reachable.

Update (22:20 CET):

We have found a way to mitigate the issues temporarily and start applying the fix.

Update (22:50 CET):

We applied the fix everywhere. We don't see any broken clusters anymore.

Update 2024-12-10 12:30 UTC+01:00 (CET):

We identified the root cause: An unintended side-effect of an upgrade to a kube-proxy setting changed the proxy mode. This created iptables rules which were not cleaned up and through which traffic was dropped.

We are taking measures to prevent this in the future.

Affected Components: SysEleven Stack, region dus2

Incident Start: **2024-12-06 17:45 UTC+01:00 (CET)

Description:

• We are seeing some network connectivity issues in the region and are investigating.

Customer Impact:

• Connectivity is degraded for MetaKube Services in dus2
• Connectivity to Database as a Service in dus2 is degraded

Update: 2024-12-06 18:45 UTC+01:00 (CET)

• We identified an issue with one of our gateways and are working on a fix

Affected Components: SysEleven Stack API, region FES

Incident Start: 2024-12-04 19:07 UTC+01:00 (CET)

Incident End: 2024-12-04 20:10 UTC+01:00 (CET)

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Requests on OpenStack API may return an error code
• Spawning new virtual machines (VMs) or changing existing resources may fail

Update: 2024-12-04 20:00 UTC+01:00 (CET)

We identified the likely root cause and a fix is being applied.

Update: 2024-12-04 20:10 UTC+01:00 (CET)

OpenStack API is now working again as expected.

Affected Components: metakube control plane services, region ham1

Incident Start: 2024-11-29 11:00 UTC+01:00 (CET)

Incident End: 2024-11-29 13:40 UTC+01:00 (CET)

Description:

The metakube control plane services in ham1 can't be reached currently due to slow i/o

Customer Impact:

• Metakube services e.g. clusters in ham1 can't be reached

Customer Actions:

• Please inform us if you notice any irregularities

Update 13:23

The situation improved.

Affected Components: SysEleven IAM, regions DUS and HAM

Incident Start: 2024-11-28 12:00 UTC+01:00 (CET)

Description:

• We're currently investigating a service degradation in the SysEleven IAM. Inviting users to an organization is currently not possible.

Customer Impact:

• Inviting users to an organization is currently not possible.

UPDATE 2024-11-28 13:10 UTC+01:00 (CET)

The issue has been resolved and inviting users to organizations is possible again

Affected Components: MetaKube Control Plane Services, region FES

Incident Start: 2024-11-18 11:30 UTC+01:00 (CET)

Description:

Infrastructure hosting the MetaKube Control Plane Services has problems.

Customer Impact:

• MetaKube Control Plane might be slow or not answering

UPDATE 2024-11-18 12:30 UTC+01:00 (CET)

We have identified networking problems as the cause, currently working to resolve them.

UPDATE 2024-11-18 13:27 UTC+01:00 (CET)

We have increased conntrack table size on hardware nodes to avoid networking problems.

We continue to have issues with overloaded pods which we are working on.

UPDATE 2024-11-18 14:00 UTC+01:00 (CET)

We managed to get the overloaded pods running by isolating them on dedicated nodes and raising the resource limits. This stopped other issues as well.

We still need to investigate what caused the overloading of certain pods.

Incident is over.

Affected Components: SysEleven Stack, region CBK

Incident Start: 2024-11-12 22:35 Incident End: 2024-11-13 00:00

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• Connectivity is restricted

Update: 23:08

The announced maintenance is having a bigger impact than expected, we are investigating the situation

Update: 23:45

We were able to pin down the rootcause and prepare a fix to mitigate the problems

Update: 12:00

The network problems were mitigated. If you still encounter issues please contact us!