Past Incidents

Affected Components: SysEleven Stack, region DBL

Incident Start: 2024-18-12 00:47 Incident Start: 2024-18-12 01:27

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• Connectivity is restricted

Update: 2024-18-12 01:27

• We could see problems with cross region connectivity outgoing from the DBL region, between 00:35 - 01:10, at the moment traffic seems to have normalized again, we are still investigating

Update: 2024-18-12 02:00

• Device causing the network issues was identified, root cause will be further investigated

Affected Components: SysEleven Stack, region DBL

Incident Start: 2024-12-13 14:09 UTC+01:00 (CET)

Incident End: 2024-12-13 16:09 UTC+01:00 (CET)

Description:

• DBL network has performance issue

Customer Impact:

• Network performance degradation

Update: 2024-12-13 15:17 UTC+01:00 (CET)

Situation is back to normal.

Update: 2024-12-13 15:57 UTC+01:00 (CET)

We notice performance degradation again.

Update: 2024-12-13 16:09 UTC+01:00 (CET)

Situation is back to normal.

Affected Components: Control Planes in Regions FES, DBL, CBK

Incident Start: 2024-12-09 18:30 UTC+01:00 (CET)

Description:

Some cluster control planes are not reachable.

Update (22:20 CET):

We have found a way to mitigate the issues temporarily and start applying the fix.

Update (22:50 CET):

We applied the fix everywhere. We don't see any broken clusters anymore.

Update 2024-12-10 12:30 UTC+01:00 (CET):

We identified the root cause: An unintended side-effect of an upgrade to a kube-proxy setting changed the proxy mode. This created iptables rules which were not cleaned up and through which traffic was dropped.

We are taking measures to prevent this in the future.

Affected Components: SysEleven Stack, region dus2

Incident Start: **2024-12-06 17:45 UTC+01:00 (CET)

Description:

• We are seeing some network connectivity issues in the region and are investigating.

Customer Impact:

• Connectivity is degraded for MetaKube Services in dus2
• Connectivity to Database as a Service in dus2 is degraded

Update: 2024-12-06 18:45 UTC+01:00 (CET)

• We identified an issue with one of our gateways and are working on a fix

Affected Components: SysEleven Stack API, region FES

Incident Start: 2024-12-04 19:07 UTC+01:00 (CET)

Incident End: 2024-12-04 20:10 UTC+01:00 (CET)

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Requests on OpenStack API may return an error code
• Spawning new virtual machines (VMs) or changing existing resources may fail

Update: 2024-12-04 20:00 UTC+01:00 (CET)

We identified the likely root cause and a fix is being applied.

Update: 2024-12-04 20:10 UTC+01:00 (CET)

OpenStack API is now working again as expected.

Affected Components: metakube control plane services, region ham1

Incident Start: 2024-11-29 11:00 UTC+01:00 (CET)

Incident End: 2024-11-29 13:40 UTC+01:00 (CET)

Description:

The metakube control plane services in ham1 can't be reached currently due to slow i/o

Customer Impact:

• Metakube services e.g. clusters in ham1 can't be reached

Customer Actions:

• Please inform us if you notice any irregularities

Update 13:23

The situation improved.

Affected Components: SysEleven IAM, regions DUS and HAM

Incident Start: 2024-11-28 12:00 UTC+01:00 (CET)

Description:

• We're currently investigating a service degradation in the SysEleven IAM. Inviting users to an organization is currently not possible.

Customer Impact:

• Inviting users to an organization is currently not possible.

UPDATE 2024-11-28 13:10 UTC+01:00 (CET)

The issue has been resolved and inviting users to organizations is possible again