SysEleven Status and Incidents

INCIDENT: PVC Failure of a hardware node, region BKI

Tue, 01 Oct 2024 05:15:00 +0000

Affected Components: Hardware node failure, region XXX

Incident Start: 2024-10-01 09:15 UTC+02:00 (CEST) Incident End: 2024-10-01 09:36 UTC+02:00 (CEST)

Description:

Malfunction of a hardware node
Restart of the hardware node is necessary

Customer Impact:

During the period of restart, there will be a short interruption in the availability of the systems.
Affected customers were notified via E-Mail.
Please check the affected systems for their full functionality.

INCIDENT: SysEleven STACK Network performance issues, region DBL

Tue, 01 Oct 2024 10:17:00 +0000

Affected Components: SysEleven Stack Network, region DBL

Incident Start: 2024-10-01 14:17 UTC+02:00 (CEST)

Incident End: 2024-10-01 14:34 UTC+02:00 (CEST)

Description:

At the moment, we are facing issues with the Network in Region DBL.

Customer Impact:

Network performance issue

Update: 2024-10-01 14:34 UTC+02:00 (CEST)

The incident is over, and all services are operational.

INCIDENT: SysEleven STACK issues in region HAM1

Thu, 10 Oct 2024 19:55:00 +0000

Affected Components: SysEleven Stack, region HAM1

Incident Start: 2024-10-10 23:55 Incident End: 2024-10-11 00:00

Description:

Occurring errors were investigated.

Customer Impact:

Connectivity was restricted

Update: 01:00

We can observe further short term issues with the HAM1 region connectivity, the provider is aware of the issues and is currently investigating the situation

Update: 01:30

The network provider is proceeding with a network maintenance until 05:00, we are on standby

INCIDENT: SysEleven STACK API issues

Wed, 30 Oct 2024 07:30:00 +0000

Affected Components: SysEleven Stack API

Incident Start: 2024-10-30 09:30 CET

Incident End: 2024-10-30 12:05 CET

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 10:40

We are still investigating the situation and are in contact with our external network provider to further analyze the problems.

Update: 11:30

The issue has been identified, we are waiting for our external network provider to further fix the situation.

Update: 12:05

The issue has been resolved.

INCIDENT: SysEleven STACK Object Storage issues, region DBL

Tue, 12 Nov 2024 15:45:00 +0000

Affected Components: SysEleven Stack Object Storage, region DBL

Incident Start: 2024-11-12 17:45 UTC+01:00 (CET)

Incident End: 2024-11-12 18:40 UTC+01:00 (CET)

Description:

At the moment we are facing issues with the Object Storage in Region DBL.

Customer Impact:

Writing or reading of objects maybe restricted.

Update: 2024-11-12 18:40 UTC+01:00 (CET)

We mitigated the problem and do further investigation

INCIDENT: SysEleven STACK issues in region CBK

Tue, 12 Nov 2024 20:35:00 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2024-11-12 22:35 Incident End: 2024-11-13 00:00

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

Update: 23:08

The announced maintenance is having a bigger impact than expected, we are investigating the situation

Update: 23:45

We were able to pin down the rootcause and prepare a fix to mitigate the problems

Update: 12:00

The network problems were mitigated. If you still encounter issues please contact us!

INCIDENT: Partial outage of MetaKube Control Plane Services Region in region FES

Mon, 18 Nov 2024 09:30:00 +0000

Affected Components: MetaKube Control Plane Services, region FES

Incident Start: 2024-11-18 11:30 UTC+01:00 (CET)

Description:

Infrastructure hosting the MetaKube Control Plane Services has problems.

Customer Impact:

MetaKube Control Plane might be slow or not answering

UPDATE 2024-11-18 12:30 UTC+01:00 (CET)

We have identified networking problems as the cause, currently working to resolve them.

UPDATE 2024-11-18 13:27 UTC+01:00 (CET)

We have increased conntrack table size on hardware nodes to avoid networking problems.

We continue to have issues with overloaded pods which we are working on.

UPDATE 2024-11-18 14:00 UTC+01:00 (CET)

We managed to get the overloaded pods running by isolating them on dedicated nodes and raising the resource limits. This stopped other issues as well.

We still need to investigate what caused the overloading of certain pods.

Incident is over.

INCIDENT: Partial degradation of SysEleven IAM services

Thu, 28 Nov 2024 10:00:00 +0000

Affected Components: SysEleven IAM, regions DUS and HAM

Incident Start: 2024-11-28 12:00 UTC+01:00 (CET)

Description:

We're currently investigating a service degradation in the SysEleven IAM. Inviting users to an organization is currently not possible.

Customer Impact:

Inviting users to an organization is currently not possible.

UPDATE 2024-11-28 13:10 UTC+01:00 (CET)

The issue has been resolved and inviting users to organizations is possible again

INCIDENT: major outage of metakube control plane services in ham1

Fri, 29 Nov 2024 09:00:00 +0000

Affected Components: metakube control plane services, region ham1

Incident Start: 2024-11-29 11:00 UTC+01:00 (CET)

Incident End: 2024-11-29 13:40 UTC+01:00 (CET)

Description:

The metakube control plane services in ham1 can't be reached currently due to slow i/o

Customer Impact:

Metakube services e.g. clusters in ham1 can't be reached

Customer Actions:

Please inform us if you notice any irregularities

Update 13:23

The situation improved.

INCIDENT: SysEleven STACK Storage issues, region HAM1

Fri, 29 Nov 2024 09:00:00 +0000

Affected Components: SysEleven Stack, Storage, region HAM1

Incident Start: 2024-11-29 11:00 UTC+01:00 (CET)

Incident End: 2024-11-29 13:40 UTC+01:00 (CET)

Description:

We are facing issues with the distributed file system, a core component of the SysEleven Stack.

Customer Impact:

Starting of virtual machines (VMs) partially not possible.
Writing Access to volumes (VM disks) maybe restricted.

Update 13:23

The situation improved, storage access latencies are back to normal.

INCIDENT: SysEleven STACK API issues, region FES

Wed, 04 Dec 2024 17:07:00 +0000

Affected Components: SysEleven Stack API, region FES

Incident Start: 2024-12-04 19:07 UTC+01:00 (CET)

Incident End: 2024-12-04 20:10 UTC+01:00 (CET)

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Requests on OpenStack API may return an error code
Spawning new virtual machines (VMs) or changing existing resources may fail

Update: 2024-12-04 20:00 UTC+01:00 (CET)

We identified the likely root cause and a fix is being applied.

Update: 2024-12-04 20:10 UTC+01:00 (CET)

OpenStack API is now working again as expected.

INCIDENT: SysEleven STACK issues in region dus2

Fri, 06 Dec 2024 15:45:00 +0000

Affected Components: SysEleven Stack, region dus2

Incident Start: **2024-12-06 17:45 UTC+01:00 (CET)

Description:

We are seeing some network connectivity issues in the region and are investigating.

Customer Impact:

Connectivity is degraded for MetaKube Services in dus2
Connectivity to Database as a Service in dus2 is degraded

Update: 2024-12-06 18:45 UTC+01:00 (CET)

We identified an issue with one of our gateways and are working on a fix

INCIDENT: Partial outage of Control Planes in Regions FES, DBL, CBK

Mon, 09 Dec 2024 16:30:00 +0000

Affected Components: Control Planes in Regions FES, DBL, CBK

Incident Start: 2024-12-09 18:30 UTC+01:00 (CET)

Description:

Some cluster control planes are not reachable.

Update (22:20 CET):

We have found a way to mitigate the issues temporarily and start applying the fix.

Update (22:50 CET):

We applied the fix everywhere. We don't see any broken clusters anymore.

Update 2024-12-10 12:30 UTC+01:00 (CET):

We identified the root cause: An unintended side-effect of an upgrade to a kube-proxy setting changed the proxy mode. This created iptables rules which were not cleaned up and through which traffic was dropped.

We are taking measures to prevent this in the future.

INCIDENT: SysEleven STACK network issues in region DBL

Fri, 13 Dec 2024 12:09:00 +0000

Affected Components: SysEleven Stack, region DBL

Incident Start: 2024-12-13 14:09 UTC+01:00 (CET)

Incident End: 2024-12-13 16:09 UTC+01:00 (CET)

Description:

DBL network has performance issue

Customer Impact:

Network performance degradation

Update: 2024-12-13 15:17 UTC+01:00 (CET)

Situation is back to normal.

Update: 2024-12-13 15:57 UTC+01:00 (CET)

We notice performance degradation again.

Update: 2024-12-13 16:09 UTC+01:00 (CET)

Situation is back to normal.

INCIDENT: SysEleven STACK issues in region DBL

Tue, 17 Dec 2024 22:40:00 +0000

Affected Components: SysEleven Stack, region DBL

Incident Start: 2024-18-12 00:47 Incident Start: 2024-18-12 01:27

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

Update: 2024-18-12 01:27

We could see problems with cross region connectivity outgoing from the DBL region, between 00:35 - 01:10, at the moment traffic seems to have normalized again, we are still investigating

Update: 2024-18-12 02:00

Device causing the network issues was identified, root cause will be further investigated

INCIDENT: SysEleven STACK issues in region CBK

Mon, 06 Jan 2025 08:35:12 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-06-01 08:18

Incident Start: 2025-06-01 08:50

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

INCIDENT: SysEleven STACK issues in region CBK

Mon, 06 Jan 2025 10:14:32 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-06-01 10:00

Incident End: 2025-06-01 10:30

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

INCIDENT: SysEleven STACK issues in region CBK

Tue, 07 Jan 2025 09:25:38 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-01-07 09:16

Incident End: 2025-01-07 09:45

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

Update : 09:45

Situation stabilized again, we are investigating the situation at the moment

INCIDENT: minor outage of Database as a Service

Tue, 21 Jan 2025 14:41:37 +0000

Affected Components: Database as a Service, all regions

Incident Start: 2025-01-21 10:30 UTC+01:00 (CET) Incident End: 2025-01-21 14:45 UTC+01:00 (CET)

Description:

Database as a Service is currently only available via the API and terraform, the UI is not working

Update 2025-01-21 14:45

We fixed the underlying issue, therefore Database as a Service is also working again in the UI

INCIDENT: MetaKube Control Plane issues, region FES

Tue, 11 Feb 2025 08:04:01 +0000

Affected Components: MetaKube Control Planes, region FES

Incident Start: 2025-02-11 06:00 UTC+01:00 (CET)

State: Resolved

Description:

Accessibility of the MetaKube API is not ensured.
After a scheduled maintenance to the network in FES, the MetaKube control cluster (which is hosting the customer control planes) has problems reaching DNS. This is causing issues to the customer control planes.
This also affects Database as a Service and Observability as a Service
All times below are CET

UPDATE 2025-02-13 12:20 - We consider all service disruptions of the incident to be mitigated - Although we are not expecting any more service disruptions, we are still watching all systems closly

Previous Updates in reverse chronological order

UPDATE 2025-02-11 10:00

Still investigating the DNS issue. We sent out a notifier to all potentially affected customers.

UPDATE 2025-02-11 11:15 - We have used the time since the last update to narrow down the root cause of the incident. We excluded some possibilites but did not find the root-cause. We are now preparing a partial rollback to downgrade the SDN again.

UPDATE 2025-02-11 12:05 - We completed a partial OVN/SDN downgrade, however this has not yet resolved the incident. - We are investigating further

UPDATE 2025-02-11 12:25 - We’re exploring further downgrade approaches (previous rollbacks were, as announced, partial) and are in parallel investigating further.

UPDATE 2025-02-11 13:00 - As we originally updated the SDN due to a critical security gap, it was decided that we will not perform a full OVN/SDN rollback to the initial state. - We have now activated several teams who will be developing and evaluating different solutions until 1.30 pm. An update on how we proceed will follow then.

UPDATE 2025-02-11 13:50 - Our Teams will continue developing and evaluating solutions in break out session as there are further leads but no breakthrough, yet. - In parallel we are preparing a failover for IAM and Alloy to DUS/HAM

UPDATE 2025-02-11 15:30 - Our teams investigation in SDN traffic loss is ongoing - Our teams continue developing and evaluating solutions and possible workarounds - In parallel we are evaluating a rebuild of the SDN (software defined network)

UPDATE 2025-02-11 17:47 - Part of the services are still not functional - We are still working hard to resolve the issues but we will roll back the update of the SDN if no progress is made. - The planned maintenance period begins today, 11 February 2025, at 23:00 and is expected to last until around 06:00 CET on 12 February 2025, during which time there may be repeated interruptions to services. - The maintenance is also announced via notifier. You will get an info of the end of the maintenance also via notifier.

UPDATE 2025-02-11 20:15 - IAM, Alloy and Observability as a Service are restored to full functionality - The Database as a Service API has also been restored, but the API still has some issues which are related to the wider SDN problem. The Databases themselves were at no point affected by the incident.

UPDATE 2025-02-11 23:00 - The situation with the API improved. We will continue watching it.

UPDATE 2025-02-12 10:15 - The previous maintenance work did not achieve the desired success. - Workarounds have been implemented, so operations should be able to continue without disruptions. - Maintainance work will continue during the upcoming night to fully resolve the incident.

UPDATE 2025-02-12 14:55 - We scheduled another maintenance window for this night, February 12th, from 11:00 PM to 6:00 AM the following day. - During this maintenance window, there may be brief interruptions or limited availability of certain services. - The goal is the complete resolution of the incident caused by Monday's update. - An RfO will be available in our Helpdesk after the incident is mitigated.

UPDATE 2025-02-13 15:30 - Incident is resolved