SysEleven Status and Incidents

INCIDENT: SysEleven STACK issues in region FES

Sat, 08 Mar 2025 14:29:22 +0000

Affected Components: SysEleven Stack, region FES

Incident Start: 2025-03-08 13:00

Incident End: 2025-03-08 15:30

Description:

Occurring errors are currently being investigated.

Customer Impact:

IPv6 Connectivity is restricted

Update: 16:25

Flaky ipv6 connectivity was repaired around 15:30, a mismatch in configuration was identified

INCIDENT: SysEleven STACK API issues, region CBK

Thu, 13 Mar 2025 14:43:01 +0000

Affected Components: SysEleven Stack API, region CBK

Incident Start: 2025-03-13 14:30 Incident End: 2025-03-13 17:15

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Spawning new virtual machines (VMs) or changing existing resources is not possible.

INCIDENT: Partial degradation of SysEleven IAM services

Mon, 17 Mar 2025 10:39:16 +0000

We're currently investigating an issue in our IAM systems that prevents creation and deletion of projects.

INCIDENT: all platforms

Fri, 11 Apr 2025 00:22:15 +0000

Affected Components: All regions

Incident Start: 2025-04-10 23:40 (CETS)

Incident End: 2025-04-11 01:08 (CETS)

Description:

We have major outage in all regions

Update: 01:01 am

Issue identified and we are working on it.

Update: 01:08 am

Issue is resolved and we are whatching it.

INCIDENT: SysEleven STACK Object Storage issues, region FES, HAM1, DUS2

Mon, 14 Apr 2025 16:51:56 +0000

Affected Components: SysEleven Stack Object Storage, region FES, DUS2, HAM1

Incident Start: 2025-04-14 16:26 UTC+01:00 (CET)

Description:

At the moment we are facing issues with the Object Storage in Region FES, DUS2, HAM1.

Customer Impact:

Some objects might be corrupted regarding latest known issues in Ceph. https://docs.ceph.com/en/latest/releases/squid/#v19-2-2-squid
We are investigating the situation and working on updates to mitigate the issues

Update: 18:40

The Ceph S3 endpoint of the DUS2 region is currently being updated, after successful testing we will proceed with the HAM1 and FES region

Update: 19:45

The update for the DUS2 region went through without problems, we are proceeding to update HAM1 and the FES region
Affected customers will be contacted individually once we got a complete overview of potential affected buckets and objects

Update: 21:10

Update of HAM1 region also finished, FES region is in progress

Update: 22:52

Update of FES region is still ongoing

Update: 15-04-2025 09:36

Update of FES region is still ongoing and everything works as intended

Update: 16:40

Update of FES region is expected to be finished to the end of the business day

Update: 16-04-2025 03:08

Update of FES is over

INCIDENT: SysEleven STACK issues in region DBL

Thu, 17 Apr 2025 10:11:22 +0000

Affected Components: SysEleven Stack, region DBL

Incident Start: 2025-04-17 10:00

Incident End: 2025-04-17 10:45

Description:

Occurring errors are currently being investigated.

Update [2025-04-17 10:28 am]: Alloy (GUI) & API Authentication HAM & DUS, Metakube API in DBL & CBK are also affected

Update [2025-04-17 10:40 am]: DBL Cloud Gateways have been identified as possible source of malfunction, we are bringing them back to function, first services are back online, next Update at 11:00 am

Update [2025-04-17 11:00 am]: All services are up and running, we are still watching all environments, incident is declared as ended

INCIDENT: SysEleven STACK issues in region CBK

Fri, 25 Apr 2025 17:18:12 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)

Description:

Occurring errors are currently being investigated.

Customer Impact:

We are expirencing a partial outage of the Network where some connection might get dropped.

INCIDENT: SysEleven Managed Hosting Network partial outage, region BLU1

Fri, 25 Apr 2025 17:46:54 +0000

Affected Components: SysEleven Managed Hosting Network partial outage, region BLU1

Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)

Description: - Occurring errors are currently being investigated.

Customer Impact: - We are expirencing a partial outage of the Network where some connection might get dropped.

INCIDENT: SysEleven STACK issues in region CBK (Network)

Tue, 29 Apr 2025 18:01:27 +0000

Affected Components: SysEleven Stack Network, region CBK

Incident Start: 2025-04-29 17:30 UTC+02:00 (CEST) Incident End: 2025-04-29 17:50 UTC+02:00 (CEST)

Description:

Network connectivity to cloud region CBK was impaired

Customer Impact:

Connectivity is restricted

Update: 17:59

Network connectivity issues were resolved around 17:50

INCIDENT: Partial outage of Metakube control plane in DUS

Wed, 30 Apr 2025 16:10:27 +0000

Affected Components: MetaKube Control Plane, region DUS

Incident Start: 2025-04-30 16:00 UTC+02:00

Description:

The loadbalancer for the kubernetes apiserver does not properly work for some clusters.

Customer Impact: - Potential connection loss to the apiserver

Update: 2025-04-30 16:30 UTC+02:00

We have recovered the affected clusters but haven't found the root cause. Right now we can't see any other affected clusters but will continue to monitor the situation.

Update: 2025-04-30 16:52 UTC+02:00 We have identified and fixed the root-cause of the incident. All clusters are working properly again.

INCIDENT: Outage of Metakube control plane in HAM

Thu, 08 May 2025 15:20:50 +0000

Affected Components: MetaKube Control Plane, region HAM

Incident Start: 2025-05-08 14:43 UTC+02:00

Description:

The loadbalancer for the kubernetes apiserver does not properly work for some clusters.

Customer Impact: - Potential connection loss to the apiserver

Update: 2025-05-08 15:02 UTC+02:00 We have identified and fixed the root-cause of the incident. All clusters are fully operational again.

INCIDENT: SysEleven STACK issues in region CBK (Network)

Thu, 08 May 2025 17:09:28 +0000

Affected Components: SysEleven Stack Network, region CBK

Incident Start: 2025-05-08 16:45 UTC+02:00 (CEST)

Incident End: 2025-05-08 17:00 UTC+02:00 (CEST)

Description:

Network connectivity to cloud region CBK was impaired

Customer Impact:

Connectivity is restricted

Network connectivity issues were resolved around 17:00

INCIDENT: Outage of Metakube cluster creation in HAM and DUS

Fri, 09 May 2025 10:36:19 +0000

Affected Components: SysEleven MetaKube, region HAM and DUS

Incident Start: 2025-05-09 09:30 UTC+02:00 (CEST)

Description: - The cluster creation in DUS and HAM is currently not working. We are currently investigating.

Customer Impact: - No new clusters can be created

Update: 2025-05-09 11:23 UTC+02:00 (CEST) - We are continuing to work on rectifying the situation

Update: 2025-05-09 11:43 UTC+02:00 (CEST) - Currently testing a possibly fix.

Update: 2025-05-09 12:53 UTC+02:00 (CEST) - We are currently in the process of applying the proposed fix.

INCIDENT: Database as a Service connectivity issues, region dus2

Fri, 09 May 2025 12:12:14 +0000

Affected Components: DBaaS in dus2

Incident Start: 2025-05-09 12:00 UTC+02:00 (CEST)
Incident End: 2025-05-09 12:10 UTC+02:00 (CEST)

Description:

We are currently seeing connectivity issues with DBaaS in our dus2 region

Customer Impact:

Databases in dus2 are partly unreachable
New Databases in dus2 can not be created

Update: 2025-05-09 12:10 UTC+02:00 (CEST) - The issue has been identified and resolved

INCIDENT: SysEleven STACK Block Storage issues, region cbk

Tue, 20 May 2025 13:17:37 +0000

Affected Components: SysEleven Stack, Block Storage, region cbk

Incident Start: 2025-05-20 12:46 UTC+02:00 (CEST)

Incident End: 2025-05-20 13:30 UTC+02:00 (CEST)

Description:

We are facing issues with the distributed block storage system in region cbk.

Customer Impact:

Reduced performance of block storage

INCIDENT: PVC Failure of a hardware node, region bki

Wed, 28 May 2025 18:13:11 +0000

Affected Components: Hardware node failure, region bki

Incident Start: **2024-05-28 17:15 UTC+01:00 (CET)

Description:

Malfunction of a hardware node

Customer Impact:

During the period of restart, there will be a short interruption in the availability of the systems.
Affected customers were notified via E-Mail.
Please check the affected systems for their full functionality.

INCIDENT: SysEleven STACK API issues, region FES

Sat, 31 May 2025 16:27:46 +0000

Affected Components: SysEleven Stack API, region FES

Incident Start: 2025-05-31 16:00 Incident End: 2025-05-31 16:30

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Spawning new virtual machines (VMs) or changing existing resources is not possible.

INCIDENT: SysEleven STACK API issues, region HAM1

Sat, 31 May 2025 16:28:18 +0000

Affected Components: SysEleven Stack API, region HAM1

Incident Start: 2025-05-31 16:00 Incident End: 2025-05-31 16:30

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Spawning new virtual machines (VMs) or changing existing resources is not possible.

INCIDENT: SysEleven STACK API issues, region DUS2

Sat, 31 May 2025 16:29:09 +0000

Affected Components: SysEleven Stack API, region DUS2

Incident Start: 2025-05-31 16:00 Incident End: 2025-05-31 16:30

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Spawning new virtual machines (VMs) or changing existing resources is not possible.

INCIDENT: SysEleven STACK issues in region CBK

Fri, 04 Jul 2025 04:44:35 +0000

Affected Components: SysEleven Stack, region XXX

Incident Start: 2025-07-04 04:18 Incident End: 2025-07-04 04:45

Second Incident Start: 2025-07-04 05:15 Second Incident End: 2025-07-04 05:42

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted'