https://syseleven-status.de 2025-06-28T20:50:34.836848+00:00 SysEleven support@syseleven.de python-feedgen https://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.png Get all incidents by feed 555 2025-06-28T20:50:35.047378+00:00 <p>Affected Components: <strong>MetaKube Control Planes, region FES</strong></p> <p>Incident Start: <strong>2025-02-11 06:00 UTC+01:00 (CET)</strong></p> <p><strong>State: Resolved</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the MetaKube API is not ensured.</li> <li>After a scheduled maintenance to the network in FES, the MetaKube control cluster (which is hosting the customer control planes) has problems reaching DNS. This is causing issues to the customer control planes.</li> <li>This also affects Database as a Service and Observability as a Service</li> <li>All times below are CET</li> </ul> <p><strong>UPDATE 2025-02-13 12:20</strong> - We consider all service disruptions of the incident to be mitigated - Although we are not expecting any more service disruptions, we are still watching all systems closly</p> <p><strong>Previous Updates in reverse chronological order</strong></p> <p><strong>UPDATE 2025-02-11 10:00</strong></p> <ul> <li>Still investigating the DNS issue. We sent out a notifier to all potentially affected customers.</li> </ul> <hr /> <p><strong>UPDATE 2025-02-11 11:15</strong> - We have used the time since the last update to narrow down the root cause of the incident. We excluded some possibilites but did not find the root-cause. We are now preparing a partial rollback to downgrade the SDN again.</p> <hr /> <p><strong>UPDATE 2025-02-11 12:05</strong> - We completed a partial OVN/SDN downgrade, however this has not yet resolved the incident. - We are investigating further</p> <hr /> <p><strong>UPDATE 2025-02-11 12:25</strong> - We’re exploring further downgrade approaches (previous rollbacks were, as announced, partial) and are in parallel investigating further.</p> <hr /> <p><strong>UPDATE 2025-02-11 13:00</strong> - As we originally updated the SDN due to a critical security gap, it was decided that we will not perform a full OVN/SDN rollback to the initial state. - We have now activated several teams who will be developing and evaluating different solutions until 1.30 pm. An update on how we proceed will follow then.</p> <hr /> <p><strong>UPDATE 2025-02-11 13:50</strong> - Our Teams will continue developing and evaluating solutions in break out session as there are further leads but no breakthrough, yet. - In parallel we are preparing a failover for IAM and Alloy to DUS/HAM</p> <hr /> <p><strong>UPDATE 2025-02-11 15:30</strong> - Our teams investigation in SDN traffic loss is ongoing - Our teams continue developing and evaluating solutions and possible workarounds - In parallel we are evaluating a rebuild of the SDN (software defined network)</p> <hr /> <p><strong>UPDATE 2025-02-11 17:47</strong> - Part of the services are still not functional - We are still working hard to resolve the issues but we will roll back the update of the SDN if no progress is made. - The planned maintenance period begins today, 11 February 2025, at 23:00 and is expected to last until around 06:00 CET on 12 February 2025, during which time there may be repeated interruptions to services. - The maintenance is also announced via notifier. You will get an info of the end of the maintenance also via notifier.</p> <hr /> <p><strong>UPDATE 2025-02-11 20:15</strong> - IAM, Alloy and Observability as a Service are restored to full functionality - The Database as a Service API has also been restored, but the API still has some issues which are related to the wider SDN problem. The Databases themselves were at no point affected by the incident.</p> <hr /> <p><strong>UPDATE 2025-02-11 23:00</strong> - The situation with the API improved. We will continue watching it.</p> <hr /> <p><strong>UPDATE 2025-02-12 10:15</strong> - The previous maintenance work did not achieve the desired success. - Workarounds have been implemented, so operations should be able to continue without disruptions. - Maintainance work will continue during the upcoming night to fully resolve the incident.</p> <hr /> <p><strong>UPDATE 2025-02-12 14:55</strong> - We scheduled another maintenance window for this night, February 12th, from 11:00 PM to 6:00 AM the following day. - During this maintenance window, there may be brief interruptions or limited availability of certain services. - The goal is the complete resolution of the incident caused by Monday's update. - An RfO will be available in our Helpdesk after the incident is mitigated.</p> <hr /> <p><strong>UPDATE 2025-02-13 12:20</strong> - We consider all service disruptions of the incident to be mitigated - Although we are not expecting any more service disruptions, we are still watching all systems closly</p> <hr /> <p><strong>UPDATE 2025-02-13 15:30</strong> - Incident is resolved</p> 2025-02-11T08:04:01+00:00 556 2025-06-28T20:50:35.045042+00:00 <p>Affected Components: <strong>SysEleven Stack, region FES</strong></p> <p>Incident Start: <strong>2025-03-08 13:00</strong></p> <p>Incident End: <strong>2025-03-08 15:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>IPv6 Connectivity is restricted</li> </ul> <hr /> <p><strong>Update: 16:25 </strong></p> <p>Flaky ipv6 connectivity was repaired around 15:30, a mismatch in configuration was identified</p> 2025-03-08T14:29:22+00:00 557 2025-06-28T20:50:35.043129+00:00 <p>Affected Components: <strong>SysEleven Stack API, region CBK</strong></p> <p>Incident Start: <strong>2025-03-13 14:30</strong> Incident End: <strong>2025-03-13 17:15</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-03-13T14:43:01+00:00 558 2025-06-28T20:50:35.042326+00:00 <p>We're currently investigating an issue in our IAM systems that prevents creation and deletion of projects.</p> 2025-03-17T10:39:16+00:00 560 2025-06-28T20:50:35.039908+00:00 <p>Affected Components: <strong>All regions</strong></p> <p>Incident Start: <strong>2025-04-10 23:40 (CETS)</strong></p> <p>Incident End: <strong>2025-04-11 01:08 (CETS)</strong></p> <hr /> <p>Description:</p> <p>We have major outage in all regions</p> <hr /> <p><strong>Update: 01:01 am</strong></p> <p>Issue identified and we are working on it.</p> <hr /> <p><strong>Update: 01:08 am</strong></p> <p>Issue is resolved and we are whatching it.</p> 2025-04-11T00:22:15+00:00 561 2025-06-28T20:50:35.034827+00:00 <p>Affected Components: <strong>SysEleven Stack Object Storage, region FES, DUS2, HAM1</strong></p> <p>Incident Start: <strong>2025-04-14 16:26 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>At the moment we are facing issues with the Object Storage in Region FES, DUS2, HAM1.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Some objects might be corrupted regarding latest known issues in Ceph. https://docs.ceph.com/en/latest/releases/squid/#v19-2-2-squid</li> <li>We are investigating the situation and working on updates to mitigate the issues</li> </ul> <hr /> <p><strong>Update: 18:40</strong></p> <ul> <li>The Ceph S3 endpoint of the DUS2 region is currently being updated, after successful testing we will proceed with the HAM1 and FES region</li> </ul> <p><strong>Update: 19:45</strong></p> <ul> <li>The update for the DUS2 region went through without problems, we are proceeding to update HAM1 and the FES region</li> <li>Affected customers will be contacted individually once we got a complete overview of potential affected buckets and objects</li> </ul> <p><strong>Update: 21:10</strong></p> <ul> <li>Update of HAM1 region also finished, FES region is in progress</li> </ul> <p><strong>Update: 22:52</strong></p> <ul> <li>Update of FES region is still ongoing</li> </ul> <p><strong>Update: 15-04-2025 09:36</strong></p> <ul> <li>Update of FES region is still ongoing and everything works as intended</li> </ul> <p><strong>Update: 16:40</strong></p> <ul> <li>Update of FES region is expected to be finished to the end of the business day</li> </ul> <p><strong>Update: 16-04-2025 03:08</strong></p> <ul> <li>Update of FES is over</li> </ul> 2025-04-14T16:51:56+00:00 562 2025-06-28T20:50:35.032067+00:00 <p>Affected Components: <strong>SysEleven Stack, region DBL</strong></p> <p>Incident Start: <strong>2025-04-17 10:00</strong></p> <p>Incident End: <strong>2025-04-17 10:45</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <p>Update [2025-04-17 10:28 am]: Alloy (GUI) &amp; API Authentication HAM &amp; DUS, Metakube API in DBL &amp; CBK are also affected</p> <p>Update [2025-04-17 10:40 am]: DBL Cloud Gateways have been identified as possible source of malfunction, we are bringing them back to function, first services are back online, next Update at 11:00 am</p> <p>Update [2025-04-17 11:00 am]: All services are up and running, we are still watching all environments, incident is declared as ended</p> 2025-04-17T10:11:22+00:00 563 2025-06-28T20:50:35.030614+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <h2>Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)</h2> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>We are expirencing a partial outage of the Network where some connection might get dropped.</li> </ul> 2025-04-25T17:18:12+00:00 564 2025-06-28T20:50:35.029521+00:00 <p>Affected Components: SysEleven Managed Hosting Network partial outage, region BLU1</p> <p>Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)</p> <p>Description: - Occurring errors are currently being investigated.</p> <p>Customer Impact: - We are expirencing a partial outage of the Network where some connection might get dropped.</p> 2025-04-25T17:46:54+00:00 565 2025-06-28T20:50:35.027131+00:00 <p>Affected Components: <strong>SysEleven Stack Network, region CBK</strong></p> <p>Incident Start: <strong>2025-04-29 17:30 UTC+02:00 (CEST)</strong> Incident End: <strong>2025-04-29 17:50 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Network connectivity to cloud region CBK was impaired</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p><strong>Update: 17:59</strong></p> <p>Network connectivity issues were resolved around 17:50</p> 2025-04-29T18:01:27+00:00 566 2025-06-28T20:50:35.024227+00:00 <p>Affected Components: <strong>MetaKube Control Plane, region DUS</strong></p> <p>Incident Start: <strong>2025-04-30 16:00 UTC+02:00</strong></p> <hr /> <p>Description:</p> <p>The loadbalancer for the kubernetes apiserver does not properly work for some clusters.</p> <hr /> <p>Customer Impact: - Potential connection loss to the apiserver</p> <hr /> <p>Update: <strong>2025-04-30 16:30 UTC+02:00</strong></p> <p>We have recovered the affected clusters but haven't found the root cause. Right now we can't see any other affected clusters but will continue to monitor the situation.</p> <p>Update: <strong>2025-04-30 16:52 UTC+02:00</strong> We have identified and fixed the root-cause of the incident. All clusters are working properly again.</p> 2025-04-30T16:10:27+00:00 567 2025-06-28T20:50:35.022889+00:00 <p>Affected Components: <strong>MetaKube Control Plane, region HAM</strong></p> <p>Incident Start: <strong>2025-05-08 14:43 UTC+02:00</strong></p> <hr /> <p>Description:</p> <p>The loadbalancer for the kubernetes apiserver does not properly work for some clusters.</p> <hr /> <p>Customer Impact: - Potential connection loss to the apiserver</p> <hr /> <p>Update: <strong>2025-05-08 15:02 UTC+02:00</strong> We have identified and fixed the root-cause of the incident. All clusters are fully operational again.</p> 2025-05-08T15:20:50+00:00 568 2025-06-28T20:50:35.021064+00:00 <p>Affected Components: <strong>SysEleven Stack Network, region CBK</strong></p> <p>Incident Start: <strong>2025-05-08 16:45 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2025-05-08 17:00 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Network connectivity to cloud region CBK was impaired</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p>Network connectivity issues were resolved around 17:00</p> 2025-05-08T17:09:28+00:00 569 2025-06-28T20:50:35.018499+00:00 <p>Affected Components: <strong>SysEleven MetaKube, region HAM and DUS</strong></p> <p>Incident Start: <strong>2025-05-09 09:30 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description: - The cluster creation in DUS and HAM is currently not working. We are currently investigating.</p> <hr /> <p>Customer Impact: - No new clusters can be created</p> <hr /> <p>Update: <strong>2025-05-09 11:23 UTC+02:00 (CEST)</strong> - We are continuing to work on rectifying the situation</p> <hr /> <p>Update: <strong>2025-05-09 11:43 UTC+02:00 (CEST)</strong> - Currently testing a possibly fix.</p> <hr /> <p>Update: <strong>2025-05-09 12:53 UTC+02:00 (CEST)</strong> - We are currently in the process of applying the proposed fix.</p> 2025-05-09T10:36:19+00:00 570 2025-06-28T20:50:35.015833+00:00 <p>Affected Components: <strong>DBaaS in dus2</strong></p> <p>Incident Start: <strong>2025-05-09 12:00 UTC+02:00 (CEST)</strong><br /> Incident End: <strong>2025-05-09 12:10 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>We are currently seeing connectivity issues with DBaaS in our dus2 region</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Databases in dus2 are partly unreachable</li> <li>New Databases in dus2 can not be created</li> </ul> <hr /> <p>Update: <strong>2025-05-09 12:10 UTC+02:00 (CEST)</strong> - The issue has been identified and resolved</p> 2025-05-09T12:12:14+00:00 571 2025-06-28T20:50:35.013250+00:00 <p>Affected Components: <strong>SysEleven Stack, Block Storage, region cbk</strong></p> <p>Incident Start: <strong>2025-05-20 12:46 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2025-05-20 13:30 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>We are facing issues with the distributed block storage system in region cbk.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Reduced performance of block storage</li> </ul> 2025-05-20T13:17:37+00:00 572 2025-06-28T20:50:35.008006+00:00 <p>Affected Components: <strong>Hardware node failure, region bki</strong></p> <p>Incident Start: **2024-05-28 17:15 UTC+01:00 (CET)</p> <hr /> <p>Description:</p> <ul> <li>Malfunction of a hardware node</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>During the period of restart, there will be a short interruption in the availability of the systems.</li> <li>Affected customers were notified via E-Mail.</li> <li>Please check the affected systems for their full functionality.</li> </ul> 2025-05-28T18:13:11+00:00 573 2025-06-28T20:50:35.005398+00:00 <p>Affected Components: <strong>SysEleven Stack API, region FES</strong></p> <p>Incident Start: <strong>2025-05-31 16:00</strong> Incident End: <strong>2025-05-31 16:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-05-31T16:27:46+00:00 574 2025-06-28T20:50:35.002322+00:00 <p>Affected Components: <strong>SysEleven Stack API, region HAM1</strong></p> <p>Incident Start: <strong>2025-05-31 16:00</strong> Incident End: <strong>2025-05-31 16:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-05-31T16:28:18+00:00 575 2025-06-28T20:50:34.998248+00:00 <p>Affected Components: <strong>SysEleven Stack API, region DUS2</strong></p> <p>Incident Start: <strong>2025-05-31 16:00</strong> Incident End: <strong>2025-05-31 16:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-05-31T16:29:09+00:00