https://syseleven-status.de 2025-08-09T15:48:58.802084+00:00 SysEleven support@syseleven.de python-feedgen https://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.png Get all incidents by feed 560 2025-08-09T15:48:59.002238+00:00 <p>Affected Components: <strong>All regions</strong></p> <p>Incident Start: <strong>2025-04-10 23:40 (CETS)</strong></p> <p>Incident End: <strong>2025-04-11 01:08 (CETS)</strong></p> <hr /> <p>Description:</p> <p>We have major outage in all regions</p> <hr /> <p><strong>Update: 01:01 am</strong></p> <p>Issue identified and we are working on it.</p> <hr /> <p><strong>Update: 01:08 am</strong></p> <p>Issue is resolved and we are whatching it.</p> 2025-04-11T00:22:15+00:00 561 2025-08-09T15:48:58.997172+00:00 <p>Affected Components: <strong>SysEleven Stack Object Storage, region FES, DUS2, HAM1</strong></p> <p>Incident Start: <strong>2025-04-14 16:26 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>At the moment we are facing issues with the Object Storage in Region FES, DUS2, HAM1.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Some objects might be corrupted regarding latest known issues in Ceph. https://docs.ceph.com/en/latest/releases/squid/#v19-2-2-squid</li> <li>We are investigating the situation and working on updates to mitigate the issues</li> </ul> <hr /> <p><strong>Update: 18:40</strong></p> <ul> <li>The Ceph S3 endpoint of the DUS2 region is currently being updated, after successful testing we will proceed with the HAM1 and FES region</li> </ul> <p><strong>Update: 19:45</strong></p> <ul> <li>The update for the DUS2 region went through without problems, we are proceeding to update HAM1 and the FES region</li> <li>Affected customers will be contacted individually once we got a complete overview of potential affected buckets and objects</li> </ul> <p><strong>Update: 21:10</strong></p> <ul> <li>Update of HAM1 region also finished, FES region is in progress</li> </ul> <p><strong>Update: 22:52</strong></p> <ul> <li>Update of FES region is still ongoing</li> </ul> <p><strong>Update: 15-04-2025 09:36</strong></p> <ul> <li>Update of FES region is still ongoing and everything works as intended</li> </ul> <p><strong>Update: 16:40</strong></p> <ul> <li>Update of FES region is expected to be finished to the end of the business day</li> </ul> <p><strong>Update: 16-04-2025 03:08</strong></p> <ul> <li>Update of FES is over</li> </ul> 2025-04-14T16:51:56+00:00 562 2025-08-09T15:48:58.996004+00:00 <p>Affected Components: <strong>SysEleven Stack, region DBL</strong></p> <p>Incident Start: <strong>2025-04-17 10:00</strong></p> <p>Incident End: <strong>2025-04-17 10:45</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <p>Update [2025-04-17 10:28 am]: Alloy (GUI) &amp; API Authentication HAM &amp; DUS, Metakube API in DBL &amp; CBK are also affected</p> <p>Update [2025-04-17 10:40 am]: DBL Cloud Gateways have been identified as possible source of malfunction, we are bringing them back to function, first services are back online, next Update at 11:00 am</p> <p>Update [2025-04-17 11:00 am]: All services are up and running, we are still watching all environments, incident is declared as ended</p> 2025-04-17T10:11:22+00:00 563 2025-08-09T15:48:58.994832+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <h2>Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)</h2> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>We are expirencing a partial outage of the Network where some connection might get dropped.</li> </ul> 2025-04-25T17:18:12+00:00 564 2025-08-09T15:48:58.994059+00:00 <p>Affected Components: SysEleven Managed Hosting Network partial outage, region BLU1</p> <p>Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)</p> <p>Description: - Occurring errors are currently being investigated.</p> <p>Customer Impact: - We are expirencing a partial outage of the Network where some connection might get dropped.</p> 2025-04-25T17:46:54+00:00 565 2025-08-09T15:48:58.992696+00:00 <p>Affected Components: <strong>SysEleven Stack Network, region CBK</strong></p> <p>Incident Start: <strong>2025-04-29 17:30 UTC+02:00 (CEST)</strong> Incident End: <strong>2025-04-29 17:50 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Network connectivity to cloud region CBK was impaired</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p><strong>Update: 17:59</strong></p> <p>Network connectivity issues were resolved around 17:50</p> 2025-04-29T18:01:27+00:00 566 2025-08-09T15:48:58.990297+00:00 <p>Affected Components: <strong>MetaKube Control Plane, region DUS</strong></p> <p>Incident Start: <strong>2025-04-30 16:00 UTC+02:00</strong></p> <hr /> <p>Description:</p> <p>The loadbalancer for the kubernetes apiserver does not properly work for some clusters.</p> <hr /> <p>Customer Impact: - Potential connection loss to the apiserver</p> <hr /> <p>Update: <strong>2025-04-30 16:30 UTC+02:00</strong></p> <p>We have recovered the affected clusters but haven't found the root cause. Right now we can't see any other affected clusters but will continue to monitor the situation.</p> <p>Update: <strong>2025-04-30 16:52 UTC+02:00</strong> We have identified and fixed the root-cause of the incident. All clusters are working properly again.</p> 2025-04-30T16:10:27+00:00 567 2025-08-09T15:48:58.988319+00:00 <p>Affected Components: <strong>MetaKube Control Plane, region HAM</strong></p> <p>Incident Start: <strong>2025-05-08 14:43 UTC+02:00</strong></p> <hr /> <p>Description:</p> <p>The loadbalancer for the kubernetes apiserver does not properly work for some clusters.</p> <hr /> <p>Customer Impact: - Potential connection loss to the apiserver</p> <hr /> <p>Update: <strong>2025-05-08 15:02 UTC+02:00</strong> We have identified and fixed the root-cause of the incident. All clusters are fully operational again.</p> 2025-05-08T15:20:50+00:00 568 2025-08-09T15:48:58.986030+00:00 <p>Affected Components: <strong>SysEleven Stack Network, region CBK</strong></p> <p>Incident Start: <strong>2025-05-08 16:45 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2025-05-08 17:00 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Network connectivity to cloud region CBK was impaired</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p>Network connectivity issues were resolved around 17:00</p> 2025-05-08T17:09:28+00:00 569 2025-08-09T15:48:58.984386+00:00 <p>Affected Components: <strong>SysEleven MetaKube, region HAM and DUS</strong></p> <p>Incident Start: <strong>2025-05-09 09:30 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description: - The cluster creation in DUS and HAM is currently not working. We are currently investigating.</p> <hr /> <p>Customer Impact: - No new clusters can be created</p> <hr /> <p>Update: <strong>2025-05-09 11:23 UTC+02:00 (CEST)</strong> - We are continuing to work on rectifying the situation</p> <hr /> <p>Update: <strong>2025-05-09 11:43 UTC+02:00 (CEST)</strong> - Currently testing a possibly fix.</p> <hr /> <p>Update: <strong>2025-05-09 12:53 UTC+02:00 (CEST)</strong> - We are currently in the process of applying the proposed fix.</p> 2025-05-09T10:36:19+00:00 570 2025-08-09T15:48:58.982257+00:00 <p>Affected Components: <strong>DBaaS in dus2</strong></p> <p>Incident Start: <strong>2025-05-09 12:00 UTC+02:00 (CEST)</strong><br /> Incident End: <strong>2025-05-09 12:10 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>We are currently seeing connectivity issues with DBaaS in our dus2 region</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Databases in dus2 are partly unreachable</li> <li>New Databases in dus2 can not be created</li> </ul> <hr /> <p>Update: <strong>2025-05-09 12:10 UTC+02:00 (CEST)</strong> - The issue has been identified and resolved</p> 2025-05-09T12:12:14+00:00 571 2025-08-09T15:48:58.981106+00:00 <p>Affected Components: <strong>SysEleven Stack, Block Storage, region cbk</strong></p> <p>Incident Start: <strong>2025-05-20 12:46 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2025-05-20 13:30 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>We are facing issues with the distributed block storage system in region cbk.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Reduced performance of block storage</li> </ul> 2025-05-20T13:17:37+00:00 572 2025-08-09T15:48:58.979259+00:00 <p>Affected Components: <strong>Hardware node failure, region bki</strong></p> <p>Incident Start: **2024-05-28 17:15 UTC+01:00 (CET)</p> <hr /> <p>Description:</p> <ul> <li>Malfunction of a hardware node</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>During the period of restart, there will be a short interruption in the availability of the systems.</li> <li>Affected customers were notified via E-Mail.</li> <li>Please check the affected systems for their full functionality.</li> </ul> 2025-05-28T18:13:11+00:00 573 2025-08-09T15:48:58.978152+00:00 <p>Affected Components: <strong>SysEleven Stack API, region FES</strong></p> <p>Incident Start: <strong>2025-05-31 16:00</strong> Incident End: <strong>2025-05-31 16:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-05-31T16:27:46+00:00 574 2025-08-09T15:48:58.976903+00:00 <p>Affected Components: <strong>SysEleven Stack API, region HAM1</strong></p> <p>Incident Start: <strong>2025-05-31 16:00</strong> Incident End: <strong>2025-05-31 16:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-05-31T16:28:18+00:00 575 2025-08-09T15:48:58.973272+00:00 <p>Affected Components: <strong>SysEleven Stack API, region DUS2</strong></p> <p>Incident Start: <strong>2025-05-31 16:00</strong> Incident End: <strong>2025-05-31 16:30</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2025-05-31T16:29:09+00:00 576 2025-08-09T15:48:58.970436+00:00 <p>Affected Components: <strong>SysEleven Stack, region XXX</strong></p> <p>Incident Start: <strong>2025-07-04 04:18</strong> Incident End: <strong>2025-07-04 04:45</strong></p> <p>Second Incident Start: <strong>2025-07-04 05:15</strong> Second Incident End: <strong>2025-07-04 05:42</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted'</li> </ul> 2025-07-04T04:44:35+00:00 577 2025-08-09T15:48:58.968837+00:00 <p>Affected Components: Hardware node failure, region BKI1</p> <p>Incident Start: 2025-07-21 19:30 UTC+02:00 (CET) Incident End: 2025-07-21 20:15 UTC+02:00 (CET)</p> <hr /> <p>Description:</p> <ul> <li>Malfunction of a hardware node</li> <li>Restart of the hardware node is necessary </li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>During the period of restart, there will be a short interruption in the availability of the systems.</li> <li>Affected customers were notified via E-Mail.</li> <li>Please check the affected systems for their full functionality.</li> </ul> 2025-07-21T20:02:45+00:00 578 2025-08-09T15:48:58.966844+00:00 <p>Affected Components: <strong>Hardware node failure, region BLU1</strong></p> <p>Incident Start: 2025-07-22 23:46 UTC+02:00 (CEST)</p> <p>Incident End: 2025-07-23 01:26 UTC+02:00 (CEST)</p> <p>We found a memory failure and we are going to evacuate the hardware node.</p> 2025-07-23T00:12:12+00:00 579 2025-08-09T15:48:58.962404+00:00 <p>Affected Components: <strong>SysEleven DBaaS, region DUS, HAM</strong></p> <p>Incident Start: <strong>2025-23-07 14:23 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2025-23-07 18:23 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>We are currently investigating the situation that no new DBaaS instances can be created in the DUS and HAM regions at present. The operation of existing instances is not affected. We will report on further developments here.</p> <hr /> <p>Update:</p> <p>The creation of DBaaS instances in the HAM region is functional again. We are still working on a solution for the DUS region and will keep you informed.</p> <hr /> <p>Update:</p> <p>While all DBaaS instances continue to function as usual, we are still working on the ability of spawning new instances again.</p> <hr /> <p>Update:</p> <p>We are pleased to announce that the creation of DBaaS instances is working properly again.</p> <hr /> 2025-07-23T14:49:29+00:00