https://syseleven-status.de 2024-11-21T06:36:41.073863+00:00 SysEleven support@syseleven.de python-feedgen https://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.png Get all incidents by feed 519 2024-11-21T06:36:41.170479+00:00 <p>Affected Components: <strong>SysEleven Stack API </strong></p> <p>Incident Start: <strong>2024-08-13 13:33</strong></p> <hr /> <p>Description:</p> <ul> <li>We are currently experiencing a loss of control for DNS zones managed by the DNSaaS API</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Changes towards DNSaaS zones and records are not propagated</li> </ul> <hr /> <p>Update: <strong>2024-08-13 16:10</strong></p> <ul> <li>Root cause was identified and fixed, we are observing the situation</li> </ul> 2024-08-13T13:33:00+00:00 520 2024-11-21T06:36:41.168712+00:00 <p>Affected Components: Setups that route traffic through Cloudflare experience issues</p> <p>Investigation Start: <strong>2024-09-02 13:20 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>We are investigating network problems reported by customers that route traffic through Cloudflare</p> <hr /> <p>Customer Impact:</p> <p>Customers that are routing traffic to SysEleven networks through Cloudflare experience high retransmission rates and traffic loss.</p> <hr /> <p><strong>Update: 2024-09-02 14:20 UTC+01:00 (CET)</strong></p> <p>We are still investigating.</p> <hr /> <p><strong>Update: 2024-09-02 14:49 UTC+01:00 (CET)</strong></p> <p>We are observing significant improvements of the retransmit rates on some affected loadbalancers but will continue to observe and analyse the issues.</p> <hr /> <p><strong>Update: 2024-09-02 16:25 UTC+01:00 (CET)</strong></p> <p>Since the Problem seems to be solved for most customers, we declare the incident as solved. Further investigations into the reasons for the problems will continue.</p> 2024-09-02T14:20:00+00:00 521 2024-11-21T06:36:41.166491+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-09-03 18:00 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-09-03 21:00 UTC+01:00 (CET)</strong></p> <p>State: <strong>SCHEDULED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-09-03T09:53:42+00:00 522 2024-11-21T06:36:41.164955+00:00 <p>Affected Components: <strong>SysEleven Stack API, region HAM1 and DUS2</strong></p> <p>Incident Start: <strong>2024-09-04 15:20 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-09-04 16:30 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>New VMs could not be created</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) </li> </ul> <hr /> <p><strong>Update: 16:20</strong></p> <p>We rolled out a configuration change and are observing.</p> <p><strong>Update: 16:30</strong></p> <p>We see that VMs can be created again in our automated tests. We are closing the incident</p> 2024-09-04T15:20:00+00:00 523 2024-11-21T06:36:41.163162+00:00 <p>Affected Components: <strong>SysEleven Stack Designate API</strong></p> <p>Incident Start: <strong>2024-09-17 20:42 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-09-17 22:15 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Managing DNS zones and records through the Designate API may fail.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Loss of control for DNS resources.</li> </ul> <hr /> <p><strong>Update: 2024-09-17 21:20 UTC+02:00 (CEST)</strong></p> <p>At 21:15, we reverted a configuration change that was rolled out earlier today, which fixed the issue. We are watching the situation.</p> <hr /> <p><strong>Update: 2024-09-17 22:15 UTC+02:00 (CEST)</strong></p> <p>Underlying issues that were persistent, but not impacting the API, are now fully resolved. Incident is over.</p> <hr /> 2024-09-17T20:42:00+00:00 525 2024-11-21T06:36:41.161804+00:00 <p>Affected Components: <strong>SysEleven Stack API, region DBL</strong></p> <p>Incident Start: <strong>2024-09-19 08:23</strong> Incident End: <strong>2024-09-19 08:35</strong></p> <hr /> <p>Description:</p> <ul> <li>Loss of control situation for volume and compute services</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Creating new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> <hr /> <p>Update: <strong>2024-09-18 08:35</strong></p> <p>Some nova services also seem to be affected, we investigated the situation and bring the services back up</p> 2024-09-19T08:23:00+00:00 526 2024-11-21T06:36:41.159947+00:00 <p>Affected Components: <strong>SysEleven Stack Block Storage, region CBK</strong></p> <p>Incident Start: <strong>2024-09-24 10:55 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>At the moment we are facing issues with the Block Storage in Region CBK.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Volumes may be unavailable</li> <li>Instances booted from a volume may be unavailable</li> </ul> <hr /> <p><strong>Update: 2024-09-24 11:00 UTC+02:00 (CEST)</strong></p> <p>Issue identified and we are fixing the problem.</p> <hr /> <p><strong>Update: 2024-09-24 12:15 UTC+02:00 (CEST)</strong></p> <p>Problem is fixed. Instances using volumes had to be restarted.</p> 2024-09-24T10:55:00+00:00 528 2024-11-21T06:36:41.157978+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-09-26 20:30 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-09-26 22:30 UTC+01:00 (CET)</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-09-26T20:33:00+00:00 529 2024-11-21T06:36:41.156728+00:00 <p>Affected Components: <strong>Hardware node failure, region XXX</strong></p> <p>Incident Start: <strong>2024-10-01 09:15 UTC+02:00 (CEST)</strong> Incident End: <strong>2024-10-01 09:36 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Malfunction of a hardware node</li> <li>Restart of the hardware node is necessary </li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>During the period of restart, there will be a short interruption in the availability of the systems.</li> <li>Affected customers were notified via E-Mail.</li> <li>Please check the affected systems for their full functionality.</li> </ul> 2024-10-01T09:15:00+00:00 530 2024-11-21T06:36:41.155175+00:00 <p>Affected Components: <strong>SysEleven Stack Network, region DBL</strong></p> <p>Incident Start: <strong>2024-10-01 14:17 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-10-01 14:34 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>At the moment, we are facing issues with the Network in Region DBL.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Network performance issue</li> </ul> <hr /> <p><strong>Update: 2024-10-01 14:34 UTC+02:00 (CEST)</strong></p> <p>The incident is over, and all services are operational.</p> <hr /> 2024-10-01T14:17:00+00:00 531 2024-11-21T06:36:41.153742+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>16 Oct 2024, 22:00 CEST</strong></p> <p>Scheduled End: <strong>16 Oct 2024, 23:00 CEST</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking switches.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>No impact is expected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-10-08T16:09:00+00:00 532 2024-11-21T06:36:41.152285+00:00 <p>Affected Components: <strong>SysEleven Stack, region HAM1</strong></p> <p>Incident Start: <strong>2024-10-10 23:55</strong> Incident End: <strong>2024-10-11 00:00</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors were investigated. </li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity was restricted</li> </ul> <hr /> <p><strong>Update: 01:00</strong></p> <ul> <li>We can observe further short term issues with the HAM1 region connectivity, the provider is aware of the issues and is currently investigating the situation</li> </ul> <hr /> <p><strong>Update: 01:30</strong></p> <ul> <li>The network provider is proceeding with a network maintenance until 05:00, we are on standby</li> </ul> 2024-10-10T23:55:00+00:00 533 2024-11-21T06:36:41.150341+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-10-29 17:00 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-10-29 20:00 UTC+01:00 (CET)</strong></p> <p>State: <strong>DONE</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-10-29T15:01:00+00:00 534 2024-11-21T06:36:41.148671+00:00 <p>Affected Components: <strong>SysEleven Stack API</strong></p> <p>Incident Start: <strong>2024-10-30 09:30 CET</strong></p> <p>Incident End: <strong>2024-10-30 12:05 CET</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> <hr /> <p><strong>Update: 10:40</strong></p> <p>We are still investigating the situation and are in contact with our external network provider to further analyze the problems.</p> <hr /> <p><strong>Update: 11:30</strong></p> <p>The issue has been identified, we are waiting for our external network provider to further fix the situation.</p> <hr /> <p><strong>Update: 12:05</strong></p> <p>The issue has been resolved.</p> 2024-10-30T09:30:00+00:00 535 2024-11-21T06:36:41.146768+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>12 Nov 2024, 22:00 CET</strong></p> <p>Scheduled End: <strong>13 Nov 2024, 00:00 CET</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking gateways.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-04T11:56:00+00:00 536 2024-11-21T06:36:41.145182+00:00 <p>Affected Components: SysEleven Stack Networking in region DBL</p> <p>Scheduled Start: <strong>19 Nov 2024, 22:00 CET</strong></p> <p>Scheduled End: <strong>20 Nov 2024, 00:00 CET</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking gateways.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-07T14:19:00+00:00 537 2024-11-21T06:36:41.143880+00:00 <p>Affected Components: <strong>SysEleven Stack Object Storage, region DBL</strong></p> <p>Incident Start: <strong>2024-11-12 17:45 UTC+01:00 (CET)</strong></p> <p>Incident End: <strong>2024-11-12 18:40 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>At the moment we are facing issues with the Object Storage in Region DBL.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Writing or reading of objects maybe restricted.</li> </ul> <hr /> <p>Update: <strong>2024-11-12 18:40 UTC+01:00 (CET)</strong></p> <p>We mitigated the problem and do further investigation</p> 2024-11-12T17:45:00+00:00 538 2024-11-21T06:36:41.142158+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <p>Incident Start: <strong>2024-11-12 22:35</strong> Incident End: <strong>2024-11-13 00:00</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p><strong>Update: 23:08</strong></p> <p>The announced maintenance is having a bigger impact than expected, we are investigating the situation</p> <hr /> <p><strong>Update: 23:45</strong></p> <p>We were able to pin down the rootcause and prepare a fix to mitigate the problems</p> <hr /> <p><strong>Update: 12:00</strong></p> <p>The network problems were mitigated. If you still encounter issues please contact us!</p> 2024-11-12T22:35:00+00:00 539 2024-11-21T06:36:41.140493+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>14 Nov 2024, 22:00 CET</strong></p> <p>Scheduled End: <strong>15 Nov 2024, 01:00 CET</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform an emergency maintenance on networking gateways in order to improve performance settings.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-14T17:14:00+00:00 541 2024-11-21T06:36:41.137753+00:00 <p>Affected Components: <strong>MetaKube Control Plane Services, region FES</strong></p> <p>Incident Start: <strong>2024-11-18 11:30 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>Infrastructure hosting the MetaKube Control Plane Services has problems.</p> <hr /> <p>Customer Impact:</p> <ul> <li>MetaKube Control Plane might be slow or not answering</li> </ul> <hr /> <p><strong>UPDATE 2024-11-18 12:30 UTC+01:00 (CET)</strong></p> <p>We have identified networking problems as the cause, currently working to resolve them.</p> <p><strong>UPDATE 2024-11-18 13:27 UTC+01:00 (CET)</strong></p> <p>We have increased conntrack table size on hardware nodes to avoid networking problems.</p> <p>We continue to have issues with overloaded pods which we are working on.</p> <p><strong>UPDATE 2024-11-18 14:00 UTC+01:00 (CET)</strong></p> <p>We managed to get the overloaded pods running by isolating them on dedicated nodes and raising the resource limits. This stopped other issues as well.</p> <p>We still need to investigate what caused the overloading of certain pods.</p> <p>Incident is over.</p> 2024-11-18T11:30:00+00:00