https://syseleven-status.de 2024-04-28T05:06:21.256755+00:00 SysEleven support@syseleven.de python-feedgen https://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.png Get all incidents by feed 481 2024-04-28T05:06:21.371246+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>05. Feb. 2024 / 6:00 pm</strong></p> <p>Scheduled End: <strong>05. Feb. 2024 / 7:00 pm</strong></p> <p>State: <strong>FINISHED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of MetaKube platform infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;1min) of the control plane possible (e.g. apiserver, controller-manager, scheduler), but not MetaKube API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> <p><strong>UPDATE 19:15</strong></p> <p>Maintenance finished succesfully.</p> 2024-01-30T14:12:00+00:00 482 2024-04-28T05:06:21.370087+00:00 <p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p> <p>Incident Start: <strong>5 Feb 2024 03:03</strong></p> <hr /> <p>Description:</p> <ul> <li>DNS servers overloaded</li> <li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>availability of dns servers is reduced</li> </ul> <hr /> <p>Update <strong>5 Feb 2024 03:22</strong></p> <p>Connectivity recovered</p> 2024-02-05T03:03:00+00:00 483 2024-04-28T05:06:21.368959+00:00 <p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p> <p>Incident Start: <strong>6 Feb 2024 17:44</strong> Incident End: <strong>6 Feb 2024 17:59</strong></p> <hr /> <p>Description:</p> <ul> <li>DNS servers overloaded</li> <li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>availability of dns servers is reduced</li> </ul> <hr /> <p>Update 18:00</p> <p>Connectivity recovered</p> 2024-02-06T17:44:00+00:00 484 2024-04-28T05:06:21.367784+00:00 <p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p> <p>Incident Start: <strong>7 Feb 2024 12:15</strong> Incident End: <strong>7 Feb 2024 12:35</strong></p> <hr /> <p>Description:</p> <ul> <li>DNS servers overloaded</li> <li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>availability of dns servers is reduced</li> </ul> <hr /> <p>Update <strong>12:35</strong></p> <p>Connectivity recovered</p> 2024-02-07T12:15:00+00:00 485 2024-04-28T05:06:21.365848+00:00 <p>Affected Components: <strong>all regions</strong></p> <p>Scheduled Start: <strong>07. Feb. 2024/06:00 pm</strong> Scheduled End: <strong>07. Feb. 2024/09:00 pm</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>Infrastructure security update in all regions.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-02-07T07:00:00+00:00 486 2024-04-28T05:06:21.363746+00:00 <p>Affected Components: <strong>Network Component Fortigate</strong></p> <p>Scheduled Start: <strong>Feb. 11 2024 / 08:00 p.m.</strong></p> <p>Scheduled End: <strong>Feb. 11 2024 / 11: p.m.</strong></p> <p>State: <strong>Scheduled Maintainance</strong></p> <hr /> <p>Description:</p> <p>Urgent security update to Fortigate network component</p> <p>(FortiOS - Out-of-bound Write in sslvpnd (RCE) (CVE-2024-21762))</p> <p>Expected Impact:</p> <p>About 1-2 short disconnections when the services are panning.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Network Service might be interrupted when actual maintainance takes place</li> </ul> <hr /> <p>Update <strong>4:30 p.m.</strong></p> <p>After further evaluating options to mitigate or patch the disclosed vulnerabilities, we learned that patches were anounced but not yet available for download. We continue to monitor the situation closely and will act as soon as patches and upgrades are available.</p> <hr /> <p>Update <strong>Sunday Feb. 11 7:00 p.m.</strong></p> <p>After evaluating and testing the provided FortiOS Updates in our datacenter Lankwitzer Straße (BLA) <strong>we will be rolling out the patches today between 8 and 11 p.m. in our datacenters BKI and BLU</strong>. We have contacted affected customers separately. As the maintenance work affects our VPN routers, short outages of VPN connections are to be expected during the maintenance window mentioned.</p> <hr /> <p>Update <strong>Sunday Feb. 11 11:00 p.m.</strong> All instances of FortiOS in our datacenters have been updated to versions secure against CVE-2024-21762.</p> 2024-02-11T20:00:00+00:00 487 2024-04-28T05:06:21.362374+00:00 <p>Affected Components: <strong>MetaKube apiserver external availability, some clusters, DBL/CBK</strong></p> <p>Incident Start: <strong>10.Feb 2024/11:00 am</strong> Incident End: <strong>10. Feb. 2024/12:35 am</strong></p> <hr /> <p>Description:</p> <p>An expired client certificate caused some load balancer configuration to go stale. Thus, a small number of clusters lost external availability of their apiservers.</p> <hr /> <p>Customer Impact:</p> <ul> <li>External apiserver availability disrupted for at most 8 clusters at peak for a maximum of 95min</li> </ul> <hr /> <p>Customer Actions:</p> <p>Examples: - No customer actions needed</p> <hr /> <p><strong>Update 12:35</strong> The certificate was renewed and a failover of the load balancer triggered. This restored availability of all clusters again.</p> 2024-02-10T11:00:00+00:00 489 2024-04-28T05:06:21.361151+00:00 <p>Affected Components: <strong>SysEleven Stack Object Storage, region FES</strong></p> <p>Incident Start: <strong>12th Feb. 2024/13:00 pm (CET - UTC +1)</strong></p> <p>Incident End: <strong>12th Feb. 2024/14:15 pm (CET - UTC +1)</strong></p> <hr /> <p>Description:</p> <p>At the moment, we are facing issues with the Object Storage (Ceph) in Region FES.</p> <hr /> <p>Customer Impact:</p> <ul> <li>API is not working correctly</li> </ul> <hr /> <p>Update <strong>14:18 p.m.</strong></p> <ul> <li>Issue is fixed and service is operational</li> </ul> <hr /> 2024-02-12T13:54:00+00:00 490 2024-04-28T05:06:21.359167+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>19. Feb 2024 / 06:00 pm</strong></p> <p>Scheduled End: <strong>19. Feb. 2024 / 9:00 pm</strong></p> <p>State: <strong>FINISHED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-02-19T18:00:00+00:00 491 2024-04-28T05:06:21.357899+00:00 <p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p> <p>Incident Start: <strong>19 Feb 2024 08:30 CET</strong></p> <p>Incident End: <strong>19 Feb 2024 08:50 CET</strong></p> <hr /> <p>Description:</p> <ul> <li>DNS servers overloaded</li> <li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>availability of dns servers is reduced</li> </ul> <hr /> <p>Update <strong>19 Feb 2024 08:50 CET</strong></p> <p>Connectivity recovered</p> 2024-02-19T08:30:00+00:00 492 2024-04-28T05:06:21.356547+00:00 <p>Affected Components: <strong>Hardware node failure, region blu</strong></p> <p>Incident Start: <strong>2024-02-23 22:35 UTC+01:00 (CET)</strong> Incident End: <strong>2024-02-23 23:45 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>Malfunction of a hardware node</li> <li>Restart of the hardware node is necessary </li> <li>Evacuation of custmer system was necessary</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>During the period of restart, there will be a short interruption in the availability of the systems.</li> <li>Affected customers were notified via E-Mail.</li> <li>Please check the affected systems for their full functionality.</li> </ul> 2024-02-23T22:35:00+00:00 493 2024-04-28T05:06:21.354776+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: **2024-02-29 18:00 UTC+01:00 (CET) </p> <p>Scheduled End: **2024-02-29 21:00 UTC+01:00 (CET)</p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler).</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-02-27T15:29:56+00:00 494 2024-04-28T05:06:21.352933+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: **2024-03-07 18:00 UTC+01:00 (CET) </p> <p>Scheduled End: **2024-03-07 19:00 UTC+01:00 (CET) </p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-03-05T13:02:00+00:00 495 2024-04-28T05:06:21.351502+00:00 <p>Affected Components: MetaKube Dashboard API</p> <p>Incident Start: <strong>2024-03-012 16:00 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven MetaKube Dashboard API is not ensured.</li> <li>Clusters may not be reconciled currently</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Access to MetaKube Dashboard and API (including Terraform) might be not possible</li> <li>Clusters may not be reconciled currently</li> </ul> <hr /> <p>Update: <strong>2024-03-012 17:30 UTC+01:00 (CET)</strong></p> <ul> <li>Issues are fixed</li> </ul> 2024-03-12T16:00:00+00:00 496 2024-04-28T05:06:21.349705+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: 2024-03-25 18:00 UTC+01:00 (CET) </p> <p>Scheduled End: 2024-03-25 21:30 UTC+01:00 (CET) </p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-03-25T18:00:00+00:00 497 2024-04-28T05:06:21.348664+00:00 <p>Affected Components: <strong>SysEleven Stack Object Storage (Quobyte), region FES</strong></p> <p>Incident Start: <strong>2024-03-24 19:14 UTC+01:00 (CET)</strong></p> <p>Incident End: <strong>2024-03-24 19:28 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>The Object Storage (Quobyte) API in region FES may have been intermittently unreachable.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Writing or reading of objects maybe restricted.</li> </ul> <hr /> 2024-03-24T19:14:00+00:00 498 2024-04-28T05:06:21.345780+00:00 <p>Affected Components: <strong>SysEleven Stack, MetaKube clusters, MetaKube API, region DBL</strong></p> <p>Incident Start: <strong>2024-03-25 10:18 UTC+01:00 (CET)</strong></p> <p>Incident End: <strong>2024-03-25 13:00 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>Major Outage of multiple Hardware Nodes</li> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Storage and Compute services in the DBL region are disrupted</li> </ul> <hr /> <p>Update : 10:39</p> <ul> <li>One Rack in the DBL Datacenter seems to be down</li> <li>Rack is getting investigated by remote hands</li> </ul> <hr /> <p>Update : 10:50</p> <ul> <li>Datacenter operator confirmed power outage</li> <li>Hardware nodes are coming back online</li> <li>Follow up problems are being tackled</li> </ul> <hr /> <p>Update : 11:03</p> <ul> <li>We are still stabilizing the affected systems</li> <li>We see some recoveries coming in already</li> </ul> <hr /> <p>Update : 11:17</p> <ul> <li>We still see issues on software loadbalancers (Octavia) which we are being investigated at the minute</li> <li>We see a few CRC problems and will contact the affected customers individually after doing a deep scrub on the storage</li> </ul> <hr /> <p>Update : 11:40</p> <ul> <li>Octavia loadbalancer recoveries and storage scrub tasks are ongoing</li> </ul> 2024-03-25T10:18:00+00:00 500 2024-04-28T05:06:21.344570+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <p>Incident Start: <strong>2024-04-08 09:02 UTC+01:00 (CET)</strong> Incident End: <strong>2024-04-08 09:30 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> 2024-04-08T09:02:00+00:00 501 2024-04-28T05:06:21.341683+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-04-09 06:00 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-04-09 07:00 UTC+01:00 (CET)</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>To ensure long-term resiliency and scalability, we migrate all MetaKube clusters' Kubernetes API endpoint to new load balancers. The endpoint will resolve to new IPs, but the migration itself will be seamless.</p> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Certain API actions won't propagate to clusters for up to 30min</li> <li>Interruptions of existing connections to the Kubernetes API during a brief amount of time (5m)</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-04-08T15:34:00+00:00 502 2024-04-28T05:06:21.339265+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <p>Incident Start: <strong>2024-04-09 10:34 UTC+01:00 (CET)</strong> Incident End: <strong>2024-04-09 11:00 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> 2024-04-09T10:34:00+00:00