Past Incidents

Affected Components: SysEleven Stack DNS Issues

Incident Start: 19 Feb 2024 08:30 CET

Incident End: 19 Feb 2024 08:50 CET

Description:

• DNS servers overloaded
• ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

• availability of dns servers is reduced

Update 19 Feb 2024 08:50 CET

Connectivity recovered

Affected Components: MetaKube Platform, all regions

Scheduled Start: 19. Feb 2024 / 06:00 pm

Scheduled End: 19. Feb. 2024 / 9:00 pm

State: FINISHED

Description:

• Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the maintenance:

• Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
• Running workloads will not be affected
• Network availability will not be affected
• Storage availability will not be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack Object Storage, region FES

Incident Start: 12th Feb. 2024/13:00 pm (CET - UTC +1)

Incident End: 12th Feb. 2024/14:15 pm (CET - UTC +1)

Description:

At the moment, we are facing issues with the Object Storage (Ceph) in Region FES.

Customer Impact:

• API is not working correctly

Update 14:18 p.m.

• Issue is fixed and service is operational

Affected Components: Network Component Fortigate

Scheduled Start: Feb. 11 2024 / 08:00 p.m.

Scheduled End: Feb. 11 2024 / 11: p.m.

State: Scheduled Maintainance

Description:

Urgent security update to Fortigate network component

(FortiOS - Out-of-bound Write in sslvpnd (RCE) (CVE-2024-21762))

Expected Impact:

About 1-2 short disconnections when the services are panning.

Customer Impact during the Maintenance:

• Network Service might be interrupted when actual maintainance takes place

Update 4:30 p.m.

After further evaluating options to mitigate or patch the disclosed vulnerabilities, we learned that patches were anounced but not yet available for download. We continue to monitor the situation closely and will act as soon as patches and upgrades are available.

Update Sunday Feb. 11 7:00 p.m.

After evaluating and testing the provided FortiOS Updates in our datacenter Lankwitzer Straße (BLA) we will be rolling out the patches today between 8 and 11 p.m. in our datacenters BKI and BLU. We have contacted affected customers separately. As the maintenance work affects our VPN routers, short outages of VPN connections are to be expected during the maintenance window mentioned.

Update Sunday Feb. 11 11:00 p.m. All instances of FortiOS in our datacenters have been updated to versions secure against CVE-2024-21762.

Affected Components: MetaKube apiserver external availability, some clusters, DBL/CBK

Incident Start: 10.Feb 2024/11:00 am Incident End: 10. Feb. 2024/12:35 am

Description:

An expired client certificate caused some load balancer configuration to go stale. Thus, a small number of clusters lost external availability of their apiservers.

Customer Impact:

• External apiserver availability disrupted for at most 8 clusters at peak for a maximum of 95min

Customer Actions:

Examples:

• No customer actions needed

Update 12:35 The certificate was renewed and a failover of the load balancer triggered. This restored availability of all clusters again.

Affected Components: all regions

Scheduled Start: 07. Feb. 2024/06:00 pm Scheduled End: 07. Feb. 2024/09:00 pm

State: COMPLETED

Description:

Infrastructure security update in all regions.

Customer Impact during the Maintenance:

• Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
• Running workloads will not be affected
• Network availability will not be affected
• Storage availability will not be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack DNS Issues

Incident Start: 6 Feb 2024 17:44 Incident End: 6 Feb 2024 17:59

Description:

• DNS servers overloaded
• ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

• availability of dns servers is reduced

Update 18:00

Connectivity recovered

Affected Components: MetaKube Platform, all regions

Scheduled Start: 05. Feb. 2024 / 6:00 pm

Scheduled End: 05. Feb. 2024 / 7:00 pm

State: FINISHED

Description:

• Scheduled maintenance of MetaKube platform infrastructure

Customer Impact during the maintenance:

• Short interruptions (<1min) of the control plane possible (e.g. apiserver, controller-manager, scheduler), but not MetaKube API
• Running workloads will not be affected
• Network availability will not be affected
• Storage availability will not be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

UPDATE 19:15

Maintenance finished succesfully.

Affected Components: SysEleven Stack API in all regions

Scheduled Start: 30 Jan 2024, 22:00 CET

Scheduled End: 31 Jan 2024, 02:00 CET

State: SCHEDULED

Description:

We will perform maintenance on the central authentication service of the SysEleven Stack.

Customer Impact during the Maintenance:

• The SysEleven Stack APIs might be unavailable for a few seconds.
• Running VMs are not affected.

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack DNS Issues

Incident Start: 28th Jan. 2024/10:48 pm

Description:

• DNS servers overloaded
• ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

• availability of dns servers is reduced

Update 28 Jan 2024/11:07

Connectivity recovered

Affected Components: SysEleven Stack Networking, region CBK

Incident Start: 27th Jan. 2024/06:57 am

Description:

• We currently experience issues with our SDN in cbk

Customer Impact:

• Connectivity is restricted

Update 07:50 : The situation has been resolved.

Affected Components: MetaKube Platform, all regions

Scheduled Start: 24. Jan. 2024 / 07:00 pm

Scheduled End: 24. Jan. 2024 / 10:00 pm

State: COMPLETED

Description:

• Scheduled maintenance of the MetaKube platform in all regions
• Small update to MetaKube infrastructure

Customer Impact during the maintenance:

• Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
• Running workloads will not be affected
• Network availability will not be affected
• Storage availability will not be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities