Past Incidents

Affected Components: SysEleven Stack Network, region DBL

Incident Start: 2024-10-01 14:17 UTC+02:00 (CEST)

Incident End: 2024-10-01 14:34 UTC+02:00 (CEST)

Description:

At the moment, we are facing issues with the Network in Region DBL.

Customer Impact:

• Network performance issue

Update: 2024-10-01 14:34 UTC+02:00 (CEST)

The incident is over, and all services are operational.

Affected Components: MetaKube Platform, all regions

Scheduled Start: 2024-09-26 20:30 UTC+01:00 (CET)

Scheduled End: 2024-09-26 22:30 UTC+01:00 (CET)

State: COMPLETED

Description:

• Scheduled maintenance of the MetaKube platform in all regions
• Small update to MetaKube infrastructure

Customer Impact during the maintenance:

• Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
• Running workloads will not be affected
• Network availability will not be affected
• Storage availability will not be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack Block Storage, region CBK

Incident Start: 2024-09-24 10:55 UTC+02:00 (CEST)

Description:

At the moment we are facing issues with the Block Storage in Region CBK.

Customer Impact:

• Volumes may be unavailable
• Instances booted from a volume may be unavailable

Update: 2024-09-24 11:00 UTC+02:00 (CEST)

Issue identified and we are fixing the problem.

Update: 2024-09-24 12:15 UTC+02:00 (CEST)

Problem is fixed. Instances using volumes had to be restarted.

Affected Components: SysEleven Stack API, region DBL

Incident Start: 2024-09-19 08:23 Incident End: 2024-09-19 08:35

Description:

• Loss of control situation for volume and compute services

Customer Impact:

• Creating new virtual machines (VMs) or changing existing resources is not possible.

Update: 2024-09-18 08:35

Some nova services also seem to be affected, we investigated the situation and bring the services back up

Affected Components: SysEleven Stack Designate API

Incident Start: 2024-09-17 20:42 UTC+02:00 (CEST)

Incident End: 2024-09-17 22:15 UTC+02:00 (CEST)

Description:

• Managing DNS zones and records through the Designate API may fail.

Customer Impact:

• Loss of control for DNS resources.

Update: 2024-09-17 21:20 UTC+02:00 (CEST)

At 21:15, we reverted a configuration change that was rolled out earlier today, which fixed the issue. We are watching the situation.

Update: 2024-09-17 22:15 UTC+02:00 (CEST)

Underlying issues that were persistent, but not impacting the API, are now fully resolved. Incident is over.

Affected Components: SysEleven Stack API, region HAM1 and DUS2

Incident Start: 2024-09-04 15:20 UTC+02:00 (CEST)

Incident End: 2024-09-04 16:30 UTC+02:00 (CEST)

Description:

• New VMs could not be created

Customer Impact:

• Spawning new virtual machines (VMs)

Update: 16:20

We rolled out a configuration change and are observing.

Update: 16:30

We see that VMs can be created again in our automated tests. We are closing the incident

Affected Components: MetaKube Platform, all regions

Scheduled Start: 2024-09-03 18:00 UTC+01:00 (CET)

Scheduled End: 2024-09-03 21:00 UTC+01:00 (CET)

State: SCHEDULED

Description:

• Scheduled maintenance of the MetaKube platform in all regions
• Small update to MetaKube infrastructure

Customer Impact during the maintenance:

• Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
• Running workloads will not be affected
• Network availability will not be affected
• Storage availability will not be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: Setups that route traffic through Cloudflare experience issues

Investigation Start: 2024-09-02 13:20 UTC+01:00 (CET)

Description:

We are investigating network problems reported by customers that route traffic through Cloudflare

Customer Impact:

Customers that are routing traffic to SysEleven networks through Cloudflare experience high retransmission rates and traffic loss.

Update: 2024-09-02 14:20 UTC+01:00 (CET)

We are still investigating.

Update: 2024-09-02 14:49 UTC+01:00 (CET)

We are observing significant improvements of the retransmit rates on some affected loadbalancers but will continue to observe and analyse the issues.

Update: 2024-09-02 16:25 UTC+01:00 (CET)

Since the Problem seems to be solved for most customers, we declare the incident as solved. Further investigations into the reasons for the problems will continue.