Past Incidents

Affected Components: SysEleven Stack in region DBL

Scheduled Start: 9 May 2023, 22:00 CEST

Scheduled End: 10 May 2023, 01:00 CEST

State: SCHEDULED

Description:

We will update some components of the SysEleven Stack.

Customer Impact during the Maintenance:

• During the update the SysEleven Stack API is unavailable
• Object Storage API is not affected

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack in region CBK

Scheduled Start: 2 May 2023, 22:00 CEST

Scheduled End: 3 May 2023, 01:00 CEST

State: SCHEDULED

Description:

We will update some components of the SysEleven Stack.

Customer Impact during the Maintenance:

• During the update the SysEleven Stack API is unavailable
• Object Storage API is not affected

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack Networking in region FES

Scheduled Start: 25 Apr 2023, 22:00 CEST

Scheduled End: 26 Apr 2023, 01:00 CEST

State: COMPLETED

Description:

We will perform maintenance on networking gateways.

Customer Impact during the Maintenance:

• Interruption of external connectivity of virtual machines of up to 2 minutes
• VPN connections could be interrupted for up to 2 minutes

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack, region FES

Incident Start: 21st Apr. 2023/07:00 am Incident End: 21st Apr. 2023/17:50 pm

Description:

• There is an increased packet loss up to 8% for the IPv6 connections. IPv4 connections are not affected.

Customer Impact:

• Network connectivity has some issues for IPv6.

Affected Components: SysEleven Stack API, region CBK

Incident Start: 13th Apr. 2023/02:12 pm

Description:

• Performance of the SysEleven Stack API is degraded.
• We are investigating the situation currently.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources can be slower.

Update:02:32 pm

The situation is worse than a performance degradation, correcting the status towards a partial outage, we are further investigating

Update:03.20 pm

We pinned down the issue around 02:56 and are now observing the situation. In the end not only API but also Compute and storage (Block+Object) storage were affected. Changes to the monitoring and alerting are being planned.

Affected Components: CloudObserver/Grafana

Incident Start: 5th April 2023/19:00 CEST
Incident Start: 5th April 2023/22:00 CEST

Description:

• We are currently experiencing issues with the CloudObserver, causing some missing trending data. We identified the problem and are working to resolve the issue.

Customer Impact:

• Metrics from dashboards in your Grafana might be missing

Update 21:30

• We identified and fixed the issue, we were also able to recover most metrics. In some rare cases you might experience a gap between ~19:00 and ~20:00
• We are still closely monitoring the system

Update 22:00

• All systems are back and fully operational

Affected Components: SysEleven Stack API, region CBK

Incident Start: 4th April. 2023/15:30 pm Incident End: 4th April. 2023/15:40 pm

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 15:42 pm The problem has been solved. The API is available as before.

Affected Components: MetaKube Platform

Scheduled Start: 03. Apr. 2023 / 07:00 pm

Scheduled End: 03. Apr. 2023 / 10:00 pm

State: COMPLETED

Description:

Scheduled maintenance of the MetaKube platform in all regions.

No customer impact expected during the maintenance.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update 08:10 pm

Maintenance completed successfully.

Affected Components: Control Planes for clusters in FES

Scheduled Start: 31. Mar. 2023/07:00 pm Scheduled End: 31. Mar. 2023/09:00 pm

State: ABORTED

Description:

We will replace our load balancer in front of the control planes of MetaKube clusters in the FES region in this maintenance.

Recent incidents indicate that components related to the load balancer may have been corrupted.

Customer Impact during the Maintenance:

During this period, the public IP of the load balancer will change. We do however not expect any disruptions.

The maintenance is only related to clusters in the FES region and on AWS.

Update (08:20 pm):

We had to abort the maintenance after we saw that the new load balancer behaved faulty under load. We reverted to the previous load balancer and the situation is stable. We will investigate the problem further.

Update 05. April 2023

Reason for Outage (RfO) (de)

Affected Components: MetaKube, region FES

Scheduled Start: 30.03. 2023/07:00 pm CEST Scheduled End: 30.03. 2023/08:00 pm CEST

State: COMPLETED

Description:

• Scheduled maintenance on the Load Balancers for MetaKube in Region FES
• It will happen in the time frame but not take the whole hour

Expected Customer Impact during the Maintenance:

• The Kubernetes-API-Server for your cluster might be unavailable for a short time
• No impact to running services in your cluster

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities