Past Incidents

Affected Components: CloudObserver/Grafana

Incident Start: 5th April 2023/19:00 CEST
Incident Start: 5th April 2023/22:00 CEST

Description:

• We are currently experiencing issues with the CloudObserver, causing some missing trending data. We identified the problem and are working to resolve the issue.

Customer Impact:

• Metrics from dashboards in your Grafana might be missing

Update 21:30

• We identified and fixed the issue, we were also able to recover most metrics. In some rare cases you might experience a gap between ~19:00 and ~20:00
• We are still closely monitoring the system

Update 22:00

• All systems are back and fully operational

Affected Components: SysEleven Stack API, region CBK

Incident Start: 4th April. 2023/15:30 pm Incident End: 4th April. 2023/15:40 pm

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 15:42 pm The problem has been solved. The API is available as before.

We're currently experiencing issues with the cluster apiserver Load Balancer in the FES region.

Update (13:08):

A failover of the Load Balancer mitigated the issue. We're watching the situation and will investigate the root cause.

Update (13:30):

It seems, the issues were caused by a failover to a faulty machine. We stopped it and will replace the Load Balancer during a emergency maintenance window, which we will announce later.

Affected Components: MetaKube, region FES

Incident Start: 22st March. 2022/10:45 am

Description:

The load balancer to the cluster control planes did not behave correctly and caused slow or dropped connections.

This in turn caused nodes to show as NotReady and problems for kubernetes controllers in the clusters.

Update at 11:30am

Services are operational again.

Update at 15:15am

The load balancer caused problems again for a few minutes.

Incident over at 15:30

If you still experience issues, please open a ticket.

Update: 24st March. 2023/13:45 pm

Reason for Outage (RfO) (eng)

Affected Components: MetaKube Control Plane, region FES

Incident Start: 21st March. 2022/08:30 am

Description:

We're investigating sporadic issues with out MetaKube Control Plane / API in our FES region that may have started already before.

Customer Impact:

Some API commands / changes are not properly executed.

Update: 09:10 am

We created a workaround for an issue in a networking component but affected pods are not restarting automatically, we are triggering that manually.

Update: 09:25 am

Most affected pods have recovered but managing the cluster is still impaired.

Update: 10:13 am

A fix has been implemented but will take some time to be rolled out.

Actions using the Cloud Controller Manager (i.e. creating Load Balancers, getting IPs for Nodes...) do not yet work for the affected clusters.

Update: 12:38 pm

The fix has been rolled out and clusters are running as expected again.

Update: 23st March. 2023/16:25 pm

Reason for Outage (RfO) (eng)