Tuesday 21st March 2023

Control Plane Services (SysEleven - FES) INCIDENT: Metakube Control Plane issues in Region FES

Affected Components: MetaKube Control Plane, region FES

Incident Start: 21st March. 2022/08:30 am


Description:

We're investigating sporadic issues with out MetaKube Control Plane / API in our FES region that may have started already before.

Customer Impact:

Some API commands / changes are not properly executed.


Update: 09:10 am

We created a workaround for an issue in a networking component but affected pods are not restarting automatically, we are triggering that manually.


Update: 09:25 am

Most affected pods have recovered but managing the cluster is still impaired.


Update: 10:13 am

A fix has been implemented but will take some time to be rolled out.

Actions using the Cloud Controller Manager (i.e. creating Load Balancers, getting IPs for Nodes...) do not yet work for the affected clusters.


Update: 12:38 pm

The fix has been rolled out and clusters are running as expected again.

Update: 23st March. 2023/16:25 pm

Reason for Outage (RfO) (eng)