Past Incidents

Affected Components: SysEleven Stack, MetaKube clusters, MetaKube API, region DBL

Incident Start: 2024-03-25 10:18 UTC+01:00 (CET)

Incident End: 2024-03-25 13:00 UTC+01:00 (CET)

Description:

• Major Outage of multiple Hardware Nodes
• Occurring errors are currently being investigated.

Customer Impact:

• Storage and Compute services in the DBL region are disrupted

Update : 10:39

• One Rack in the DBL Datacenter seems to be down
• Rack is getting investigated by remote hands

Update : 10:50

• Datacenter operator confirmed power outage
• Hardware nodes are coming back online
• Follow up problems are being tackled

Update : 11:03

• We are still stabilizing the affected systems
• We see some recoveries coming in already

Update : 11:17

• We still see issues on software loadbalancers (Octavia) which we are being investigated at the minute
• We see a few CRC problems and will contact the affected customers individually after doing a deep scrub on the storage

Update : 11:40

• Octavia loadbalancer recoveries and storage scrub tasks are ongoing

Affected Components: SysEleven Stack Object Storage (Quobyte), region FES

Incident Start: 2024-03-24 19:14 UTC+01:00 (CET)

Incident End: 2024-03-24 19:28 UTC+01:00 (CET)

Description:

The Object Storage (Quobyte) API in region FES may have been intermittently unreachable.

Customer Impact:

• Writing or reading of objects maybe restricted.

Affected Components: MetaKube Dashboard API

Incident Start: 2024-03-012 16:00 UTC+01:00 (CET)

Description:

• Accessibility of the SysEleven MetaKube Dashboard API is not ensured.
• Clusters may not be reconciled currently

Customer Impact:

• Access to MetaKube Dashboard and API (including Terraform) might be not possible
• Clusters may not be reconciled currently

Update: 2024-03-012 17:30 UTC+01:00 (CET)

• Issues are fixed