Affected Components: SysEleven Stack, MetaKube clusters, MetaKube API, region DBL
Incident Start: 2024-03-25 10:18 UTC+01:00 (CET)
Incident End: 2024-03-25 13:00 UTC+01:00 (CET)
Description:
- Major Outage of multiple Hardware Nodes
- Occurring errors are currently being investigated.
Customer Impact:
- Storage and Compute services in the DBL region are disrupted
Update : 10:39
- One Rack in the DBL Datacenter seems to be down
- Rack is getting investigated by remote hands
Update : 10:50
- Datacenter operator confirmed power outage
- Hardware nodes are coming back online
- Follow up problems are being tackled
Update : 11:03
- We are still stabilizing the affected systems
- We see some recoveries coming in already
Update : 11:17
- We still see issues on software loadbalancers (Octavia) which we are being investigated at the minute
- We see a few CRC problems and will contact the affected customers individually after doing a deep scrub on the storage
Update : 11:40
- Octavia loadbalancer recoveries and storage scrub tasks are ongoing
