Past Incidents

Affected Components: SysEleven Stack Ceph Object Storage, region FES

Incident Start: 18th Sept. 2023/14:51

Incident End: 18th Sept. 2023/15:33

Description:

At the moment we are facing issues with the Ceph Object Storage in Region FES.

Customer Impact:

• Writing or reading of objects in our Ceph Object Storage in region FES is impaired.
• The Quobyte Object Storage in region FES is currently not affected.

Update 15:33 :

• The Ceph Object Storage is recovering.

Affected Components: SysEleven Stack, region FES

Incident Start: 18th Oct. 2023/00:40

Incident End: 18th Oct. 2023/03:56

Description:

• Occurring errors are currently being investigated.
• During the maintenance the SDN cluster started to cause issues, we are investigating
• API is also not available during recovery

Customer Impact:

• Connectivity is restricted

Update: 01:48

Recovery is still ongoing, network connectivity could be restored partially

Update: 02:45

Recovery is mostly through, we are still fixing octavia loadbalancers

Update: 03:56

Last recoveries of octavia loadalancers are being finished up

Affected Components: SysEleven Stack API, all regions

Incident Start: 2023-10-06 12:09

Incident End: 2023-10-06 12:25

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 12:25

The API is generally working again (with a concurring incident of smaller scope still persisting, see separate entry).

Affected Components: SysEleven Stack API, region DBL

Incident Start: 27th Sept. 2023/03:23 pm

Incident End: 27th Sept. 2023/03:58 pm

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Managing (creating/deleting/attaching/detaching) block storage (cinder) volumes may fail or is not possible at all. Volumes in continuous use are not affected.
• Creating new instances may also fail.

Update: 03:58 pm: Services have been restarted, we are checking functionality.

Update: 04:03 pm: Affected services are restored, we are cleaning up some side-effects that occured during the incident. All newly triggered actions should succeed again.

Update: 04:16 pm: All services are verified to be fully operational again.