Past Incidents

** Between January 25th, 2019 00:00 AM and 06:00 AM (local time) we will perform planned maintenance in our bno1 Location **

** Reason: Software Updates **

Should you have any trouble following the maintenance work please contact us.

Between January 23rd, 2018 00:00 AM and 06:00 AM (UTC +1) we will perform planned storage maintenance in Region DBL

During the maintenance window we will update the distributed storage software in our DBL region in order to fix a memory leak.

It is possible that IO requests are slow during the maintenance window.

Affected services:

• block storage
• ephemeral storage
• object storage

Not affected:

• local SSD storage instances

UPDATE 04:32 AM (CET): The maintenance has been completed successfully.

Should you have any trouble following the maintenance work please contact us at cloudsupport@syseleven.de

Between January 17th, 2019 07:00 AM and 08:00 AM (UTC +1) we will perform planned maintenance in Region DBL

During the maintenance window we will update the SDN software in our DBL region in order to fix a memory leak.

During the maintenance window the following services will not be accessible:

• access to the SysEleven Stack Network API (Neutron) will be interrupted in region DBL

The following services will not be affected:

• network availability will not be affected
• state of virtual machines will not be affected
• storage availability will not be affected
• SEOS (S3) will not be affected

Should you have any trouble following the maintenance work please contact us.

Root cause:

We created a oauth service with dex to authenticate our web applications against. Dex writes all of the session information into the etcd cluster. Due to a misconfigured monitoring we created over 500000 sessions in the etcd cluster. The database size grow from about 300MB to 700MB. That slowed down all cluster operations. During the whole incident customer applications were not affected, the MetaKube dashboard including cluster creation were regularly not working correctly when one of the master apiservers ran out of memory.

At the moment we are facing an issue with the Nova API in the region DBL.

Creating new VMs results in an error since 11:30 AM CET.

We are investigating.

We will keep you informed and apologise for any inconveniences.

UPDATE 11:53 CET: We have resolved the issue, and isolated the root cause.

We were facing issues with the SysEleven Stack Neutron API between Dec 29th 2018 and Jan 2nd 2019. Some Neutron API requests were slow, and a few requests might have timed out and failed. The reason was an almost-out-of-memory situation in the software of one of the three SDN controllers. The problem was fixed at 11:26 CET on Jan 2nd 2019.

Network connectivity of VMs were not affected by this issue.

We apologise for any inconveniences.