Past Incidents

Currently you may experience troubles with the network API . Spawning and deleting instances or other network resources is not possible. If you experience any unexpected behaviour don't hesitate to contact us.

We apologise for any inconveniences.

UPDATE 22:45

We have identified and resolved the networking API troubles.

We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines.

We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem.

UPDATE 15. October

We are currently working the following action points:

• We identified and are working to resolve the root cause in our SDN software
• We will improve the monitoring of our networking APIs, to identify and correlate future troubles earlier
• We will improve monitoring of virtual machine port binding of our SDN software, so we can identify and fix troubled VMs faster in the future

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects changing network resources, as well as spawning new virtual machines.

Existing virtual machines are not affected.

Please contact us, should you have any troubles or need any assistance.

We apologise for any inconveniences.

UPDATE 16:15:

We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail.

UPDATE 17:25:

We resolved the API issues and connectivity issues for virtual machines.

All customers that have been affected by connectivity issues should have been informed by e-mail.

Please contact us should you still have issues or need assistance.

At the moment it is possible that virtual machines face issues with network connectivity. We are working on a solution. We will keep you informed and apologise for any inconveniences.

Update 21/09/2018 14:40
Most systems are accessible and we are working on some remaining problems.

Update 21/09/2018 15:20
Systems experienced another downtime of 10 minutes and are coming online now.

Between September 14th, 2018 11:15 AM and 12:00 noon (UTC +2) we will perform emergency maintenance in both Regions

We urgently fix the problem that caused the incident yesterday evening to prevent the issue from reoccurring. There should be no impact, however, there is a little risk that similar problems occour one last time.

• storage availability may be affected
• SEOS (S3) availability may be affected
• virtual machines may be affected indirectly by "freezes" accessing their storage
• network availability will not be affected but services might hang while waiting for storage
• the API will not be affected

Should you have any trouble following the maintenance work please contact us.

Around 21:00 CEST we had an incident which affected storage of SysEleven Stack instances in both regions. The issue disappeared and we are still investigating cause and effect. Affected customers are encouraged to open support tickets with affected resources, obeservations and timestamps, if available.

Update 10:40 CEST: The cause has been identified and we are planning an emergency maintenance to fix it to avoid the problem to reoccur.

Update 11:15 CEST: We are fixing the root cause in an emergency maintenance.

Update 12:00 CEST: The root cause has been fixed. To our knowledge there were some glitches in region DBL, but everything went smooth in region CBK.