Past Incidents

We are currently facing a service failure in our central storage appliance in our datacenter Berlin Kitzingstraße. This leads to problems in access to NFS shares. We are working on a solution with high pressure.

[update 10/06/2021 13:44]

Storage Cluster seems to have switched off due to a power outage. Power supply is fixed and nodes are booting up.

[update 10/06/2021 14:23]

The cluster could not yet be recovered completely. We are still working on the solution.

[update 10/06/2021 15:17]

We're still working on the complete recovery of all cluster nodes.

[update 10/06/2021 17:27]

The functionality of the Isilon Storage Cluster could not yet be fully restored. After 2 of the 6 cluster nodes could not be started successfully, the cluster was initially not writable. In the meantime, we are observing successful write operations in the cluster and are taking further measures to further stabilize the status by remounting the shares.

While we continue to work with the manufacturer's support to restore functionality, we have activated the emergency plan and are preparing to move the clients to the second Isilon storage cluster at the Berlin Lützowstraße site. There, the shares are available with a daily synchronized status, i.e. up to 24h old. We try to synchronize the changes from the original cluster or, as an alternative, provide the old share as an additional read-only mountpoint.

If the timely recovery of the functionality is hopeless, we will start pivoting the shares soon, without additional announcement.

[update 10/06/2021 18:40]

Systems are beginning to recover. We are still working with high pressure to permanently fix the problem.

[update 10/06/2021 19:14]

We are currently preparing the announced move to the cluster in the second data center.

[update 10/06/2021 19:50]

At 19:50, we started moving customer by customer to the cluster in the other data center. We do this one at a time so that the load on the cluster does not increase to fast.

[update 10/06/2021 22:30]

We have restored all systems. At the moment, rework is still in progress.

[update 11/06/2021 00:03]

We were able to restore functionality for the most part by migrating the shares to the replacement system. Where migration did not take place, we contacted you directly. According to our monitoring systems, all services have been restored.

If you still experience problems, please contact us.

We regret the outage and are continuing to investigate the exact cause and may contact you to take further action.

[update 12/06/2021 12:00]

We monitored the situation in detail overnight and were able to observe stable operations. We will continue to analyze the cause and the effects and contact our affected customers with detailed error analysis.

If you have any questions or unexpected behavior occurs in your setup, please directly contact our support.

Between June 02th, 2021 12:00 (UTC +1) and 13:50 (UTC +1) we will perform planned maintenance

We will update of our seed infrastructure and ingress addon components. No actions required from the customers.

Should you have any trouble following the maintenance work please contact us.

There were problems with authoritative cloud name servers, which could lead to partial or full incapability to resolve DNS queries for zones hosted in SysEleven Stack DNSaaS. We have identified the problem and applied all necessary fixes at 03.06.2021 00:06. We are sorry for any inconvenience.

We will be carrying out maintenance work on hardware components from 01.06.2021 23:00 to 02.06.2021 06:00. There will be short interruptions in the accessibility of your systems due to server restarts.

Affected are virtual servers hosted on PCS6 infrastructure in datacenters Berlin Lützowstraße and Berlin Kitzingstraße. All affected customers have been informed individually in advance.

We need to recreate some of the external load balancers services in front of our managed applications. During the migration there will be some downtime. Affected services are SysEleven Harbour and SysEleven SSO.

-- ENGLISH VERSION BELOW --

------ deutsche Version -----

Zur Sicherstellung der Stabilität und Verfügbarkeit des NFS Shared Storage werden wir in der Zeit vom 25.05.2021 23:00 Uhr bis 26.05.2021 06:00 Uhr Wartungsarbeiten an den Storage Clustern an den Standorten Berlin Kitzingstraße und Berlin Lützowstraße durchführen.

Während der Wartungsarbeiten werden einzelne Nodes des Storage Clusters außer Betrieb genommen. Dazu werden aktive Verbindungen zunächst auf die verbleibenden Nodes umgezogen. Es kann dabei zu kurzen Unterbrechungen in der Erreichbarkeit der NFS Shares kommen.

--- english version ---

In order to ensure the stability and availability of the NFS shared storage, we will carry out maintenance work on the storage clusters at the Berlin Kitzingstraße and Berlin Lützowstraße sites from May 25, 2021, 23:00 to May 26, 2021, 06:00.

During the maintenance work, individual nodes of the storage clusters will be taken out of operation. For this purpose, active connections will first be moved to the remaining nodes. This may lead to brief interruptions in the accessibility of the NFS shares.

01:30 - Wartungsarbeiten sind erfolgreich abgeschlossen

Between May 18th, 2021 12:20 (UTC +1) and 13:00 (UTC +1) we will perform planned maintenance

This will recreate some of the cluster control plane pods. No downtime is expected.

Should you have any trouble following the maintenance work please contact us.