Past Incidents

Affected Components: Metakube API, region CBK and FES

Incident Start: 24th Nov. 2022/12:15 pm

Description:

We are currently experiencing a partial outage with control plane services in our DBL region. This may result in issues when accessing the Metakube API in the Regions CBK and FES as well. We will update you shortly.

Update 01:10 pm:

Metakube API timeouts have been resolved.

Update 01:35 pm:

All arisen issues have been mitigated and we continue to monitor the issue at hand. Should you experience any limitations or reduction / disruption please contact us.

Update 04:09 pm: We were able to fix the underlying issue with an update. We also haven't seen any issues with any control planes.

Affected Components: SysEleven Network, SysEleven API, all regions potentially affected**

Incident Start: 19th Nov. 2022/07:35 pm

Description:

We are currently experiencing issues within our network. Should you be affected, please contact us via our emergency hotline. Opening a ticket prior to calling us, helps us to evaluate, rectify and resolve issues that may persist with your setup.

Update 09:10 pm: All issues have been resolved. As far as we know at the moment, one of the couters at our Berlin Luetzowstraße datacenter was unfunctional and needed to be resetted. These routers are redundant, however, there was impact seen shortly after the failure of the router until all connections successfully moved to the redundant system.

Affected Components: SysEleven Stack, region CBK

Scheduled Start: 17. Nov. 2022/10:00 pm Scheduled End: 18. Nov. 2022/01:00 am

State: ** DONE**

Description:

• Scheduled maintenance of our SysEleven Stack in Region CBK

Customer Impact during the Maintenance:

• network availability will be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack, region DBL

Scheduled Start: 15. Nov. 2022/10:00 pm Scheduled End: 16. Nov. 2022/01:00 am

State: DONE

Description:

• Scheduled maintenance of our SysEleven Stack in Region DBL

Customer Impact during the Maintenance:

• network availability will be affected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: Monitoring Icinga2 problems, region BKI/BLU

Incident Start: 14th Sept. 2022/03:30 am

Incident End: 14th Sept. 2022/5:45 am

Description:

• Problem with icinga monitoring system
• expected false positive alarms

Customer Impact:

• false positive alarms
• potential missed alarms

Affected Components: Hardware node failure, region BLU1

Incident Start: 10th Nov. 2022/10:17 am Incident End: 10th Nov. 2022/10:35 am

Description:

• Malfunction of a hardware node
• Restart of the hardware node was necessary

Customer Impact:

• During the period of restart, there was a short interruption in the availability of the systems.
• Affected customers were notified via E-Mail.
• Please check the affected systems for their full functionality.

Affected Components: SysEleven CloudObserver

Scheduled Start: 08. Nov. 2022/01:45 pm Scheduled End: 11. Nov. 2022/06:00 pm

State: SCHEDULED

Description:

• We are migrating various components of the SysEleven CloudObserver from Cortex to Mimir.

Customer Impact during the Maintenance:

• No customer impact expected

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: Core Routers, region BKI

Scheduled Start: 7. Nov. 2022/10:00 pm

Scheduled End: 8. Nov. 2022/03:00 am

State: SCHEDULED

Description:

to ensure highest levels of stability and availability, we will perform planned maintenance on our core-router(s) at Berlin Kitzingstraße between 2022-11-07 22:00 and 22-11-08 03:00. We will perform software updates and exchange configuration. Customers will experience service interruptions and re-routings during this maintenance window. In the case of redundant services, the corresponding traffic is moved before the maintenance in order to avoid service impact. We apologize for any inconvenience this maintenance work may cause.

Customer Impact during the Maintenance:

• PVC: reduced redundancy
• CBK: reduced redundancy
• Fortigate Firewall / VPN: reduced redundancy
• DDoS Scrubbing: loss of protection, but no service impact

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update:

01:45 AM: The planned maintenance on our core-routers has been completed successfully. According to our monitoring systems, all services are up and running.

Affected Components: internal DNS infrastructure

Scheduled Start: 01. Nov. 2022/06:00 am

Scheduled End: 01. Nov. 2022/08:00 am

State: FIXED

Description:

Starting at 6:00 am on November 1st 2022, we will migrate part of our internal DNS infrastructure to an up-to-date platform. This Maintenance will last over the course of the morning, but will have no impact on the operation of your customer systems. DNS queries will continue to be answered throughout.

However, if you observe any problems contrary to expectations, we will be happy to answer any queries about this via our helpdesk or by e-mail to support@syseleven.de.

Customer Impact during the Maintenance:

• No customer impact expected