HAM1 S3 sync issues

S3 data replication between DUS2 and HAM1 is back to normal since yesterday 19:10

S3 data replication between DUS2 and HAM1 is back to normal. We will keep watching overnight and close the incident tomorrow if it remains stable.

The situation is stable except the replication. We are working on this one.

The overall situation has improved, we have still issues with the replication and are still watching.

We are still watching and come back to you in one hour.

We are still watching if the situation is stabilizing. We will get back to you within the next hour.

• Replicated S3 storage as well as Volume Storage currently experiencing performance issues leading to higher latency times and/or timeouts
• The situation is currently stabilizing, but not yet fully contained
• Further maintenances might be necessary to fix the issue completey and will announce them on the status page on short notice

We'll get back to you within the next hour.

We are still working on the solution we will get back to you within the next hour.

The slow DUS2 to HAM1 synchronization is likely caused by a version mismatch. Upgrading HAM1 to match DUS2 would trigger a known Ceph bug that causes OSD restarts. We are testing a solution to prevent the OSD restarts, which should also resolve the sync slowness.

We are actively investigating the issue. We are evaluating and working on potential options. We will post an update within the next hour.

We are still investigating the issue and we will inform you as soon as we got new information.

Currently investigating.

We wanted to inform you that we are currently investigating issues with the S3 sync between DUS2 and HAM1 region.

As soon as new information arrives we will update you accordingly.