On the morning of 12 February 2020, a major disruption occurred due to a network failure. The failure affected all our services and servers including our telephone exchange.
The error, which was detected at around 10:30am, has now been rectified and most services and websites are back up and running since around 2:10pm.
A few Managed Servers experienced problems with databases when they came back online after the network problem was resolved. The problem required manual action by our technicians. It has been resolved since 17:45.
First of all, we are sorry for the problems they have caused you as a customer. We understand that you expect your services to be online, something we work hard to deliver. Today we have not achieved that and for that we apologize.
We have published an incident report. It describes in more technical detail what the problems were, and how we’re working to prevent similar issues from happening again.
Frequently asked questions and answers
- On both hosting and Managed Server we have an uptime guarantee of 99.6% although our actual uptime is much higher than that. In 2019, we had 99.994%, which is about 2.5 minutes of downtime in a month.
- During the downtime, no email was received. How it gets through now that everything is back up and running is a little different depending on the sending mail server.
Either the sender will get the mail back with a note saying it couldn’t be delivered. Or it will try to be delivered a few more times. Then it should arrive but may take a number of hours depending on how it is set up.
If it’s really time critical and you don’t want to wait for the sending mail server to try again, get in touch with the people you’re waiting for mail from.
In any case, they don’t disappear into thin air and the sender is usually notified if it doesn’t arrive. - If you want higher guaranteed uptime and more redundancy than 99.6%, we offer this in separate SLA packages. Please contact us and we will tell you more.
- We have a status page that is completely separate from our own site. You can always use it to get information in the event of a breakdown.
You can find it at oderland-status.eu. - Of course.
It’s important for us to be transparent and tell you about problems and failures.
We have (2020-02-13) published an incident report telling you what went wrong and how we are working to prevent similar problems in the future. - At 14:10, we published an overly positive forecast for Managed Server because there were still a few servers affected by the database issue. The update on the database problem was then delayed too long (15:30).
Here we should have communicated more frequently and better.
Status updates from 12 February
Update 17:45 All Managed Servers with database problems have now been fixed and are online. This means that all of our services and servers are fully online and the outage is fully over.
The network is stable and working well.
We will be back shortly with an incident report with information both for those who are technically interested and for those who are not.
—
Update 17:30: Right now we have 4st Managed Servers with corrupted databases left to fix and are starting to see the light in the tunnel.
—
Update 17:10: We would like to clarify an ambiguity from 16:50. So our engineers are fixing the databases that have been corrupted so that no data is lost. So there is no question of a restore of the backup. This is also why it takes a little longer for each server.
—
Update 16:50: We regret that our update at 14:30 about Managed Servers was optimistic. Some Managed Servers (about 10pcs) came back with corrupted databases due to the network problem.
Our engineers are working on restoring the databases on these servers, and it is a full-strength manual server-by-server effort. These will come online gradually throughout the evening.
—
Update 15:30: Some Managed Servers are still experiencing some consequential problems with corrupted databases due to the network failure. We are in the process of betaing off one by one and restoring.
—
Update 14:10 The last consequential issues with some Managed Servers have been identified and fixed. All should be back online, or coming online in a few minutes.
—
Update 14:05 It looks like the vast majority of servers are now online, including our own site.
—
Update 13:45 We continue to have problems with some Managed Servers not wanting to go live. Engineers are working on it and we hope this too will be resolved soon.
—
Update 13:15 Most of our servers are up and running again, except for our own. This will also affect our own email for a while longer. The telephone exchange is back up and running. If you have a question about the downtime, we ask you to wait a bit if you can. We will send out information to all customers as soon as possible. If you have a support question, you are of course always welcome as usual.
—
Update 12:35 We are in the process of bypassing the network switching equipment that is most buggy at the moment, and some websites and servers are already online.
It should get progressively better with more servers online on an ongoing basis now, although we still can’t guarantee it’s 100% resolved yet.
We will continue to update you on an ongoing basis.
—
Update 12:20 Sorry there’s been a bit of a delay between updates now. Those of you who are technicians yourselves know what it’s like to troubleshoot. Time passes and you think you are close all the time and want to try “one more thing”… We’ll try to update more frequently again now.
What we thought would solve the problem at 11:50 turned out not to really do so. Our engineers have still identified the area where the problem is, but as you have noticed, they have had a hard time finding a definitive solution.
That’s why we don’t have an ETA yet either.
For those of you technically interested, our analysis afterwards will of course contain even more technical information than now.
—
Update 11:50: The technicians in the hall are testing a change now that we hope solves all the problems, and will just configure the last thing. We hope this solves the problem and we’ll be back up very soon!
—
Update 11:30: We are still troubleshooting. Technicians have found the root of the problem but have no timetable yet for when it will all be resolved.
—
Update 11:00: Troubleshooting of the network problem is still in full swing. We have technicians on their way out to the server hall to resolve the issue on the spot while other technicians are sitting and trying to remedy the problem. We will update here as soon as we have any more information available.