Unfortunately, our host that manages our Cloud Virtual Private Server (VPS) has suffered a catastrophic failure. The kind of failure that is very rare. Below is the email we received from them and we are working very diligently to bring SeaDooSportBoats.com back online. In the mean time, please follow our twitter, @sdsb
, for the latest updates.
At this time we are working to restore a local backup we have from 30 days ago (12/20). We know this is not ideal, but we are trying to stay positive since 30 days of lost data is better than losing everything, so we are hoping this backup restores without any major hitches. We will keep you posted as we attempt to restore this backup and hope to have service restored as soon as possible. We sincerely apologize for this huge inconvenience!
Feel free to comment below to support us during this difficult time and we will provide updates as soon as we have them.
As many of you are aware, the A Small Orange Cloud Hosting platform suffered a prolonged outage that began in the late afternoon (Eastern time) on Friday, January 20th. At the most basic level, this outage was due to a disk failure in the array.
The cloud storage configuration, which is made up of a 16 disk SAN array, can tolerate one drive failure and continue to operate normally. This automatic failover provides the reliability that has made cloud platforms popular. In last night’s incident, however, the SAN array had two drive failures, which is beyond the tolerance fault level of the system.
Numerous attempts have been made to rebuild the array and regain access to the data, but so far those attempts have failed. Due to the fact that two drives were lost, we feel at this point that all data across the entire cloud platform is lost and is likely unrecoverable.
We are continuing to work on the storage system and with its manufacturers to attempt to recover the data, but we wanted to let you know what the situation is at this point.
We are deeply sorry for the downtime you have experienced so far. As a web hosting company, letting customers know that data has been lost and is unrecoverable is the worst thing we ever have to do. We stand behind the products we sell and, in the rare occasions when there are failures that have a potentially damaging affect on our customers, it is a difficult thing to have to share with you. We know you count on us, and we hate falling short of expectations. Your trust has helped us grow into the company we are today and we are deeply grateful.
We know you will have many questions as to why and how this happened, and we will be happy to answer whatever questions we can. Unfortunately, A Small Orange leased the cloud platform’s hardware from a third party datacenter for our customers’ use. In some cases, we may not be able to fully answer your questions until we are done investigating. The monitoring of the integrity of the hardware system and the disks were the responsibility of the data center we leased the hardware from. We are working to get answers regarding when the disk issues occurred, how they occurred, whether there was any notification of impending disaster and if that took place. While we can tell you what has happened to the system, unfortunately, we don't have answers yet as to why it happened. This is as frustrating for us as it likely is for you and we are sorry we do not have more information to share with you at this time.
Our lack of control over our current cloud hosting platform and our inability to monitor certain aspects of it (both due to the fact it was leased hardware), is what led us four months ago to invest significantly in the design of our own cloud platform. Many of you have gotten the emails regarding the migration that was due to begin this upcoming Monday. Our new cloud platform is wholly owned, controlled, and monitored by A Small Orange, and is designed to be fault tolerant to two disk drives vs. the current cloud set up, which is fault tolerant to only one failure. Our new cloud is currently up and working well and new customers are already being placed on it.
While we realize that there is little we can do to correct a situation as catastrophic as this, we do want to do whatever we can to help you recover from this situation. For those customers that would like to stay with A Small Orange, we will:
- provide you with 6 free months on the new cloud
- automatically upgrade you to the next highest package level at no cost
- offer you 2 GB of swap memory at no cost
- offer our R1Soft Bare Metal Backup Solution with multiple points of restore at no cost
You can find details on how to get your new cloud hosting account activated at: http://asmallorange.com/cloud-2012.php
We do realize that for some, this is a catastrophic failure that will result in your choosing to leave A Small Orange. While we hope that we can retain your trust and your business, we understand this won't be possible for everyone. If you choose to leave A Small Orange we will, of course, refund you any pre-paid fees as well as pay for your first six months of comparably priced hosting at a new hosting company. You can find details on this at: http://asmallorange.com/cloud-2012.php
We will be reaching out to many of you over the next several days by phone to discuss what has happened and to answer any questions you might have. Customer service continues to be our number one priority at A Small Orange and we are working hard to do whatever we can to make this situation right.
We again extend our deepest apologies.
Very truly yours,
The Team at A Small Orange