First let me apologize to everyone for taking so long to get back online. I'm very sorry for that. There is no reason why we could not have been online early the next morning instead of 3 days later. We are in the process of changing host, but since that will require more downtime, we are going to wait a few weeks before we make the move.
About the actual incident, I will not be providing technical specifics, but will outline what generally happened.
On Friday night, June 24th, we had an hardware issue with one of our drives which caused the server to fail. While one drive could no longer be used, the data on the drives was completely intact. One drive needed to be removed and replaced and the data transferred to a new drive. This is when we learned our host provider does not have anyone physically working at the data center over the weekend. Since this happened on Friday night there was no one there until Monday morning, so nothing could be done until Monday.
However we do keep remote backups. Early Sunday morning our host thought using the oldest backup we had from June 1st would solve the problem and restored that data without confirming with us first and in the middle of the night. This then caused panic and confusion since 24 days of data was missing. The site was shut down a few hours later. I was also at fault, I did not ask them what backup they were going to use, I had just assumed it would be the most recent backup, not a month old backup.
We then tried to restore from the last daily backup, but because the incident occurred on Friday night, just before our daily backup runs, the last backup was missing 22 hours worth of data. Since we had all the data, these was no reason to restore from this backup and lose data.
On Monday, I again stupidly assumed this would be a top priority for our host and we would be back online before noon. However our host did not get in contact with us until mid-day then it took about 12 hours to complete everything.
I apologize, I'm just as mad as you guys are and should be. This entire situation was completely unacceptable. We will be addressing all of these issues with our new host and new server once we move.
UPDATED: Seems we had another error over night which caused the site to go down again. This issue was been resolved. Hopefully that will be the last one. Sorry.
----------------------
All June 26th payments were refunded or voided.
About the actual incident, I will not be providing technical specifics, but will outline what generally happened.
On Friday night, June 24th, we had an hardware issue with one of our drives which caused the server to fail. While one drive could no longer be used, the data on the drives was completely intact. One drive needed to be removed and replaced and the data transferred to a new drive. This is when we learned our host provider does not have anyone physically working at the data center over the weekend. Since this happened on Friday night there was no one there until Monday morning, so nothing could be done until Monday.
However we do keep remote backups. Early Sunday morning our host thought using the oldest backup we had from June 1st would solve the problem and restored that data without confirming with us first and in the middle of the night. This then caused panic and confusion since 24 days of data was missing. The site was shut down a few hours later. I was also at fault, I did not ask them what backup they were going to use, I had just assumed it would be the most recent backup, not a month old backup.
We then tried to restore from the last daily backup, but because the incident occurred on Friday night, just before our daily backup runs, the last backup was missing 22 hours worth of data. Since we had all the data, these was no reason to restore from this backup and lose data.
On Monday, I again stupidly assumed this would be a top priority for our host and we would be back online before noon. However our host did not get in contact with us until mid-day then it took about 12 hours to complete everything.
I apologize, I'm just as mad as you guys are and should be. This entire situation was completely unacceptable. We will be addressing all of these issues with our new host and new server once we move.
UPDATED: Seems we had another error over night which caused the site to go down again. This issue was been resolved. Hopefully that will be the last one. Sorry.
----------------------
All June 26th payments were refunded or voided.
Last edited:
