From bad to worse
My personal website noble.gen.nz is still down and news from the hosting company is getting grimmer with each update…
Tuesday 26 February 2008 – 19:00
Last updated: Wednesday 27 February 2008 – 09:34
Work to date on restoring service for those sites that are hosted on the server Mitosis, has proved to be extremely difficult, with further unexpected issues being encountered. The restore process has had to be restarted, extending out the ETA of service restoration that we had hoped would be by close of business today. It is unlikely at this stage that the service will be able to be restored this evening. We are taking this opportunity while restoring content to virtualise the server, mitigating issues after the content restoration has been completed, and also providing some future redundancy should issues of this nature ever arise again.
We regret that we have not been able to restore service as this point in time to those customers affected by this outage, and are committed to ensuring a resolution as quickly as possible.
–update— 23:00 26 Feb
The restore process so far is going as expected with no reason to expect any issues at this stage to hamper the full restore
–update— 07:20 27 Feb
Good progress has been made overnight with the data replication, the process however has taken longer than expected. Senior technicians will soon be able to ascertain an ETA for having the remaining steps finalised before full restoration is made.
–update— 09:34 27 Feb
Because of the complexity of cpanel restoration as well as the varying size of accounts on the server to be restored a firm eta is impossible. We have dedicated a tech to the restoration of this system as well as made all resources available to him. As we are rebuilding this machine we are also virtualizing it to make restoration a much easier and faster from here on in. Again we appologize for this issue
but wait, there’s more…
Wednesday 27 February 2008 – 13:30
The issues caused by the failure of Mitosis have had far wider reaching impact than we have expected. Initially the issues were related to a hardware failure. Replacement of the parts to get the server online proceeded without any issues experienced. The issues begun when we started restore on the operating system, cPanel and customer content. Due to data being corrupt, the restore process has taken longer than expected. Once the restore was complete (this morning) testing begun to bring the server back online. The net result of the testing showed that the server had been compromised via an insecure web page. The person or person’s who have accessed the server have uploaded a number of exploits which have caused further damage to the data upon the server. At this time we are hoping for resolution toward the latter part of today. We apologize for the inconvenience at this time and ask your patience while we work through the issues. We are doing all we can to ensure that the services are resumed as quickly and securely as possible.
By my reckoning this outage is rapidly approaching 36 hours (or more). Just as well I don’t depend on the domain for business and/or income…
The one piece of praise I can give Kiwi Web Hosting is that they are keeping customers informed. Kudos for not leaving the end users in the dark (despite the fact the server is!)