Since last night, the site has been closed due to a corruption of the sql database that runs this site. The main table which contains the posts started showing errors and was not responding to rapairs and it is a 1gb table and our server /tmp folder was not big enough to hold the temp files required to do the work.
I looked to the backup files, but to my horror, the backup system was faulty. It was running the backup, but somehow creating empty files. In all honesty, at this point I almost puked, I felt so ill, knowing the only full working backup we had was August last year (i.e. 250,000 posts missing). I didn't sleep well last night at all and had to beg my boss for the day off to try and sort this. I really felt like we were likely to have to use the August backup.
After try to sort this out myself (and after much pulling out of hair), we had to opt to have the table repaired professionally by our server hosts. Unfortunately, they charge over £100 an hour labour. We do not have the final invioce yet, but it will be at 2 hours work.
Obviously, this cost needs to come of the club funds, which will impact on the money available for other club activities. As such, we would welcome any donations to help us cover this recovery cost via PayPal.
I have already created a backup of the site, so we are safe now. We also have a new server ready to move to in the next week. This new server runs different backup software as well as having a remote backup system. This sort of thing should not happen in the future. I will of course be increasing the size of the /tmp partition before I do anything else...
Again, donations would be greatly appreciate - the money spent has enabled us to recover all the posts, and no data was lost. Many thanks.