Server Downtime Info

[MikeWarner]

Since last night, the site has been closed due to a corruption of the sql database that runs this site. The main table which contains the posts started showing errors and was not responding to rapairs and it is a 1gb table and our server /tmp folder was not big enough to hold the temp files required to do the work.

I looked to the backup files, but to my horror, the backup system was faulty. It was running the backup, but somehow creating empty files. In all honesty, at this point I almost puked, I felt so ill, knowing the only full working backup we had was August last year (i.e. 250,000 posts missing). I didn't sleep well last night at all and had to beg my boss for the day off to try and sort this. I really felt like we were likely to have to use the August backup.

After try to sort this out myself (and after much pulling out of hair), we had to opt to have the table repaired professionally by our server hosts. Unfortunately, they charge over £100 an hour labour. We do not have the final invioce yet, but it will be at 2 hours work.

Obviously, this cost needs to come of the club funds, which will impact on the money available for other club activities. As such, we would welcome any donations to help us cover this recovery cost via PayPal sent to admin@migweb.co.uk

I have already created a backup of the site, so we are safe now. We also have a new server ready to move to in the next week. This new server runs different backup software as well as having a remote backup system. This sort of thing should not happen in the future. I will of course be increasing the size of the /tmp partition before I do anything else...

Again, donations would be greatly appreciate - the money spent has enabled us to recover all the posts, and no data was lost. Many thanks.

Mike

[alistairolsen]

Well done mike!


Whats with the new appearance,was that timetabled, or a result of the downtime?

[MikeWarner]

New look came online yesterday before the issues. The problem came up when I ran a table repair (which is a standard maintainence things).

The lack of temp floder space is what caused the crash. Bascially, the table gets bigger all the time as it holds all the posts on the site. When you run a repair and optimise, it uses the /tmp folder to store it, so the same space is needed in the temp folder as the size of the table. The table is now just over the size of the folder, hence why this became a problem that wasn't an issue before.

[Melv]

most of that is gobbledegook to me Mike but well done for your efforts.

[Paul]

Bugger, glad its sorted now!

[MSG]

Was it the thunder and lightening that caused the corruption of the sql database that runs this site?

[Dom]

I reckon it was Lethal ............

[Lugy]

We should get a back up site lmao

EDIT, when I pressed "post reply" I got a message saying something like "post reply.php Do you want to save" or something

[Dom]

We should get a back up site lmao

We have - its called VXON.

When MIG goes down we all bugger off over there and vice versa

[Lugy]

We have - its called VXON.

When MIG goes down we all bugger off over there and vice versa

It's one place I've never been lmao

[Stephen SRi]

mike you are a god
was missing mig like crazy even went to the dark side ie xvon

ps lugy you dont want 2 lmao

[MikeWarner]

Thanks to those who have donated already.

[Jack]

If you mount /tmp to a ramdrive...

Nah. I'll shut up.

[Benn]

Well done on getting it sorted, sorry to hear of all the pissing about you had to go thru and the stress.

[DaveB]

I thought you were just trying to be funny with the "back in a few moments" comment that was live for nearly 4 hours

[MikeWarner]

Yeah - though it would be a quick fix, but I was wrong....