[rocksmyfaceoff.info]

I think the title is self-explanatory; nothing is demonstrably wrong with the box. Am investigating.

1729 CDT: The issue is limited to rocksmyfaceoff.net, not www.rocksmyfaceoff.net. Very strange.

1924 CDT: It has apparently re-resolved. :shrug:

Okay, so I thought that we were just getting mailbombed. [We were.] Turns out that Mark is testing out a different piece of forum software, and he’s been building a search index since last night. Makes so much more sense.

• We’re back online. Total downtime ~20 minutes. Better’n Twitter. #
• Wuhoh … time to run REPAIR TABLES on The Rumor Forum! Sonofa. #
• Okay, everybody back into the pool. And thanks to Smiley for getting the database tables repaired quickly. #

Powered by Twitter Tools.

A spammer is performing a botnet-based spamming attack on various non-existent addresses on rmfo-blogs.com right now, and that’s causing significant load on the server. It’s not thing that the server can’t handle, but it is working extra hard, and so other services may be slow. Because of the distributed nature of this mailbombing, there’s really no way to ban specific IP addresses and have that solve the problem for us.

Oddly enough, I’m not sure that there’s ever been a legitimate email address on rmfo-blogs.com.

The database server began acting up about ten minutes ago, so I attempted to shut it down for a reboot. It never wanted to do so successfully, so I took the Web server offline to allow the database server to close, but it never did. I forced a reboot of the machine, but it’s not come back online yet. Will advise.

1345 CST: We’re back online. Sorry about that!

1350 CST: Yes, we know that the tables are hosed on The Rumor Forum. A nice REPAIR TABLES run will work, but it’ll take time. Go play in traffic or something if you’re angry about this.

• WordPress upgrades are complete, save for those users who do it themselves. Those folks are asked to upgrade as soon as is possible! –GFM #

Powered by Twitter Tools.

Sounds like a broken record, I know; thankfully, the nice folks at ConfigServer gave us a 20% discount on the tuning they do for us because we just had it done 10 days ago. [I didn’t even ask for it; they just did it. They rock.]

As a result, I expect that the tuning will happen tonight. In conjunction with the changes already made [we’re now running Apache2 and PHP5], they’re going to upgrade MySQL to 5.0 from 4.1. There really shouldn’t be any issues with that, but if you use a database-based application and have problems after the upgrade, let me know and I’ll help you work them out. [WordPress, SMF, and phpBB are all abstracted enough that I really doubt it’ll matter.]

ConfigServer’s based in the UK, so any downtime would be late Monday evening into early Tuesday morning.

• There won’t be another backup run at 11:30 tonight. Not enough time for it to be completed. #
• Account restorations are underway and are approximately 40% done by account count [not by time spent]. Will update as progress is made. #
• Some, but not all, of the IP’s have been reallocated. I’ll finish the rest ASAP. #

Powered by Twitter Tools.

Everything should be up and running now.

Please let me know if not.

Services may be wonky … report it and I’ll look after it.

Going to nap now.

I’ve nearly completed the command line work for restoring all accounts. Because I’m at work, I haven’t completed the IP re-allocation work. I’m going to try to sneak home here at lunch and do that, but it may be later this afternoon before I can perform this. Your data appears to have survived; I just have to get it all back up and online.

Thank you for your continued patience.

Update 1505: All the accounts have been unpacked. Most of the IPs have been re-allocated, but not all. I’ll finish that this evening, and then we ought to be back up and running. I’m going home to see my folks in the morning, rather than tonight, so … I should be able to finish everything.