[rocksmyfaceoff.info]

Based on the response time previously, RAM installs happen within 12 hours of the request being inputted. If I can get that to happen today, then I’ll have the folks at Way to the Web perform a full server tuning after that. I actually expected it to be performed earlier today, but they didn’t get to it, which is just as well since we’ll have more RAM when it’s ready.

I’ll update this entry as soon as I have more information.

Update, 1230 GMT, 28 Feb: The NOC has the RAM and are working to schedule time for their techs to do the work. Their standard downtime window for this is four hours, but my general experience is that such downtime never exceeds an hour and averages 30 minutes. I’ll give as much notice as I can on the downtime; they notified me at 0220 CST that they were ready, and I let them know at 0600 that I was ready whenever they were.

Update, 1615 GMT, 28 Feb: The NOC is ready to install the RAM. There will be downtime of up to four hours, but more than likely one hour.

Update, 1700 GMT, 28 Feb: The NOC has installed the RAM and the server is back online. Performance may still be a bit wonky because services aren’t fully configured to fully use the RAM, and the server is always slow after first starting up.

Well, I’ve done all I know to do to try to resolve our Apache issues, and putting us under the DDoS-stopping firewall at the NOC hasn’t solved things. At this point, I’ve turned things over to the NOC to see if they can see something I’m not seeing. There’s probably something terribly obvious that I’m just not getting.

I really apologize, folks. I know how much this sucks.

Update, 1830 GMT: The NOC is actively working the problem, although I’ve not gotten an update. I know that part of what they’re doing to fix the issue is driving the load even higher, to the point that the server is largely unusable. If they can’t get it solved, I’ve found a vendor capable of doing performance tuning in 24 hours that will suffice. The cost is more than I’d like, but I’ll pay the cost to have a working server again.

Update, 2005 GMT: The NOC is finished with their work, and things seem to have improved, but I’ve talked it over with Mark, and we’re going ahead and biting the bullet and getting a cPanel/WHM expert involved as well. I’ve already placed the order, and as the service provider is in the UK, it’ll be early on Tuesday morning US time before they’ll get to it, likely. The server’s running slowly, but it is running. Sorta.

All: An excessive amount of incoming HTTP requests are significantly slowing down the server today, as you may well have noticed. I’m actively monitoring the situation, although the nature of botnet spammers is that they’re programmed so as to distribute the load and foil any IP-based blocking schemes.

Update, 1810 GMT: Until further notice, the Rumor Forum and the DMF Forum are offline. This will allow things to resolve on the server. I apologize for the inconvenience.

Update, 1930 GMT: Hang it all! I’m finally just giving up and rebooting the server, which will stop the backup process. We’ll just have to risk the data security.

Update, 1946 GMT: Server’s back online, so I’m bringing the forum back up.

We had another spammer hammer on the box pretty hard this morning, and MySQL failed to shut down properly. We already know of one database that is corrupted and having issues, but it’s our biggest one. Please let me know if any of your MySQL databases have any issues and I’ll help you resolve the problem.