[rocksmyfaceoff.info]

I’m attempting to fix a configuration error with PHP and its use of curl, so there may be minor downtime this afternoon.

I’m also doing my monthly audit of software on the machine that may be out of date.

In some good news for this Weblog for once, the exploit that kicked off all of this recent instability has been verified as closed with the datacenter. I’m relieved—that ticket was outstanding for about three weeks!

As you might have noticed, we’ve had Apache downtime on the server since ~8:45 a.m. today [1345 GMT 21-Jul-2005]. I’m continuing efforts to resurrect it, but unlike Lazarus, it just doesn’t want to come back to life.

Anyhow, catch up on today’s bombings in London in the meantime.

9:48: I’m now rebooting the full server to see if that will resolve the issue. There aren’t any reasonable warnings showing up.

9:55: That resolved it.

10:07: It appears that the software firewall is locking everyone out but me. I’m working on it.

I’ve got the nightly backups dumping straight to the NAS now, which is certainly preferable to leaving them on the machine. Last night’s copy was even interrupted by the automated backup routine. Again, I’ll be auditing these backups before taking any further steps.

We finally resolved all connection issues with the NAS backup at the datacenter. I am presently running cp -r /backup/* /mnt/backup to have a nice backup of everything on the NAS. I’ll probably “accidentally” delete one of my accounts tomorrow and verify the efficacy of a full restore from these backup files before further contemplating the wipe-reinstall-restore option.

Thank you for your continued patience.

The datacenter is troubleshooting some issues we’ve been having with the firewall solution we use. The server may be unreachable for brief periods of time. We’ll do our best to minimize that downtime.

I’ll update when the troubleshooting session is over.

As a NASA contractor, I’m beginning to feel some symbiosis between the travails of the NAS Backup server and the current issues holding up the launch of Discovery on STS-114.

Anyhow, unless the datacenter gets the backup server in place by 6:00 p.m. Central tonight, I rather doubt that I’ll be undertaking a backup-reload-restore process this weekend. I want to make sure that I’m not rushed for time and can do the job right, and that means having a weekend to do it. I hope that you’ll all understand.

It does seem that the security measures in place are holding up for now. I’m having to spend more time monitoring the server than I’d like, but it’s performing adequately.

As always, thank you for your continued patience.

I was planning to begin the server downtime at 2300 GMT [6:00 p.m. Central], but because the datacenter has not yet given me access to the NAS backup server due to downtime on their part. I don’t plan to start the downtime anytime before 0500 GMT [midnight Central] tonight. I wish to use the NAS backup due to the ability to stay within the datacenter itself and, therefore, leverage the high bandwidth of that internal-to-the-datacenter connection to minimize the downtime after the operating system is reloaded.

Until the downtime begins, there may be sporadic periods of server instability as we are still vulnerable to brute force attacks. This vulnerability will be addressed during the downtime.

Thank you for your continued patience.

Update, 0327 GMT [10:27 p.m. CDT]: Still no word from the DC. I’ll be on the phone with them in the morning.

There will be downtime this weekend while I back up everything on the server in preparation for an operating system reload. As you’re probably aware, we had a server exploit with a spammer that’s caused some significant downtime this week. This, combined with some filesystem issues, has caused me to move towards an OS reload.

I will disable services on the server prior to the backups, and we’ll restore from the backups.

Please check here for further updates, including an announcement on time. I was planning for 6:00 p.m. Central tonight, but at present, the datacenter’s NAS backup solution is still offline, so I’m not going to do anything until then.

[rocksmyfaceoff.info] is all about the server status of [rocksmyfaceoff.net] and sites hosted there. Please check here if you’re worried about the server being down or something.